FAST TRAINING FOR A DATA PIPELINE MONITORING SYSTEM

Various embodiments comprise systems and methods to determine output attributes of a data pipeline. In some examples, a data pipeline monitoring system retrieves historical data generated by a data pipeline and determines generation dates for the historical outputs. The system identifies one or more...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
1. Verfasser: Haile, J. Mitchell
Format: Patent
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Various embodiments comprise systems and methods to determine output attributes of a data pipeline. In some examples, a data pipeline monitoring system retrieves historical data generated by a data pipeline and determines generation dates for the historical outputs. The system identifies one or more attributes of the historical data outputs. The system generates an output model that indicates expected output attributes based on the identified attributes of the historical outputs. The system generates an error threshold based on the model and applies the error threshold to outputs generated by the data pipeline. The system generates alerts when the outputs trigger the error threshold.