Systems and Methods for Creating Modular Data Processing Pipelines

Systems, methods, and apparatuses are described herein that allow users to create and manage flexible, highly modular data processing pipelines. Such pipelines may be associated with any number of connected nodes connected via dependency injection to define the location and type of data that a pipel...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Lee, Edward, Kipers, Chris, Sun, Hai Po, Cha, Theodore
Format: Patent
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Systems, methods, and apparatuses are described herein that allow users to create and manage flexible, highly modular data processing pipelines. Such pipelines may be associated with any number of connected nodes connected via dependency injection to define the location and type of data that a pipeline uses as input or output and the operations to be performed by the pipeline. The pipelines may also be associated with context information, which specifies dataset-specific configurations and includes logic required to generate and execute the associated nodes. The context information may further include logic that allows for node substitution, caching of node output, data filtering, and/or dynamic node modification.