Big data DAG task flow scheduling method and system and storage medium
The invention provides a big data DAG task flow scheduling method. The method specifically comprises the steps that visual scheduling configuration is conducted on a DAG process based on nginx; an external system is accessed to the scheduling platform through a cross-platform interface schema-api, a...
Gespeichert in:
Hauptverfasser: | , , , , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention provides a big data DAG task flow scheduling method. The method specifically comprises the steps that visual scheduling configuration is conducted on a DAG process based on nginx; an external system is accessed to the scheduling platform through a cross-platform interface schema-api, and an instruction is transmitted to a scheduling center schema-master for further processing; in response to a scheduling algorithm preset by a scheduling center super-master, completing a DAG process, and carrying out splitting, task storage, result callback and execution notification operation; and the execution center schema-worker loads the required task engine to further complete the execution of the task and the progress tracking record, and stores the task and the progress tracking record. Through the design of each link, high performance and high expansion are brought into full play, along with the development of the Internet, the problem that new and old data, generally super-large data, are processed, int |
---|