Big data batch processing process control system and method

The invention relates to a big data batch processing process control system and method, and the system comprises a distribution platform which is used for carrying out the task analysis of big data of a resource pool, and publishing a task or dividing the task according to the task information obtai...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: WEI SHENGBAO, HAO JIANGBO, WANG JUNDE, ZHOU MING
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention relates to a big data batch processing process control system and method, and the system comprises a distribution platform which is used for carrying out the task analysis of big data of a resource pool, and publishing a task or dividing the task according to the task information obtained through the task analysis; different task information is sent to the corresponding execution agents through Kafka; the task information comprises task quantity and task state information; the execution agents are used for receiving the task state information through Kafka and executing corresponding data processing tasks according to different task state information; the resource pool is used for storing to-be-processed big data; the database is used for storing task information of the distribution platform; and in response to a request for executing the proxy or distribution platform, returning the task information or updating the task information. According to the method, multiple scheduling strategies are ma