Big data batch processing process control system and method
The invention relates to a big data batch processing process control system and method, and the system comprises a distribution platform which is used for carrying out the task analysis of big data of a resource pool, and publishing a task or dividing the task according to the task information obtai...
Gespeichert in:
Hauptverfasser: | , , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention relates to a big data batch processing process control system and method, and the system comprises a distribution platform which is used for carrying out the task analysis of big data of a resource pool, and publishing a task or dividing the task according to the task information obtained through the task analysis; different task information is sent to the corresponding execution agents through Kafka; the task information comprises task quantity and task state information; the execution agents are used for receiving the task state information through Kafka and executing corresponding data processing tasks according to different task state information; the resource pool is used for storing to-be-processed big data; the database is used for storing task information of the distribution platform; and in response to a request for executing the proxy or distribution platform, returning the task information or updating the task information. According to the method, multiple scheduling strategies are ma |
---|