Storm task expansion scheduling algorithm based on data stream prediction

The invention relates to a Storm task expansion scheduling algorithm based on data stream prediction, and belongs to the field of data exchange networks. Through a monitoring module, the real-time operation data of a Topology task submitted by a user can be obtained, the degree of parallelism of a c...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: DUAN HANGBIAO, XIONG ANPING, ZHU QINGYI, JIANG YAXIONG, JIANG YI
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention relates to a Storm task expansion scheduling algorithm based on data stream prediction, and belongs to the field of data exchange networks. Through a monitoring module, the real-time operation data of a Topology task submitted by a user can be obtained, the degree of parallelism of a connected component in Topology under a situation that a component load is met is solved, and then, the degrees of parallelism of all components in the Topology can be solved through iteration. A time series model is used for predicting a data size which needs to be processed by the Topology, the optimal degree of parallelism of a startup component spout in the Topology under the situation is solved, the optimal degree of parallelism of each component in the Topology under a prediction condition is obtained, and scheduling is carried out. In scheduling, an on-line scheduling algorithm is used to reduce inter-node network communication to a largest degree and guarantee the load balance of a cluster. By use of the alg