Storm task expansion scheduling algorithm based on data stream prediction
The invention relates to a Storm task expansion scheduling algorithm based on data stream prediction, and belongs to the field of data exchange networks. Through a monitoring module, the real-time operation data of a Topology task submitted by a user can be obtained, the degree of parallelism of a c...
Gespeichert in:
Hauptverfasser: | , , , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention relates to a Storm task expansion scheduling algorithm based on data stream prediction, and belongs to the field of data exchange networks. Through a monitoring module, the real-time operation data of a Topology task submitted by a user can be obtained, the degree of parallelism of a connected component in Topology under a situation that a component load is met is solved, and then, the degrees of parallelism of all components in the Topology can be solved through iteration. A time series model is used for predicting a data size which needs to be processed by the Topology, the optimal degree of parallelism of a startup component spout in the Topology under the situation is solved, the optimal degree of parallelism of each component in the Topology under a prediction condition is obtained, and scheduling is carried out. In scheduling, an on-line scheduling algorithm is used to reduce inter-node network communication to a largest degree and guarantee the load balance of a cluster. By use of the alg |
---|