Data dynamic partitioning system based on node load
The invention provides a data dynamic partitioning system based on a node load. The data dynamic partitioning system comprises a load monitoring acquisition module, a prediction module, a data pre-partitioning module, a data migration module and the like, the node load is predicted by adopting a sec...
Gespeichert in:
Hauptverfasser: | , , , , , , , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention provides a data dynamic partitioning system based on a node load. The data dynamic partitioning system comprises a load monitoring acquisition module, a prediction module, a data pre-partitioning module, a data migration module and the like, the node load is predicted by adopting a secondary smoothing method, the AHP and entropy index weight method is combined, corresponding partitioning strategies can be obtained according to different data analysis applications, the load balance of the system is dynamically adjusted, and the response speed of the applications is increased. Thesystem aims at an application scene of a Spark and Messql distributed integrated framework, due to heterogeneity of node resources existing in a distributed environment, in order to reduce data transmission consumption between nodes, computing resources of each node are fully utilized, and parallel computing efficiency of application analysis is improved through load balance. Therefore, the invention provides a node load- |
---|