Data dynamic partitioning system based on node load

The invention provides a data dynamic partitioning system based on a node load. The data dynamic partitioning system comprises a load monitoring acquisition module, a prediction module, a data pre-partitioning module, a data migration module and the like, the node load is predicted by adopting a sec...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: JIANG JINING, LIU ZHE, WU JIAYI, LI HUIKE, HE CHENGLONG, GU XUEHAI, MENG LINGWU, DING CAN, CHEN ZHENG
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention provides a data dynamic partitioning system based on a node load. The data dynamic partitioning system comprises a load monitoring acquisition module, a prediction module, a data pre-partitioning module, a data migration module and the like, the node load is predicted by adopting a secondary smoothing method, the AHP and entropy index weight method is combined, corresponding partitioning strategies can be obtained according to different data analysis applications, the load balance of the system is dynamically adjusted, and the response speed of the applications is increased. Thesystem aims at an application scene of a Spark and Messql distributed integrated framework, due to heterogeneity of node resources existing in a distributed environment, in order to reduce data transmission consumption between nodes, computing resources of each node are fully utilized, and parallel computing efficiency of application analysis is improved through load balance. Therefore, the invention provides a node load-