Method for reducing memory contention applied to Spark computing platform

The invention provides a method for reducing memory contention applied to a Spark computing platform. The method comprises the following steps: 1, calculating values of all persistent RDDs in a memory, and sorting all the RDDs according to the values to form a priority queue; step 2, if the memory s...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: WANG JINJIANG, ZHAO ZIMING, LI BOHAN, PAN SHUNJIE, YU JUNYANG, LI SHAOHUA, SONG YIXIN
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention provides a method for reducing memory contention applied to a Spark computing platform. The method comprises the following steps: 1, calculating values of all persistent RDDs in a memory, and sorting all the RDDs according to the values to form a priority queue; step 2, if the memory state is monitored to know that the memory reaches a bottleneck, executing step 3; step 3, circularly releasing the RDD with lower value in the priority queue; 4, after the current cyclic release operation is carried out, if it is judged that an idle processor core exists, the step 3 continues to be executed, and otherwise, the step 5 is executed; 5, calculating a memory persistence acceleration utility when the current RDD needing to be persisted is persisted into the memory and a hard disk persistence acceleration utility when the current RDD needing to be persisted is persisted into the hard disk; and 6, based on the memory persistence acceleration utility and the hard disk persistence acceleration utility, solvi