Method for reducing memory contention applied to Spark computing platform
The invention provides a method for reducing memory contention applied to a Spark computing platform. The method comprises the following steps: 1, calculating values of all persistent RDDs in a memory, and sorting all the RDDs according to the values to form a priority queue; step 2, if the memory s...
Gespeichert in:
Hauptverfasser: | , , , , , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention provides a method for reducing memory contention applied to a Spark computing platform. The method comprises the following steps: 1, calculating values of all persistent RDDs in a memory, and sorting all the RDDs according to the values to form a priority queue; step 2, if the memory state is monitored to know that the memory reaches a bottleneck, executing step 3; step 3, circularly releasing the RDD with lower value in the priority queue; 4, after the current cyclic release operation is carried out, if it is judged that an idle processor core exists, the step 3 continues to be executed, and otherwise, the step 5 is executed; 5, calculating a memory persistence acceleration utility when the current RDD needing to be persisted is persisted into the memory and a hard disk persistence acceleration utility when the current RDD needing to be persisted is persisted into the hard disk; and 6, based on the memory persistence acceleration utility and the hard disk persistence acceleration utility, solvi |
---|