Data scheduling strategy based on self-adaptive reserved memory in Spark environment
The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the m...
Gespeichert in:
Hauptverfasser: | , , , , , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre |
---|