Data scheduling strategy based on self-adaptive reserved memory in Spark environment

The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the m...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: ZHAO YUXI, LI BOHAN, PAN SHUNJIE, XIN ZHIYI, HE YIRU, GU HANGYU, HE XIN
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre