Data scheduling strategy based on self-adaptive reserved memory in Spark environment

The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the m...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: ZHAO YUXI, LI BOHAN, PAN SHUNJIE, XIN ZHIYI, HE YIRU, GU HANGYU, HE XIN
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator ZHAO YUXI
LI BOHAN
PAN SHUNJIE
XIN ZHIYI
HE YIRU
GU HANGYU
HE XIN
description The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre
format Patent
fullrecord <record><control><sourceid>epo_EVB</sourceid><recordid>TN_cdi_epo_espacenet_CN116700988A</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>CN116700988A</sourcerecordid><originalsourceid>FETCH-epo_espacenet_CN116700988A3</originalsourceid><addsrcrecordid>eNqNyjEOgkAQRmEaC6PeYTwACcREoTSosbKRnozwgxuX2c3OSsLttfAAVq_43jKpTxyZtH2ie1sjA2kMHDHM9GBFR05IYfuUO_bRTKAARZi-MmJ0YSYjdPccXgSZTHAyQuI6WfRsFZtfV8n2cq6rawrvGqjnFoLYVLc83x-yrCyK4-6f5wOzPjkl</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>patent</recordtype></control><display><type>patent</type><title>Data scheduling strategy based on self-adaptive reserved memory in Spark environment</title><source>esp@cenet</source><creator>ZHAO YUXI ; LI BOHAN ; PAN SHUNJIE ; XIN ZHIYI ; HE YIRU ; GU HANGYU ; HE XIN</creator><creatorcontrib>ZHAO YUXI ; LI BOHAN ; PAN SHUNJIE ; XIN ZHIYI ; HE YIRU ; GU HANGYU ; HE XIN</creatorcontrib><description>The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre</description><language>chi ; eng</language><subject>CALCULATING ; COMPUTING ; COUNTING ; ELECTRIC DIGITAL DATA PROCESSING ; PHYSICS</subject><creationdate>2023</creationdate><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&amp;date=20230905&amp;DB=EPODOC&amp;CC=CN&amp;NR=116700988A$$EHTML$$P50$$Gepo$$Hfree_for_read</linktohtml><link.rule.ids>230,308,780,885,25563,76318</link.rule.ids><linktorsrc>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&amp;date=20230905&amp;DB=EPODOC&amp;CC=CN&amp;NR=116700988A$$EView_record_in_European_Patent_Office$$FView_record_in_$$GEuropean_Patent_Office$$Hfree_for_read</linktorsrc></links><search><creatorcontrib>ZHAO YUXI</creatorcontrib><creatorcontrib>LI BOHAN</creatorcontrib><creatorcontrib>PAN SHUNJIE</creatorcontrib><creatorcontrib>XIN ZHIYI</creatorcontrib><creatorcontrib>HE YIRU</creatorcontrib><creatorcontrib>GU HANGYU</creatorcontrib><creatorcontrib>HE XIN</creatorcontrib><title>Data scheduling strategy based on self-adaptive reserved memory in Spark environment</title><description>The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre</description><subject>CALCULATING</subject><subject>COMPUTING</subject><subject>COUNTING</subject><subject>ELECTRIC DIGITAL DATA PROCESSING</subject><subject>PHYSICS</subject><fulltext>true</fulltext><rsrctype>patent</rsrctype><creationdate>2023</creationdate><recordtype>patent</recordtype><sourceid>EVB</sourceid><recordid>eNqNyjEOgkAQRmEaC6PeYTwACcREoTSosbKRnozwgxuX2c3OSsLttfAAVq_43jKpTxyZtH2ie1sjA2kMHDHM9GBFR05IYfuUO_bRTKAARZi-MmJ0YSYjdPccXgSZTHAyQuI6WfRsFZtfV8n2cq6rawrvGqjnFoLYVLc83x-yrCyK4-6f5wOzPjkl</recordid><startdate>20230905</startdate><enddate>20230905</enddate><creator>ZHAO YUXI</creator><creator>LI BOHAN</creator><creator>PAN SHUNJIE</creator><creator>XIN ZHIYI</creator><creator>HE YIRU</creator><creator>GU HANGYU</creator><creator>HE XIN</creator><scope>EVB</scope></search><sort><creationdate>20230905</creationdate><title>Data scheduling strategy based on self-adaptive reserved memory in Spark environment</title><author>ZHAO YUXI ; LI BOHAN ; PAN SHUNJIE ; XIN ZHIYI ; HE YIRU ; GU HANGYU ; HE XIN</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-epo_espacenet_CN116700988A3</frbrgroupid><rsrctype>patents</rsrctype><prefilter>patents</prefilter><language>chi ; eng</language><creationdate>2023</creationdate><topic>CALCULATING</topic><topic>COMPUTING</topic><topic>COUNTING</topic><topic>ELECTRIC DIGITAL DATA PROCESSING</topic><topic>PHYSICS</topic><toplevel>online_resources</toplevel><creatorcontrib>ZHAO YUXI</creatorcontrib><creatorcontrib>LI BOHAN</creatorcontrib><creatorcontrib>PAN SHUNJIE</creatorcontrib><creatorcontrib>XIN ZHIYI</creatorcontrib><creatorcontrib>HE YIRU</creatorcontrib><creatorcontrib>GU HANGYU</creatorcontrib><creatorcontrib>HE XIN</creatorcontrib><collection>esp@cenet</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>ZHAO YUXI</au><au>LI BOHAN</au><au>PAN SHUNJIE</au><au>XIN ZHIYI</au><au>HE YIRU</au><au>GU HANGYU</au><au>HE XIN</au><format>patent</format><genre>patent</genre><ristype>GEN</ristype><title>Data scheduling strategy based on self-adaptive reserved memory in Spark environment</title><date>2023-09-05</date><risdate>2023</risdate><abstract>The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre</abstract><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier
ispartof
issn
language chi ; eng
recordid cdi_epo_espacenet_CN116700988A
source esp@cenet
subjects CALCULATING
COMPUTING
COUNTING
ELECTRIC DIGITAL DATA PROCESSING
PHYSICS
title Data scheduling strategy based on self-adaptive reserved memory in Spark environment
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-10T23%3A36%3A27IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-epo_EVB&rft_val_fmt=info:ofi/fmt:kev:mtx:patent&rft.genre=patent&rft.au=ZHAO%20YUXI&rft.date=2023-09-05&rft_id=info:doi/&rft_dat=%3Cepo_EVB%3ECN116700988A%3C/epo_EVB%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true