Data scheduling strategy based on self-adaptive reserved memory in Spark environment
The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the m...
Gespeichert in:
Hauptverfasser: | , , , , , , |
---|---|
Format: | Patent |
Sprache: | chi ; eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | |
container_volume | |
creator | ZHAO YUXI LI BOHAN PAN SHUNJIE XIN ZHIYI HE YIRU GU HANGYU HE XIN |
description | The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre |
format | Patent |
fullrecord | <record><control><sourceid>epo_EVB</sourceid><recordid>TN_cdi_epo_espacenet_CN116700988A</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>CN116700988A</sourcerecordid><originalsourceid>FETCH-epo_espacenet_CN116700988A3</originalsourceid><addsrcrecordid>eNqNyjEOgkAQRmEaC6PeYTwACcREoTSosbKRnozwgxuX2c3OSsLttfAAVq_43jKpTxyZtH2ie1sjA2kMHDHM9GBFR05IYfuUO_bRTKAARZi-MmJ0YSYjdPccXgSZTHAyQuI6WfRsFZtfV8n2cq6rawrvGqjnFoLYVLc83x-yrCyK4-6f5wOzPjkl</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>patent</recordtype></control><display><type>patent</type><title>Data scheduling strategy based on self-adaptive reserved memory in Spark environment</title><source>esp@cenet</source><creator>ZHAO YUXI ; LI BOHAN ; PAN SHUNJIE ; XIN ZHIYI ; HE YIRU ; GU HANGYU ; HE XIN</creator><creatorcontrib>ZHAO YUXI ; LI BOHAN ; PAN SHUNJIE ; XIN ZHIYI ; HE YIRU ; GU HANGYU ; HE XIN</creatorcontrib><description>The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre</description><language>chi ; eng</language><subject>CALCULATING ; COMPUTING ; COUNTING ; ELECTRIC DIGITAL DATA PROCESSING ; PHYSICS</subject><creationdate>2023</creationdate><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20230905&DB=EPODOC&CC=CN&NR=116700988A$$EHTML$$P50$$Gepo$$Hfree_for_read</linktohtml><link.rule.ids>230,308,780,885,25563,76318</link.rule.ids><linktorsrc>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20230905&DB=EPODOC&CC=CN&NR=116700988A$$EView_record_in_European_Patent_Office$$FView_record_in_$$GEuropean_Patent_Office$$Hfree_for_read</linktorsrc></links><search><creatorcontrib>ZHAO YUXI</creatorcontrib><creatorcontrib>LI BOHAN</creatorcontrib><creatorcontrib>PAN SHUNJIE</creatorcontrib><creatorcontrib>XIN ZHIYI</creatorcontrib><creatorcontrib>HE YIRU</creatorcontrib><creatorcontrib>GU HANGYU</creatorcontrib><creatorcontrib>HE XIN</creatorcontrib><title>Data scheduling strategy based on self-adaptive reserved memory in Spark environment</title><description>The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre</description><subject>CALCULATING</subject><subject>COMPUTING</subject><subject>COUNTING</subject><subject>ELECTRIC DIGITAL DATA PROCESSING</subject><subject>PHYSICS</subject><fulltext>true</fulltext><rsrctype>patent</rsrctype><creationdate>2023</creationdate><recordtype>patent</recordtype><sourceid>EVB</sourceid><recordid>eNqNyjEOgkAQRmEaC6PeYTwACcREoTSosbKRnozwgxuX2c3OSsLttfAAVq_43jKpTxyZtH2ie1sjA2kMHDHM9GBFR05IYfuUO_bRTKAARZi-MmJ0YSYjdPccXgSZTHAyQuI6WfRsFZtfV8n2cq6rawrvGqjnFoLYVLc83x-yrCyK4-6f5wOzPjkl</recordid><startdate>20230905</startdate><enddate>20230905</enddate><creator>ZHAO YUXI</creator><creator>LI BOHAN</creator><creator>PAN SHUNJIE</creator><creator>XIN ZHIYI</creator><creator>HE YIRU</creator><creator>GU HANGYU</creator><creator>HE XIN</creator><scope>EVB</scope></search><sort><creationdate>20230905</creationdate><title>Data scheduling strategy based on self-adaptive reserved memory in Spark environment</title><author>ZHAO YUXI ; LI BOHAN ; PAN SHUNJIE ; XIN ZHIYI ; HE YIRU ; GU HANGYU ; HE XIN</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-epo_espacenet_CN116700988A3</frbrgroupid><rsrctype>patents</rsrctype><prefilter>patents</prefilter><language>chi ; eng</language><creationdate>2023</creationdate><topic>CALCULATING</topic><topic>COMPUTING</topic><topic>COUNTING</topic><topic>ELECTRIC DIGITAL DATA PROCESSING</topic><topic>PHYSICS</topic><toplevel>online_resources</toplevel><creatorcontrib>ZHAO YUXI</creatorcontrib><creatorcontrib>LI BOHAN</creatorcontrib><creatorcontrib>PAN SHUNJIE</creatorcontrib><creatorcontrib>XIN ZHIYI</creatorcontrib><creatorcontrib>HE YIRU</creatorcontrib><creatorcontrib>GU HANGYU</creatorcontrib><creatorcontrib>HE XIN</creatorcontrib><collection>esp@cenet</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>ZHAO YUXI</au><au>LI BOHAN</au><au>PAN SHUNJIE</au><au>XIN ZHIYI</au><au>HE YIRU</au><au>GU HANGYU</au><au>HE XIN</au><format>patent</format><genre>patent</genre><ristype>GEN</ristype><title>Data scheduling strategy based on self-adaptive reserved memory in Spark environment</title><date>2023-09-05</date><risdate>2023</risdate><abstract>The invention discloses a data scheduling strategy based on a self-adaptive reserved memory in a Spark environment, which comprises the following steps of: firstly, judging whether a memory space meets an execution condition of a memory with an optimal execution task parallelism degree, and if the memory space meets the execution condition and a task is found to be blocked, redistributing the execution memory and preferentially ensuring the task parallelism degree; secondly, if the tasks in the memory run normally, a self-adaptive adjustment algorithm of the memory is triggered, and redundant memory is recycled; and finally, controlling the RDD to select a storage position with lower calculation cost according to the size of the allocation space of the memory so as to make enough space to preferentially guarantee the space reservation of the execution memory. According to the method, the execution memory space is adaptively reserved, the distributed execution memory space is selected to be compressed or incre</abstract><oa>free_for_read</oa></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | |
ispartof | |
issn | |
language | chi ; eng |
recordid | cdi_epo_espacenet_CN116700988A |
source | esp@cenet |
subjects | CALCULATING COMPUTING COUNTING ELECTRIC DIGITAL DATA PROCESSING PHYSICS |
title | Data scheduling strategy based on self-adaptive reserved memory in Spark environment |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-10T23%3A36%3A27IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-epo_EVB&rft_val_fmt=info:ofi/fmt:kev:mtx:patent&rft.genre=patent&rft.au=ZHAO%20YUXI&rft.date=2023-09-05&rft_id=info:doi/&rft_dat=%3Cepo_EVB%3ECN116700988A%3C/epo_EVB%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |