Smart reduce task scheduler
A system and a method for scheduling a reduce task on nodes is disclosed. The various nodes in a cluster of nodes are bucketized into intermediate data items. A counter is created that provides a count of the intermediate data items that are placed into the each of the buckets for the node. This cou...
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Patent |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | |
| container_volume | |
| creator | Wang, Ting Liu, Yang Yang, ZhiZhi Sui, Guang Han Gui, HaoChen |
| description | A system and a method for scheduling a reduce task on nodes is disclosed. The various nodes in a cluster of nodes are bucketized into intermediate data items. A counter is created that provides a count of the intermediate data items that are placed into the each of the buckets for the node. This counter value is provided to a scheduler. From the counter information the scheduler is able to determine the cost of moving the intermediate data for the bucket to different ones of the nodes. Once the cost of moving the intermediate data is determined the scheduler is able to determine which of the nodes should perform the reduce task for that particular bucket. The scheduler minimizes the amount of shuffling of the intermediate data between the nodes for each of the buckets, by determining the lowest cost shuffle option for each of the buckets. |
| format | Patent |
| fullrecord | <record><control><sourceid>epo_EVB</sourceid><recordid>TN_cdi_epo_espacenet_US10360065B2</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>US10360065B2</sourcerecordid><originalsourceid>FETCH-epo_espacenet_US10360065B23</originalsourceid><addsrcrecordid>eNrjZJAOzk0sKlEoSk0pTU5VKEkszlYoTs4A8nJSi3gYWNMSc4pTeaE0N4Oim2uIs4duakF-fGpxQWJyal5qSXxosKGBsZmBgZmpk5ExMWoAcfgjcQ</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>patent</recordtype></control><display><type>patent</type><title>Smart reduce task scheduler</title><source>esp@cenet</source><creator>Wang, Ting ; Liu, Yang ; Yang, ZhiZhi ; Sui, Guang Han ; Gui, HaoChen</creator><creatorcontrib>Wang, Ting ; Liu, Yang ; Yang, ZhiZhi ; Sui, Guang Han ; Gui, HaoChen</creatorcontrib><description>A system and a method for scheduling a reduce task on nodes is disclosed. The various nodes in a cluster of nodes are bucketized into intermediate data items. A counter is created that provides a count of the intermediate data items that are placed into the each of the buckets for the node. This counter value is provided to a scheduler. From the counter information the scheduler is able to determine the cost of moving the intermediate data for the bucket to different ones of the nodes. Once the cost of moving the intermediate data is determined the scheduler is able to determine which of the nodes should perform the reduce task for that particular bucket. The scheduler minimizes the amount of shuffling of the intermediate data between the nodes for each of the buckets, by determining the lowest cost shuffle option for each of the buckets.</description><language>eng</language><subject>CALCULATING ; COMPUTING ; COUNTING ; ELECTRIC DIGITAL DATA PROCESSING ; PHYSICS</subject><creationdate>2019</creationdate><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20190723&DB=EPODOC&CC=US&NR=10360065B2$$EHTML$$P50$$Gepo$$Hfree_for_read</linktohtml><link.rule.ids>230,308,776,881,25542,76290</link.rule.ids><linktorsrc>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20190723&DB=EPODOC&CC=US&NR=10360065B2$$EView_record_in_European_Patent_Office$$FView_record_in_$$GEuropean_Patent_Office$$Hfree_for_read</linktorsrc></links><search><creatorcontrib>Wang, Ting</creatorcontrib><creatorcontrib>Liu, Yang</creatorcontrib><creatorcontrib>Yang, ZhiZhi</creatorcontrib><creatorcontrib>Sui, Guang Han</creatorcontrib><creatorcontrib>Gui, HaoChen</creatorcontrib><title>Smart reduce task scheduler</title><description>A system and a method for scheduling a reduce task on nodes is disclosed. The various nodes in a cluster of nodes are bucketized into intermediate data items. A counter is created that provides a count of the intermediate data items that are placed into the each of the buckets for the node. This counter value is provided to a scheduler. From the counter information the scheduler is able to determine the cost of moving the intermediate data for the bucket to different ones of the nodes. Once the cost of moving the intermediate data is determined the scheduler is able to determine which of the nodes should perform the reduce task for that particular bucket. The scheduler minimizes the amount of shuffling of the intermediate data between the nodes for each of the buckets, by determining the lowest cost shuffle option for each of the buckets.</description><subject>CALCULATING</subject><subject>COMPUTING</subject><subject>COUNTING</subject><subject>ELECTRIC DIGITAL DATA PROCESSING</subject><subject>PHYSICS</subject><fulltext>true</fulltext><rsrctype>patent</rsrctype><creationdate>2019</creationdate><recordtype>patent</recordtype><sourceid>EVB</sourceid><recordid>eNrjZJAOzk0sKlEoSk0pTU5VKEkszlYoTs4A8nJSi3gYWNMSc4pTeaE0N4Oim2uIs4duakF-fGpxQWJyal5qSXxosKGBsZmBgZmpk5ExMWoAcfgjcQ</recordid><startdate>20190723</startdate><enddate>20190723</enddate><creator>Wang, Ting</creator><creator>Liu, Yang</creator><creator>Yang, ZhiZhi</creator><creator>Sui, Guang Han</creator><creator>Gui, HaoChen</creator><scope>EVB</scope></search><sort><creationdate>20190723</creationdate><title>Smart reduce task scheduler</title><author>Wang, Ting ; Liu, Yang ; Yang, ZhiZhi ; Sui, Guang Han ; Gui, HaoChen</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-epo_espacenet_US10360065B23</frbrgroupid><rsrctype>patents</rsrctype><prefilter>patents</prefilter><language>eng</language><creationdate>2019</creationdate><topic>CALCULATING</topic><topic>COMPUTING</topic><topic>COUNTING</topic><topic>ELECTRIC DIGITAL DATA PROCESSING</topic><topic>PHYSICS</topic><toplevel>online_resources</toplevel><creatorcontrib>Wang, Ting</creatorcontrib><creatorcontrib>Liu, Yang</creatorcontrib><creatorcontrib>Yang, ZhiZhi</creatorcontrib><creatorcontrib>Sui, Guang Han</creatorcontrib><creatorcontrib>Gui, HaoChen</creatorcontrib><collection>esp@cenet</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Wang, Ting</au><au>Liu, Yang</au><au>Yang, ZhiZhi</au><au>Sui, Guang Han</au><au>Gui, HaoChen</au><format>patent</format><genre>patent</genre><ristype>GEN</ristype><title>Smart reduce task scheduler</title><date>2019-07-23</date><risdate>2019</risdate><abstract>A system and a method for scheduling a reduce task on nodes is disclosed. The various nodes in a cluster of nodes are bucketized into intermediate data items. A counter is created that provides a count of the intermediate data items that are placed into the each of the buckets for the node. This counter value is provided to a scheduler. From the counter information the scheduler is able to determine the cost of moving the intermediate data for the bucket to different ones of the nodes. Once the cost of moving the intermediate data is determined the scheduler is able to determine which of the nodes should perform the reduce task for that particular bucket. The scheduler minimizes the amount of shuffling of the intermediate data between the nodes for each of the buckets, by determining the lowest cost shuffle option for each of the buckets.</abstract><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | |
| ispartof | |
| issn | |
| language | eng |
| recordid | cdi_epo_espacenet_US10360065B2 |
| source | esp@cenet |
| subjects | CALCULATING COMPUTING COUNTING ELECTRIC DIGITAL DATA PROCESSING PHYSICS |
| title | Smart reduce task scheduler |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-28T18%3A56%3A15IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-epo_EVB&rft_val_fmt=info:ofi/fmt:kev:mtx:patent&rft.genre=patent&rft.au=Wang,%20Ting&rft.date=2019-07-23&rft_id=info:doi/&rft_dat=%3Cepo_EVB%3EUS10360065B2%3C/epo_EVB%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |