Distributed big data parallel computing method based on Hadoop MapReduce
The invention relates to a distributed big data parallel computing method based on Hadoop MapReduce. The distributed big data parallel computing method comprises Map, Shuffle, and Reduce steps of a Hadoop framework, wherein a GPU computing module is added between a Hadoop MapReduce framework and a u...
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Patent |
| Sprache: | chi ; eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | |
| container_volume | |
| creator | LI PENG DING GANGYI HUANG TIANYU MAO XUKUN |
| description | The invention relates to a distributed big data parallel computing method based on Hadoop MapReduce. The distributed big data parallel computing method comprises Map, Shuffle, and Reduce steps of a Hadoop framework, wherein a GPU computing module is added between a Hadoop MapReduce framework and a user; the user submits a specific Map function and a specific Reduce function to a GPU computing module, and the GPU computing module processes a whole data block distributed by a working node as a value of a key value pair through an interface provided by Hadoop before the Map step; in the Map step,the GPU computing module packages the Map function submitted by the user into a new Map function and submits the new Map function to the Hadoop framework; and the new Map function receives the data block from the Hadoop framework, and key value pairs are further divided, and each key value pair is allocated to different GPU threads, and each GPU thread calls the Map function submitted by the userfor parallel computing. A |
| format | Patent |
| fullrecord | <record><control><sourceid>epo_EVB</sourceid><recordid>TN_cdi_epo_espacenet_CN110187970A</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>CN110187970A</sourcerecordid><originalsourceid>FETCH-epo_espacenet_CN110187970A3</originalsourceid><addsrcrecordid>eNqNyjEOwjAMAMAsDAj4g3kAUiOGwogKKAsMiL1yG7dESmOrcf5PBx7AdMutjbuGrHPoipKHLozgUREEZ4yRIvQ8SdGQRphIP7wUzEvkBA49s8AD5UW-9LQ1qwFjpt3Pjdnfb-_GHUi4pSzYUyJtm6e1lT3V57q6HP85X4OHM_M</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>patent</recordtype></control><display><type>patent</type><title>Distributed big data parallel computing method based on Hadoop MapReduce</title><source>esp@cenet</source><creator>LI PENG ; DING GANGYI ; HUANG TIANYU ; MAO XUKUN</creator><creatorcontrib>LI PENG ; DING GANGYI ; HUANG TIANYU ; MAO XUKUN</creatorcontrib><description>The invention relates to a distributed big data parallel computing method based on Hadoop MapReduce. The distributed big data parallel computing method comprises Map, Shuffle, and Reduce steps of a Hadoop framework, wherein a GPU computing module is added between a Hadoop MapReduce framework and a user; the user submits a specific Map function and a specific Reduce function to a GPU computing module, and the GPU computing module processes a whole data block distributed by a working node as a value of a key value pair through an interface provided by Hadoop before the Map step; in the Map step,the GPU computing module packages the Map function submitted by the user into a new Map function and submits the new Map function to the Hadoop framework; and the new Map function receives the data block from the Hadoop framework, and key value pairs are further divided, and each key value pair is allocated to different GPU threads, and each GPU thread calls the Map function submitted by the userfor parallel computing. A</description><language>chi ; eng</language><subject>CALCULATING ; COMPUTING ; COUNTING ; ELECTRIC DIGITAL DATA PROCESSING ; PHYSICS</subject><creationdate>2019</creationdate><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20190830&DB=EPODOC&CC=CN&NR=110187970A$$EHTML$$P50$$Gepo$$Hfree_for_read</linktohtml><link.rule.ids>230,308,780,885,25564,76547</link.rule.ids><linktorsrc>$$Uhttps://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20190830&DB=EPODOC&CC=CN&NR=110187970A$$EView_record_in_European_Patent_Office$$FView_record_in_$$GEuropean_Patent_Office$$Hfree_for_read</linktorsrc></links><search><creatorcontrib>LI PENG</creatorcontrib><creatorcontrib>DING GANGYI</creatorcontrib><creatorcontrib>HUANG TIANYU</creatorcontrib><creatorcontrib>MAO XUKUN</creatorcontrib><title>Distributed big data parallel computing method based on Hadoop MapReduce</title><description>The invention relates to a distributed big data parallel computing method based on Hadoop MapReduce. The distributed big data parallel computing method comprises Map, Shuffle, and Reduce steps of a Hadoop framework, wherein a GPU computing module is added between a Hadoop MapReduce framework and a user; the user submits a specific Map function and a specific Reduce function to a GPU computing module, and the GPU computing module processes a whole data block distributed by a working node as a value of a key value pair through an interface provided by Hadoop before the Map step; in the Map step,the GPU computing module packages the Map function submitted by the user into a new Map function and submits the new Map function to the Hadoop framework; and the new Map function receives the data block from the Hadoop framework, and key value pairs are further divided, and each key value pair is allocated to different GPU threads, and each GPU thread calls the Map function submitted by the userfor parallel computing. A</description><subject>CALCULATING</subject><subject>COMPUTING</subject><subject>COUNTING</subject><subject>ELECTRIC DIGITAL DATA PROCESSING</subject><subject>PHYSICS</subject><fulltext>true</fulltext><rsrctype>patent</rsrctype><creationdate>2019</creationdate><recordtype>patent</recordtype><sourceid>EVB</sourceid><recordid>eNqNyjEOwjAMAMAsDAj4g3kAUiOGwogKKAsMiL1yG7dESmOrcf5PBx7AdMutjbuGrHPoipKHLozgUREEZ4yRIvQ8SdGQRphIP7wUzEvkBA49s8AD5UW-9LQ1qwFjpt3Pjdnfb-_GHUi4pSzYUyJtm6e1lT3V57q6HP85X4OHM_M</recordid><startdate>20190830</startdate><enddate>20190830</enddate><creator>LI PENG</creator><creator>DING GANGYI</creator><creator>HUANG TIANYU</creator><creator>MAO XUKUN</creator><scope>EVB</scope></search><sort><creationdate>20190830</creationdate><title>Distributed big data parallel computing method based on Hadoop MapReduce</title><author>LI PENG ; DING GANGYI ; HUANG TIANYU ; MAO XUKUN</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-epo_espacenet_CN110187970A3</frbrgroupid><rsrctype>patents</rsrctype><prefilter>patents</prefilter><language>chi ; eng</language><creationdate>2019</creationdate><topic>CALCULATING</topic><topic>COMPUTING</topic><topic>COUNTING</topic><topic>ELECTRIC DIGITAL DATA PROCESSING</topic><topic>PHYSICS</topic><toplevel>online_resources</toplevel><creatorcontrib>LI PENG</creatorcontrib><creatorcontrib>DING GANGYI</creatorcontrib><creatorcontrib>HUANG TIANYU</creatorcontrib><creatorcontrib>MAO XUKUN</creatorcontrib><collection>esp@cenet</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>LI PENG</au><au>DING GANGYI</au><au>HUANG TIANYU</au><au>MAO XUKUN</au><format>patent</format><genre>patent</genre><ristype>GEN</ristype><title>Distributed big data parallel computing method based on Hadoop MapReduce</title><date>2019-08-30</date><risdate>2019</risdate><abstract>The invention relates to a distributed big data parallel computing method based on Hadoop MapReduce. The distributed big data parallel computing method comprises Map, Shuffle, and Reduce steps of a Hadoop framework, wherein a GPU computing module is added between a Hadoop MapReduce framework and a user; the user submits a specific Map function and a specific Reduce function to a GPU computing module, and the GPU computing module processes a whole data block distributed by a working node as a value of a key value pair through an interface provided by Hadoop before the Map step; in the Map step,the GPU computing module packages the Map function submitted by the user into a new Map function and submits the new Map function to the Hadoop framework; and the new Map function receives the data block from the Hadoop framework, and key value pairs are further divided, and each key value pair is allocated to different GPU threads, and each GPU thread calls the Map function submitted by the userfor parallel computing. A</abstract><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | |
| ispartof | |
| issn | |
| language | chi ; eng |
| recordid | cdi_epo_espacenet_CN110187970A |
| source | esp@cenet |
| subjects | CALCULATING COMPUTING COUNTING ELECTRIC DIGITAL DATA PROCESSING PHYSICS |
| title | Distributed big data parallel computing method based on Hadoop MapReduce |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-03T12%3A23%3A23IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-epo_EVB&rft_val_fmt=info:ofi/fmt:kev:mtx:patent&rft.genre=patent&rft.au=LI%20PENG&rft.date=2019-08-30&rft_id=info:doi/&rft_dat=%3Cepo_EVB%3ECN110187970A%3C/epo_EVB%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |