GSPMD: General and Scalable Parallelization for ML Computation Graphs
We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations. It allows users to write programs in the same way as for a single device, then give hints through a few annotations on how to distribute tensors, based on which GSPMD will parallelize the...
Gespeichert in:
| Veröffentlicht in: | arXiv.org 2021-12 |
|---|---|
| Hauptverfasser: | , , , , , , , , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | arXiv.org |
| container_volume | |
| creator | Xu, Yuanzhong Lee, HyoukJoong Chen, Dehao Hechtman, Blake Huang, Yanping Joshi, Rahul Krikun, Maxim Lepikhin, Dmitry Ly, Andy Maggioni, Marcello Pang, Ruoming Shazeer, Noam Wang, Shibo Wang, Tao Wu, Yonghui Chen, Zhifeng |
| description | We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations. It allows users to write programs in the same way as for a single device, then give hints through a few annotations on how to distribute tensors, based on which GSPMD will parallelize the computation. Its representation of partitioning is simple yet general, allowing it to express different or mixed paradigms of parallelism on a wide variety of models. GSPMD infers the partitioning for every operator based on limited user annotations, making it convenient to scale existing single-device programs. It solves several technical challenges for production usage, allowing GSPMD to achieve 50% to 62% compute utilization on up to 2048 Cloud TPUv3 cores for models with up to one trillion parameters. |
| format | Article |
| fullrecord | <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2525908537</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2525908537</sourcerecordid><originalsourceid>FETCH-proquest_journals_25259085373</originalsourceid><addsrcrecordid>eNqNisEKgkAUAJcgSMp_eNBZ2N62aV3N7JAQ2F1etZKy7dquXvr6hPqATgMzM2EBCrGKkjXijIXet5xz3MQopQhYlpfnYr-DXBnlSAOZO5Q30nTVCs40Kq1086a-sQZq66A4QWqf3dB_Ve6oe_gFm9akvQp_nLPlIbukx6hz9jUo31etHZwZU4US5ZYnUsTiv-sD2_Y55g</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2525908537</pqid></control><display><type>article</type><title>GSPMD: General and Scalable Parallelization for ML Computation Graphs</title><source>Free E- Journals</source><creator>Xu, Yuanzhong ; Lee, HyoukJoong ; Chen, Dehao ; Hechtman, Blake ; Huang, Yanping ; Joshi, Rahul ; Krikun, Maxim ; Lepikhin, Dmitry ; Ly, Andy ; Maggioni, Marcello ; Pang, Ruoming ; Shazeer, Noam ; Wang, Shibo ; Wang, Tao ; Wu, Yonghui ; Chen, Zhifeng</creator><creatorcontrib>Xu, Yuanzhong ; Lee, HyoukJoong ; Chen, Dehao ; Hechtman, Blake ; Huang, Yanping ; Joshi, Rahul ; Krikun, Maxim ; Lepikhin, Dmitry ; Ly, Andy ; Maggioni, Marcello ; Pang, Ruoming ; Shazeer, Noam ; Wang, Shibo ; Wang, Tao ; Wu, Yonghui ; Chen, Zhifeng</creatorcontrib><description>We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations. It allows users to write programs in the same way as for a single device, then give hints through a few annotations on how to distribute tensors, based on which GSPMD will parallelize the computation. Its representation of partitioning is simple yet general, allowing it to express different or mixed paradigms of parallelism on a wide variety of models. GSPMD infers the partitioning for every operator based on limited user annotations, making it convenient to scale existing single-device programs. It solves several technical challenges for production usage, allowing GSPMD to achieve 50% to 62% compute utilization on up to 2048 Cloud TPUv3 cores for models with up to one trillion parameters.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Annotations ; Computation ; Graphs ; Machine learning ; Parallel processing ; Partitioning ; Run time (computers) ; Tensors</subject><ispartof>arXiv.org, 2021-12</ispartof><rights>2021. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>777,781</link.rule.ids></links><search><creatorcontrib>Xu, Yuanzhong</creatorcontrib><creatorcontrib>Lee, HyoukJoong</creatorcontrib><creatorcontrib>Chen, Dehao</creatorcontrib><creatorcontrib>Hechtman, Blake</creatorcontrib><creatorcontrib>Huang, Yanping</creatorcontrib><creatorcontrib>Joshi, Rahul</creatorcontrib><creatorcontrib>Krikun, Maxim</creatorcontrib><creatorcontrib>Lepikhin, Dmitry</creatorcontrib><creatorcontrib>Ly, Andy</creatorcontrib><creatorcontrib>Maggioni, Marcello</creatorcontrib><creatorcontrib>Pang, Ruoming</creatorcontrib><creatorcontrib>Shazeer, Noam</creatorcontrib><creatorcontrib>Wang, Shibo</creatorcontrib><creatorcontrib>Wang, Tao</creatorcontrib><creatorcontrib>Wu, Yonghui</creatorcontrib><creatorcontrib>Chen, Zhifeng</creatorcontrib><title>GSPMD: General and Scalable Parallelization for ML Computation Graphs</title><title>arXiv.org</title><description>We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations. It allows users to write programs in the same way as for a single device, then give hints through a few annotations on how to distribute tensors, based on which GSPMD will parallelize the computation. Its representation of partitioning is simple yet general, allowing it to express different or mixed paradigms of parallelism on a wide variety of models. GSPMD infers the partitioning for every operator based on limited user annotations, making it convenient to scale existing single-device programs. It solves several technical challenges for production usage, allowing GSPMD to achieve 50% to 62% compute utilization on up to 2048 Cloud TPUv3 cores for models with up to one trillion parameters.</description><subject>Annotations</subject><subject>Computation</subject><subject>Graphs</subject><subject>Machine learning</subject><subject>Parallel processing</subject><subject>Partitioning</subject><subject>Run time (computers)</subject><subject>Tensors</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNqNisEKgkAUAJcgSMp_eNBZ2N62aV3N7JAQ2F1etZKy7dquXvr6hPqATgMzM2EBCrGKkjXijIXet5xz3MQopQhYlpfnYr-DXBnlSAOZO5Q30nTVCs40Kq1086a-sQZq66A4QWqf3dB_Ve6oe_gFm9akvQp_nLPlIbukx6hz9jUo31etHZwZU4US5ZYnUsTiv-sD2_Y55g</recordid><startdate>20211223</startdate><enddate>20211223</enddate><creator>Xu, Yuanzhong</creator><creator>Lee, HyoukJoong</creator><creator>Chen, Dehao</creator><creator>Hechtman, Blake</creator><creator>Huang, Yanping</creator><creator>Joshi, Rahul</creator><creator>Krikun, Maxim</creator><creator>Lepikhin, Dmitry</creator><creator>Ly, Andy</creator><creator>Maggioni, Marcello</creator><creator>Pang, Ruoming</creator><creator>Shazeer, Noam</creator><creator>Wang, Shibo</creator><creator>Wang, Tao</creator><creator>Wu, Yonghui</creator><creator>Chen, Zhifeng</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20211223</creationdate><title>GSPMD: General and Scalable Parallelization for ML Computation Graphs</title><author>Xu, Yuanzhong ; Lee, HyoukJoong ; Chen, Dehao ; Hechtman, Blake ; Huang, Yanping ; Joshi, Rahul ; Krikun, Maxim ; Lepikhin, Dmitry ; Ly, Andy ; Maggioni, Marcello ; Pang, Ruoming ; Shazeer, Noam ; Wang, Shibo ; Wang, Tao ; Wu, Yonghui ; Chen, Zhifeng</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_25259085373</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Annotations</topic><topic>Computation</topic><topic>Graphs</topic><topic>Machine learning</topic><topic>Parallel processing</topic><topic>Partitioning</topic><topic>Run time (computers)</topic><topic>Tensors</topic><toplevel>online_resources</toplevel><creatorcontrib>Xu, Yuanzhong</creatorcontrib><creatorcontrib>Lee, HyoukJoong</creatorcontrib><creatorcontrib>Chen, Dehao</creatorcontrib><creatorcontrib>Hechtman, Blake</creatorcontrib><creatorcontrib>Huang, Yanping</creatorcontrib><creatorcontrib>Joshi, Rahul</creatorcontrib><creatorcontrib>Krikun, Maxim</creatorcontrib><creatorcontrib>Lepikhin, Dmitry</creatorcontrib><creatorcontrib>Ly, Andy</creatorcontrib><creatorcontrib>Maggioni, Marcello</creatorcontrib><creatorcontrib>Pang, Ruoming</creatorcontrib><creatorcontrib>Shazeer, Noam</creatorcontrib><creatorcontrib>Wang, Shibo</creatorcontrib><creatorcontrib>Wang, Tao</creatorcontrib><creatorcontrib>Wu, Yonghui</creatorcontrib><creatorcontrib>Chen, Zhifeng</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Xu, Yuanzhong</au><au>Lee, HyoukJoong</au><au>Chen, Dehao</au><au>Hechtman, Blake</au><au>Huang, Yanping</au><au>Joshi, Rahul</au><au>Krikun, Maxim</au><au>Lepikhin, Dmitry</au><au>Ly, Andy</au><au>Maggioni, Marcello</au><au>Pang, Ruoming</au><au>Shazeer, Noam</au><au>Wang, Shibo</au><au>Wang, Tao</au><au>Wu, Yonghui</au><au>Chen, Zhifeng</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>GSPMD: General and Scalable Parallelization for ML Computation Graphs</atitle><jtitle>arXiv.org</jtitle><date>2021-12-23</date><risdate>2021</risdate><eissn>2331-8422</eissn><abstract>We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations. It allows users to write programs in the same way as for a single device, then give hints through a few annotations on how to distribute tensors, based on which GSPMD will parallelize the computation. Its representation of partitioning is simple yet general, allowing it to express different or mixed paradigms of parallelism on a wide variety of models. GSPMD infers the partitioning for every operator based on limited user annotations, making it convenient to scale existing single-device programs. It solves several technical challenges for production usage, allowing GSPMD to achieve 50% to 62% compute utilization on up to 2048 Cloud TPUv3 cores for models with up to one trillion parameters.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | EISSN: 2331-8422 |
| ispartof | arXiv.org, 2021-12 |
| issn | 2331-8422 |
| language | eng |
| recordid | cdi_proquest_journals_2525908537 |
| source | Free E- Journals |
| subjects | Annotations Computation Graphs Machine learning Parallel processing Partitioning Run time (computers) Tensors |
| title | GSPMD: General and Scalable Parallelization for ML Computation Graphs |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-20T17%3A05%3A52IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=GSPMD:%20General%20and%20Scalable%20Parallelization%20for%20ML%20Computation%20Graphs&rft.jtitle=arXiv.org&rft.au=Xu,%20Yuanzhong&rft.date=2021-12-23&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2525908537%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2525908537&rft_id=info:pmid/&rfr_iscdi=true |