Scale MLPerf-0.6 models on Google TPU-v3 Pods
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defines a suite of models, datasets and rules to follow when benchmarking to ensure results are comparable across hardware, f...
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | |
| container_volume | |
| creator | Kumar, Sameer Bitorff, Victor Chen, Dehao Chou, Chiachen Hechtman, Blake Lee, HyoukJoong Kumar, Naveen Mattson, Peter Wang, Shibo Wang, Tao Xu, Yuanzhong Zhou, Zongwei |
| description | The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6
training benchmark demonstrates the scalability of a suite of industry relevant
ML models. MLPerf defines a suite of models, datasets and rules to follow when
benchmarking to ensure results are comparable across hardware, frameworks and
companies. Using this suite of models, we discuss the optimizations and
techniques including choice of optimizer, spatial partitioning and weight
update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify
properties of models that make scaling them challenging, such as limited data
parallelism and unscaled weights. These optimizations contribute to record
performance in transformer, Resnet-50 and SSD in the Google MLPerf-0.6
submission. |
| doi_str_mv | 10.48550/arxiv.1909.09756 |
| format | Article |
| fullrecord | <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_1909_09756</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>1909_09756</sourcerecordid><originalsourceid>FETCH-LOGICAL-a676-d04c1ae47ef62dde09059a12a04d6ada8e69ed50e9f485df207c8513a80002633</originalsourceid><addsrcrecordid>eNotzkFqwzAQhWFtughuD5BVdQG5I8uSrWUIqVNwiaHu2gyeUQk4UZAhtLevk3b1Fj88PiHWGvKythZeMH0fr7n24HPwlXUroT5GnFi-tx2noCB38hSJp1nGs2xi_Fpa332qq5FdpPlRPAScZn7630z0r7t-u1ftoXnbblqFrnKKoBw1cllxcAURgwfrURcIJTkkrNl5Jgvsw8KiUEA11lYbrAGgcMZk4vnv9u4dLul4wvQz3NzD3W1-Af3iOnk</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Scale MLPerf-0.6 models on Google TPU-v3 Pods</title><source>arXiv.org</source><creator>Kumar, Sameer ; Bitorff, Victor ; Chen, Dehao ; Chou, Chiachen ; Hechtman, Blake ; Lee, HyoukJoong ; Kumar, Naveen ; Mattson, Peter ; Wang, Shibo ; Wang, Tao ; Xu, Yuanzhong ; Zhou, Zongwei</creator><creatorcontrib>Kumar, Sameer ; Bitorff, Victor ; Chen, Dehao ; Chou, Chiachen ; Hechtman, Blake ; Lee, HyoukJoong ; Kumar, Naveen ; Mattson, Peter ; Wang, Shibo ; Wang, Tao ; Xu, Yuanzhong ; Zhou, Zongwei</creatorcontrib><description>The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6
training benchmark demonstrates the scalability of a suite of industry relevant
ML models. MLPerf defines a suite of models, datasets and rules to follow when
benchmarking to ensure results are comparable across hardware, frameworks and
companies. Using this suite of models, we discuss the optimizations and
techniques including choice of optimizer, spatial partitioning and weight
update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify
properties of models that make scaling them challenging, such as limited data
parallelism and unscaled weights. These optimizations contribute to record
performance in transformer, Resnet-50 and SSD in the Google MLPerf-0.6
submission.</description><identifier>DOI: 10.48550/arxiv.1909.09756</identifier><language>eng</language><subject>Computer Science - Artificial Intelligence ; Computer Science - Learning ; Computer Science - Performance</subject><creationdate>2019-09</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,777,882</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/1909.09756$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.1909.09756$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Kumar, Sameer</creatorcontrib><creatorcontrib>Bitorff, Victor</creatorcontrib><creatorcontrib>Chen, Dehao</creatorcontrib><creatorcontrib>Chou, Chiachen</creatorcontrib><creatorcontrib>Hechtman, Blake</creatorcontrib><creatorcontrib>Lee, HyoukJoong</creatorcontrib><creatorcontrib>Kumar, Naveen</creatorcontrib><creatorcontrib>Mattson, Peter</creatorcontrib><creatorcontrib>Wang, Shibo</creatorcontrib><creatorcontrib>Wang, Tao</creatorcontrib><creatorcontrib>Xu, Yuanzhong</creatorcontrib><creatorcontrib>Zhou, Zongwei</creatorcontrib><title>Scale MLPerf-0.6 models on Google TPU-v3 Pods</title><description>The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6
training benchmark demonstrates the scalability of a suite of industry relevant
ML models. MLPerf defines a suite of models, datasets and rules to follow when
benchmarking to ensure results are comparable across hardware, frameworks and
companies. Using this suite of models, we discuss the optimizations and
techniques including choice of optimizer, spatial partitioning and weight
update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify
properties of models that make scaling them challenging, such as limited data
parallelism and unscaled weights. These optimizations contribute to record
performance in transformer, Resnet-50 and SSD in the Google MLPerf-0.6
submission.</description><subject>Computer Science - Artificial Intelligence</subject><subject>Computer Science - Learning</subject><subject>Computer Science - Performance</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2019</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotzkFqwzAQhWFtughuD5BVdQG5I8uSrWUIqVNwiaHu2gyeUQk4UZAhtLevk3b1Fj88PiHWGvKythZeMH0fr7n24HPwlXUroT5GnFi-tx2noCB38hSJp1nGs2xi_Fpa332qq5FdpPlRPAScZn7630z0r7t-u1ftoXnbblqFrnKKoBw1cllxcAURgwfrURcIJTkkrNl5Jgvsw8KiUEA11lYbrAGgcMZk4vnv9u4dLul4wvQz3NzD3W1-Af3iOnk</recordid><startdate>20190920</startdate><enddate>20190920</enddate><creator>Kumar, Sameer</creator><creator>Bitorff, Victor</creator><creator>Chen, Dehao</creator><creator>Chou, Chiachen</creator><creator>Hechtman, Blake</creator><creator>Lee, HyoukJoong</creator><creator>Kumar, Naveen</creator><creator>Mattson, Peter</creator><creator>Wang, Shibo</creator><creator>Wang, Tao</creator><creator>Xu, Yuanzhong</creator><creator>Zhou, Zongwei</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20190920</creationdate><title>Scale MLPerf-0.6 models on Google TPU-v3 Pods</title><author>Kumar, Sameer ; Bitorff, Victor ; Chen, Dehao ; Chou, Chiachen ; Hechtman, Blake ; Lee, HyoukJoong ; Kumar, Naveen ; Mattson, Peter ; Wang, Shibo ; Wang, Tao ; Xu, Yuanzhong ; Zhou, Zongwei</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a676-d04c1ae47ef62dde09059a12a04d6ada8e69ed50e9f485df207c8513a80002633</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2019</creationdate><topic>Computer Science - Artificial Intelligence</topic><topic>Computer Science - Learning</topic><topic>Computer Science - Performance</topic><toplevel>online_resources</toplevel><creatorcontrib>Kumar, Sameer</creatorcontrib><creatorcontrib>Bitorff, Victor</creatorcontrib><creatorcontrib>Chen, Dehao</creatorcontrib><creatorcontrib>Chou, Chiachen</creatorcontrib><creatorcontrib>Hechtman, Blake</creatorcontrib><creatorcontrib>Lee, HyoukJoong</creatorcontrib><creatorcontrib>Kumar, Naveen</creatorcontrib><creatorcontrib>Mattson, Peter</creatorcontrib><creatorcontrib>Wang, Shibo</creatorcontrib><creatorcontrib>Wang, Tao</creatorcontrib><creatorcontrib>Xu, Yuanzhong</creatorcontrib><creatorcontrib>Zhou, Zongwei</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Kumar, Sameer</au><au>Bitorff, Victor</au><au>Chen, Dehao</au><au>Chou, Chiachen</au><au>Hechtman, Blake</au><au>Lee, HyoukJoong</au><au>Kumar, Naveen</au><au>Mattson, Peter</au><au>Wang, Shibo</au><au>Wang, Tao</au><au>Xu, Yuanzhong</au><au>Zhou, Zongwei</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Scale MLPerf-0.6 models on Google TPU-v3 Pods</atitle><date>2019-09-20</date><risdate>2019</risdate><abstract>The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6
training benchmark demonstrates the scalability of a suite of industry relevant
ML models. MLPerf defines a suite of models, datasets and rules to follow when
benchmarking to ensure results are comparable across hardware, frameworks and
companies. Using this suite of models, we discuss the optimizations and
techniques including choice of optimizer, spatial partitioning and weight
update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify
properties of models that make scaling them challenging, such as limited data
parallelism and unscaled weights. These optimizations contribute to record
performance in transformer, Resnet-50 and SSD in the Google MLPerf-0.6
submission.</abstract><doi>10.48550/arxiv.1909.09756</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | DOI: 10.48550/arxiv.1909.09756 |
| ispartof | |
| issn | |
| language | eng |
| recordid | cdi_arxiv_primary_1909_09756 |
| source | arXiv.org |
| subjects | Computer Science - Artificial Intelligence Computer Science - Learning Computer Science - Performance |
| title | Scale MLPerf-0.6 models on Google TPU-v3 Pods |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-18T11%3A47%3A32IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Scale%20MLPerf-0.6%20models%20on%20Google%20TPU-v3%20Pods&rft.au=Kumar,%20Sameer&rft.date=2019-09-20&rft_id=info:doi/10.48550/arxiv.1909.09756&rft_dat=%3Carxiv_GOX%3E1909_09756%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |