Efficient Scaling of Dynamic Graph Neural Networks
We present distributed algorithms for training dynamic Graph Neural Networks (GNN) on large scale graphs spanning multi-node, multi-GPU systems. To the best of our knowledge, this is the first scaling study on dynamic GNN. We devise mechanisms for reducing the GPU memory usage and identify two execu...
Gespeichert in:
| Veröffentlicht in: | arXiv.org 2021-09 |
|---|---|
| Hauptverfasser: | , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | arXiv.org |
| container_volume | |
| creator | Chakaravarthy, Venkatesan T Pandian, Shivmaran S Raje, Saurabh Sabharwal, Yogish Suzumura, Toyotaro Ubaru, Shashanka |
| description | We present distributed algorithms for training dynamic Graph Neural Networks (GNN) on large scale graphs spanning multi-node, multi-GPU systems. To the best of our knowledge, this is the first scaling study on dynamic GNN. We devise mechanisms for reducing the GPU memory usage and identify two execution time bottlenecks: CPU-GPU data transfer; and communication volume. Exploiting properties of dynamic graphs, we design a graph difference-based strategy to significantly reduce the transfer time. We develop a simple, but effective data distribution technique under which the communication volume remains fixed and linear in the input size, for any number of GPUs. Our experiments using billion-size graphs on a system of 128 GPUs shows that: (i) the distribution scheme achieves up to 30x speedup on 128 GPUs; (ii) the graph-difference technique reduces the transfer time by a factor of up to 4.1x and the overall execution time by up to 40% |
| doi_str_mv | 10.48550/arxiv.2109.07893 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_arxiv</sourceid><recordid>TN_cdi_arxiv_primary_2109_07893</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2573634325</sourcerecordid><originalsourceid>FETCH-LOGICAL-a525-4ee305a8f414669c8ac3690e4f5b6573b5e48f222bcd9a0ca01e39580362c2053</originalsourceid><addsrcrecordid>eNotj8FOwzAQRC0kJKrSD-BEJM4pm12v4xxRKS1SBQd6jxxjQ0qaBCcB-veEltNcnmbmCXGVwFxqZrg14af8mmMC2RxSndGZmCBREmuJeCFmXbcDAFQpMtNE4NL70pau7qMXa6qyfosaH90farMvbbQKpn2PntwQTDVG_92Ej-5SnHtTdW72n1OxfVhuF-t487x6XNxtYsPIsXSOgI32MpFKZVYbSyoDJz0XilMq2EntEbGwr5kBayBxlLEGUmgRmKbi-lR7FMrbUO5NOOR_YvlRbCRuTkQbms_BdX2-a4ZQj59yHBcUSUKmX_eKTm8</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2573634325</pqid></control><display><type>article</type><title>Efficient Scaling of Dynamic Graph Neural Networks</title><source>arXiv.org</source><source>Free E- Journals</source><creator>Chakaravarthy, Venkatesan T ; Pandian, Shivmaran S ; Raje, Saurabh ; Sabharwal, Yogish ; Suzumura, Toyotaro ; Ubaru, Shashanka</creator><creatorcontrib>Chakaravarthy, Venkatesan T ; Pandian, Shivmaran S ; Raje, Saurabh ; Sabharwal, Yogish ; Suzumura, Toyotaro ; Ubaru, Shashanka</creatorcontrib><description>We present distributed algorithms for training dynamic Graph Neural Networks (GNN) on large scale graphs spanning multi-node, multi-GPU systems. To the best of our knowledge, this is the first scaling study on dynamic GNN. We devise mechanisms for reducing the GPU memory usage and identify two execution time bottlenecks: CPU-GPU data transfer; and communication volume. Exploiting properties of dynamic graphs, we design a graph difference-based strategy to significantly reduce the transfer time. We develop a simple, but effective data distribution technique under which the communication volume remains fixed and linear in the input size, for any number of GPUs. Our experiments using billion-size graphs on a system of 128 GPUs shows that: (i) the distribution scheme achieves up to 30x speedup on 128 GPUs; (ii) the graph-difference technique reduces the transfer time by a factor of up to 4.1x and the overall execution time by up to 40%</description><identifier>EISSN: 2331-8422</identifier><identifier>DOI: 10.48550/arxiv.2109.07893</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Algorithms ; Computer Science - Distributed, Parallel, and Cluster Computing ; Computer Science - Learning ; Data transfer (computers) ; Graph neural networks ; Graphs ; Neural networks</subject><ispartof>arXiv.org, 2021-09</ispartof><rights>2021. This work is published under http://creativecommons.org/licenses/by-nc-nd/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>http://creativecommons.org/licenses/by-nc-nd/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,784,885,27923</link.rule.ids><backlink>$$Uhttps://doi.org/10.48550/arXiv.2109.07893$$DView paper in arXiv$$Hfree_for_read</backlink><backlink>$$Uhttps://doi.org/10.1145/3458817.3480858$$DView published paper (Access to full text may be restricted)$$Hfree_for_read</backlink></links><search><creatorcontrib>Chakaravarthy, Venkatesan T</creatorcontrib><creatorcontrib>Pandian, Shivmaran S</creatorcontrib><creatorcontrib>Raje, Saurabh</creatorcontrib><creatorcontrib>Sabharwal, Yogish</creatorcontrib><creatorcontrib>Suzumura, Toyotaro</creatorcontrib><creatorcontrib>Ubaru, Shashanka</creatorcontrib><title>Efficient Scaling of Dynamic Graph Neural Networks</title><title>arXiv.org</title><description>We present distributed algorithms for training dynamic Graph Neural Networks (GNN) on large scale graphs spanning multi-node, multi-GPU systems. To the best of our knowledge, this is the first scaling study on dynamic GNN. We devise mechanisms for reducing the GPU memory usage and identify two execution time bottlenecks: CPU-GPU data transfer; and communication volume. Exploiting properties of dynamic graphs, we design a graph difference-based strategy to significantly reduce the transfer time. We develop a simple, but effective data distribution technique under which the communication volume remains fixed and linear in the input size, for any number of GPUs. Our experiments using billion-size graphs on a system of 128 GPUs shows that: (i) the distribution scheme achieves up to 30x speedup on 128 GPUs; (ii) the graph-difference technique reduces the transfer time by a factor of up to 4.1x and the overall execution time by up to 40%</description><subject>Algorithms</subject><subject>Computer Science - Distributed, Parallel, and Cluster Computing</subject><subject>Computer Science - Learning</subject><subject>Data transfer (computers)</subject><subject>Graph neural networks</subject><subject>Graphs</subject><subject>Neural networks</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><sourceid>GOX</sourceid><recordid>eNotj8FOwzAQRC0kJKrSD-BEJM4pm12v4xxRKS1SBQd6jxxjQ0qaBCcB-veEltNcnmbmCXGVwFxqZrg14af8mmMC2RxSndGZmCBREmuJeCFmXbcDAFQpMtNE4NL70pau7qMXa6qyfosaH90farMvbbQKpn2PntwQTDVG_92Ej-5SnHtTdW72n1OxfVhuF-t487x6XNxtYsPIsXSOgI32MpFKZVYbSyoDJz0XilMq2EntEbGwr5kBayBxlLEGUmgRmKbi-lR7FMrbUO5NOOR_YvlRbCRuTkQbms_BdX2-a4ZQj59yHBcUSUKmX_eKTm8</recordid><startdate>20210916</startdate><enddate>20210916</enddate><creator>Chakaravarthy, Venkatesan T</creator><creator>Pandian, Shivmaran S</creator><creator>Raje, Saurabh</creator><creator>Sabharwal, Yogish</creator><creator>Suzumura, Toyotaro</creator><creator>Ubaru, Shashanka</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20210916</creationdate><title>Efficient Scaling of Dynamic Graph Neural Networks</title><author>Chakaravarthy, Venkatesan T ; Pandian, Shivmaran S ; Raje, Saurabh ; Sabharwal, Yogish ; Suzumura, Toyotaro ; Ubaru, Shashanka</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a525-4ee305a8f414669c8ac3690e4f5b6573b5e48f222bcd9a0ca01e39580362c2053</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Algorithms</topic><topic>Computer Science - Distributed, Parallel, and Cluster Computing</topic><topic>Computer Science - Learning</topic><topic>Data transfer (computers)</topic><topic>Graph neural networks</topic><topic>Graphs</topic><topic>Neural networks</topic><toplevel>online_resources</toplevel><creatorcontrib>Chakaravarthy, Venkatesan T</creatorcontrib><creatorcontrib>Pandian, Shivmaran S</creatorcontrib><creatorcontrib>Raje, Saurabh</creatorcontrib><creatorcontrib>Sabharwal, Yogish</creatorcontrib><creatorcontrib>Suzumura, Toyotaro</creatorcontrib><creatorcontrib>Ubaru, Shashanka</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection><collection>arXiv Computer Science</collection><collection>arXiv.org</collection><jtitle>arXiv.org</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Chakaravarthy, Venkatesan T</au><au>Pandian, Shivmaran S</au><au>Raje, Saurabh</au><au>Sabharwal, Yogish</au><au>Suzumura, Toyotaro</au><au>Ubaru, Shashanka</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Efficient Scaling of Dynamic Graph Neural Networks</atitle><jtitle>arXiv.org</jtitle><date>2021-09-16</date><risdate>2021</risdate><eissn>2331-8422</eissn><abstract>We present distributed algorithms for training dynamic Graph Neural Networks (GNN) on large scale graphs spanning multi-node, multi-GPU systems. To the best of our knowledge, this is the first scaling study on dynamic GNN. We devise mechanisms for reducing the GPU memory usage and identify two execution time bottlenecks: CPU-GPU data transfer; and communication volume. Exploiting properties of dynamic graphs, we design a graph difference-based strategy to significantly reduce the transfer time. We develop a simple, but effective data distribution technique under which the communication volume remains fixed and linear in the input size, for any number of GPUs. Our experiments using billion-size graphs on a system of 128 GPUs shows that: (i) the distribution scheme achieves up to 30x speedup on 128 GPUs; (ii) the graph-difference technique reduces the transfer time by a factor of up to 4.1x and the overall execution time by up to 40%</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><doi>10.48550/arxiv.2109.07893</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | EISSN: 2331-8422 |
| ispartof | arXiv.org, 2021-09 |
| issn | 2331-8422 |
| language | eng |
| recordid | cdi_arxiv_primary_2109_07893 |
| source | arXiv.org; Free E- Journals |
| subjects | Algorithms Computer Science - Distributed, Parallel, and Cluster Computing Computer Science - Learning Data transfer (computers) Graph neural networks Graphs Neural networks |
| title | Efficient Scaling of Dynamic Graph Neural Networks |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-10T03%3A04%3A36IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_arxiv&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Efficient%20Scaling%20of%20Dynamic%20Graph%20Neural%20Networks&rft.jtitle=arXiv.org&rft.au=Chakaravarthy,%20Venkatesan%20T&rft.date=2021-09-16&rft.eissn=2331-8422&rft_id=info:doi/10.48550/arxiv.2109.07893&rft_dat=%3Cproquest_arxiv%3E2573634325%3C/proquest_arxiv%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2573634325&rft_id=info:pmid/&rfr_iscdi=true |