Deep Neural Message Passing With Hierarchical Layer Aggregation and Neighbor Normalization

As a unified framework for graph neural networks, message passing-based neural network (MPNN) has attracted a lot of research interest and has been shown successfully in a number of domains in recent years. However, because of over-smoothing and vanishing gradients, deep MPNNs are still difficult to...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE transaction on neural networks and learning systems 2022-12, Vol.33 (12), p.7172-7184
Hauptverfasser: Fan, Xiaolong, Gong, Maoguo, Tang, Zedong, Wu, Yue
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 7184
container_issue 12
container_start_page 7172
container_title IEEE transaction on neural networks and learning systems
container_volume 33
creator Fan, Xiaolong
Gong, Maoguo
Tang, Zedong
Wu, Yue
description As a unified framework for graph neural networks, message passing-based neural network (MPNN) has attracted a lot of research interest and has been shown successfully in a number of domains in recent years. However, because of over-smoothing and vanishing gradients, deep MPNNs are still difficult to train. To alleviate these issues, we first introduce a deep hierarchical layer aggregation (DHLA) strategy, which utilizes a block-based layer aggregation to aggregate representations from different layers and transfers the output of the previous block to the subsequent block, so that deeper MPNNs can be easily trained. Additionally, to stabilize the training process, we also develop a novel normalization strategy, neighbor normalization (NeighborNorm), which normalizes the neighbor of each node to further address the training issue in deep MPNNs. Our analysis reveals that NeighborNorm can smooth the gradient of the loss function, i.e., adding NeighborNorm makes the optimization landscape much easier to navigate. Experimental results on two typical graph pattern-recognition tasks, including node classification and graph classification, demonstrate the necessity and effectiveness of the proposed strategies for graph message-passing neural networks.
doi_str_mv 10.1109/TNNLS.2021.3084319
format Article
fullrecord <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_pubmed_primary_34106862</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>9450014</ieee_id><sourcerecordid>2742704051</sourcerecordid><originalsourceid>FETCH-LOGICAL-c351t-4f6d319ce1eb0353cbc533d4c16c8019242e04575169bd7bab851c2acf7128413</originalsourceid><addsrcrecordid>eNpdkE1LAzEQhoMoKrV_QEEWvHhpzeRjN3ss9RNqFawoXkI2O91Gtrs16R7qr3draw_mMoF53pfhIeQUaB-ApleT8Xj00meUQZ9TJTike-SYQcx6jCu1v_sn70ekG8InbV9MZSzSQ3LEBdBYxeyYfFwjLqIxNt6U0SOGYAqMnk0IriqiN7ecRfcOvfF25mxLjMwKfTQoCo-FWbq6ikyVt3FXzLLaR-Paz03pvn9XJ-RgasqA3e3skNfbm8nwvjd6unsYDkY9yyUse2Ia5-31FgEzyiW3mZWc58JCbBWFlAmGVMhEQpxmeZKZTEmwzNhpAkwJ4B1yueld-PqrwbDUcxcslqWpsG6CZpKnSikmZIte_EM_68ZX7XWaJYIlVFC5LmQbyvo6BI9TvfBubvxKA9Vr-fpXvl7L11v5beh8W91kc8x3kT_VLXC2ARwi7tapkJSC4D9fXIcJ</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2742704051</pqid></control><display><type>article</type><title>Deep Neural Message Passing With Hierarchical Layer Aggregation and Neighbor Normalization</title><source>IEEE Electronic Library (IEL)</source><creator>Fan, Xiaolong ; Gong, Maoguo ; Tang, Zedong ; Wu, Yue</creator><creatorcontrib>Fan, Xiaolong ; Gong, Maoguo ; Tang, Zedong ; Wu, Yue</creatorcontrib><description>As a unified framework for graph neural networks, message passing-based neural network (MPNN) has attracted a lot of research interest and has been shown successfully in a number of domains in recent years. However, because of over-smoothing and vanishing gradients, deep MPNNs are still difficult to train. To alleviate these issues, we first introduce a deep hierarchical layer aggregation (DHLA) strategy, which utilizes a block-based layer aggregation to aggregate representations from different layers and transfers the output of the previous block to the subsequent block, so that deeper MPNNs can be easily trained. Additionally, to stabilize the training process, we also develop a novel normalization strategy, neighbor normalization (NeighborNorm), which normalizes the neighbor of each node to further address the training issue in deep MPNNs. Our analysis reveals that NeighborNorm can smooth the gradient of the loss function, i.e., adding NeighborNorm makes the optimization landscape much easier to navigate. Experimental results on two typical graph pattern-recognition tasks, including node classification and graph classification, demonstrate the necessity and effectiveness of the proposed strategies for graph message-passing neural networks.</description><identifier>ISSN: 2162-237X</identifier><identifier>EISSN: 2162-2388</identifier><identifier>DOI: 10.1109/TNNLS.2021.3084319</identifier><identifier>PMID: 34106862</identifier><identifier>CODEN: ITNNAL</identifier><language>eng</language><publisher>United States: IEEE</publisher><subject>Agglomeration ; Aggregates ; Algorithms ; Analytical models ; Classification ; Data models ; Deep graph neural networks ; Degradation ; graph data mining ; Graph neural networks ; graph normalization ; graph representation learning ; Message passing ; Neural networks ; Neural Networks, Computer ; Optimization ; Pattern recognition ; Smoothing methods ; Task analysis ; Training</subject><ispartof>IEEE transaction on neural networks and learning systems, 2022-12, Vol.33 (12), p.7172-7184</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022</rights><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c351t-4f6d319ce1eb0353cbc533d4c16c8019242e04575169bd7bab851c2acf7128413</citedby><cites>FETCH-LOGICAL-c351t-4f6d319ce1eb0353cbc533d4c16c8019242e04575169bd7bab851c2acf7128413</cites><orcidid>0000-0002-0415-8556 ; 0000-0002-3459-5079 ; 0000-0001-8484-2069</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/9450014$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,776,780,792,27901,27902,54733</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/9450014$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/34106862$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Fan, Xiaolong</creatorcontrib><creatorcontrib>Gong, Maoguo</creatorcontrib><creatorcontrib>Tang, Zedong</creatorcontrib><creatorcontrib>Wu, Yue</creatorcontrib><title>Deep Neural Message Passing With Hierarchical Layer Aggregation and Neighbor Normalization</title><title>IEEE transaction on neural networks and learning systems</title><addtitle>TNNLS</addtitle><addtitle>IEEE Trans Neural Netw Learn Syst</addtitle><description>As a unified framework for graph neural networks, message passing-based neural network (MPNN) has attracted a lot of research interest and has been shown successfully in a number of domains in recent years. However, because of over-smoothing and vanishing gradients, deep MPNNs are still difficult to train. To alleviate these issues, we first introduce a deep hierarchical layer aggregation (DHLA) strategy, which utilizes a block-based layer aggregation to aggregate representations from different layers and transfers the output of the previous block to the subsequent block, so that deeper MPNNs can be easily trained. Additionally, to stabilize the training process, we also develop a novel normalization strategy, neighbor normalization (NeighborNorm), which normalizes the neighbor of each node to further address the training issue in deep MPNNs. Our analysis reveals that NeighborNorm can smooth the gradient of the loss function, i.e., adding NeighborNorm makes the optimization landscape much easier to navigate. Experimental results on two typical graph pattern-recognition tasks, including node classification and graph classification, demonstrate the necessity and effectiveness of the proposed strategies for graph message-passing neural networks.</description><subject>Agglomeration</subject><subject>Aggregates</subject><subject>Algorithms</subject><subject>Analytical models</subject><subject>Classification</subject><subject>Data models</subject><subject>Deep graph neural networks</subject><subject>Degradation</subject><subject>graph data mining</subject><subject>Graph neural networks</subject><subject>graph normalization</subject><subject>graph representation learning</subject><subject>Message passing</subject><subject>Neural networks</subject><subject>Neural Networks, Computer</subject><subject>Optimization</subject><subject>Pattern recognition</subject><subject>Smoothing methods</subject><subject>Task analysis</subject><subject>Training</subject><issn>2162-237X</issn><issn>2162-2388</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><sourceid>EIF</sourceid><recordid>eNpdkE1LAzEQhoMoKrV_QEEWvHhpzeRjN3ss9RNqFawoXkI2O91Gtrs16R7qr3draw_mMoF53pfhIeQUaB-ApleT8Xj00meUQZ9TJTike-SYQcx6jCu1v_sn70ekG8InbV9MZSzSQ3LEBdBYxeyYfFwjLqIxNt6U0SOGYAqMnk0IriqiN7ecRfcOvfF25mxLjMwKfTQoCo-FWbq6ikyVt3FXzLLaR-Paz03pvn9XJ-RgasqA3e3skNfbm8nwvjd6unsYDkY9yyUse2Ia5-31FgEzyiW3mZWc58JCbBWFlAmGVMhEQpxmeZKZTEmwzNhpAkwJ4B1yueld-PqrwbDUcxcslqWpsG6CZpKnSikmZIte_EM_68ZX7XWaJYIlVFC5LmQbyvo6BI9TvfBubvxKA9Vr-fpXvl7L11v5beh8W91kc8x3kT_VLXC2ARwi7tapkJSC4D9fXIcJ</recordid><startdate>20221201</startdate><enddate>20221201</enddate><creator>Fan, Xiaolong</creator><creator>Gong, Maoguo</creator><creator>Tang, Zedong</creator><creator>Wu, Yue</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7QF</scope><scope>7QO</scope><scope>7QP</scope><scope>7QQ</scope><scope>7QR</scope><scope>7SC</scope><scope>7SE</scope><scope>7SP</scope><scope>7SR</scope><scope>7TA</scope><scope>7TB</scope><scope>7TK</scope><scope>7U5</scope><scope>8BQ</scope><scope>8FD</scope><scope>F28</scope><scope>FR3</scope><scope>H8D</scope><scope>JG9</scope><scope>JQ2</scope><scope>KR7</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>P64</scope><scope>7X8</scope><orcidid>https://orcid.org/0000-0002-0415-8556</orcidid><orcidid>https://orcid.org/0000-0002-3459-5079</orcidid><orcidid>https://orcid.org/0000-0001-8484-2069</orcidid></search><sort><creationdate>20221201</creationdate><title>Deep Neural Message Passing With Hierarchical Layer Aggregation and Neighbor Normalization</title><author>Fan, Xiaolong ; Gong, Maoguo ; Tang, Zedong ; Wu, Yue</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c351t-4f6d319ce1eb0353cbc533d4c16c8019242e04575169bd7bab851c2acf7128413</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Agglomeration</topic><topic>Aggregates</topic><topic>Algorithms</topic><topic>Analytical models</topic><topic>Classification</topic><topic>Data models</topic><topic>Deep graph neural networks</topic><topic>Degradation</topic><topic>graph data mining</topic><topic>Graph neural networks</topic><topic>graph normalization</topic><topic>graph representation learning</topic><topic>Message passing</topic><topic>Neural networks</topic><topic>Neural Networks, Computer</topic><topic>Optimization</topic><topic>Pattern recognition</topic><topic>Smoothing methods</topic><topic>Task analysis</topic><topic>Training</topic><toplevel>online_resources</toplevel><creatorcontrib>Fan, Xiaolong</creatorcontrib><creatorcontrib>Gong, Maoguo</creatorcontrib><creatorcontrib>Tang, Zedong</creatorcontrib><creatorcontrib>Wu, Yue</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Aluminium Industry Abstracts</collection><collection>Biotechnology Research Abstracts</collection><collection>Calcium &amp; Calcified Tissue Abstracts</collection><collection>Ceramic Abstracts</collection><collection>Chemoreception Abstracts</collection><collection>Computer and Information Systems Abstracts</collection><collection>Corrosion Abstracts</collection><collection>Electronics &amp; Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>Materials Business File</collection><collection>Mechanical &amp; Transportation Engineering Abstracts</collection><collection>Neurosciences Abstracts</collection><collection>Solid State and Superconductivity Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>ANTE: Abstracts in New Technology &amp; Engineering</collection><collection>Engineering Research Database</collection><collection>Aerospace Database</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Civil Engineering Abstracts</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>MEDLINE - Academic</collection><jtitle>IEEE transaction on neural networks and learning systems</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Fan, Xiaolong</au><au>Gong, Maoguo</au><au>Tang, Zedong</au><au>Wu, Yue</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Deep Neural Message Passing With Hierarchical Layer Aggregation and Neighbor Normalization</atitle><jtitle>IEEE transaction on neural networks and learning systems</jtitle><stitle>TNNLS</stitle><addtitle>IEEE Trans Neural Netw Learn Syst</addtitle><date>2022-12-01</date><risdate>2022</risdate><volume>33</volume><issue>12</issue><spage>7172</spage><epage>7184</epage><pages>7172-7184</pages><issn>2162-237X</issn><eissn>2162-2388</eissn><coden>ITNNAL</coden><abstract>As a unified framework for graph neural networks, message passing-based neural network (MPNN) has attracted a lot of research interest and has been shown successfully in a number of domains in recent years. However, because of over-smoothing and vanishing gradients, deep MPNNs are still difficult to train. To alleviate these issues, we first introduce a deep hierarchical layer aggregation (DHLA) strategy, which utilizes a block-based layer aggregation to aggregate representations from different layers and transfers the output of the previous block to the subsequent block, so that deeper MPNNs can be easily trained. Additionally, to stabilize the training process, we also develop a novel normalization strategy, neighbor normalization (NeighborNorm), which normalizes the neighbor of each node to further address the training issue in deep MPNNs. Our analysis reveals that NeighborNorm can smooth the gradient of the loss function, i.e., adding NeighborNorm makes the optimization landscape much easier to navigate. Experimental results on two typical graph pattern-recognition tasks, including node classification and graph classification, demonstrate the necessity and effectiveness of the proposed strategies for graph message-passing neural networks.</abstract><cop>United States</cop><pub>IEEE</pub><pmid>34106862</pmid><doi>10.1109/TNNLS.2021.3084319</doi><tpages>13</tpages><orcidid>https://orcid.org/0000-0002-0415-8556</orcidid><orcidid>https://orcid.org/0000-0002-3459-5079</orcidid><orcidid>https://orcid.org/0000-0001-8484-2069</orcidid></addata></record>
fulltext fulltext_linktorsrc
identifier ISSN: 2162-237X
ispartof IEEE transaction on neural networks and learning systems, 2022-12, Vol.33 (12), p.7172-7184
issn 2162-237X
2162-2388
language eng
recordid cdi_pubmed_primary_34106862
source IEEE Electronic Library (IEL)
subjects Agglomeration
Aggregates
Algorithms
Analytical models
Classification
Data models
Deep graph neural networks
Degradation
graph data mining
Graph neural networks
graph normalization
graph representation learning
Message passing
Neural networks
Neural Networks, Computer
Optimization
Pattern recognition
Smoothing methods
Task analysis
Training
title Deep Neural Message Passing With Hierarchical Layer Aggregation and Neighbor Normalization
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-01T02%3A26%3A01IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Deep%20Neural%20Message%20Passing%20With%20Hierarchical%20Layer%20Aggregation%20and%20Neighbor%20Normalization&rft.jtitle=IEEE%20transaction%20on%20neural%20networks%20and%20learning%20systems&rft.au=Fan,%20Xiaolong&rft.date=2022-12-01&rft.volume=33&rft.issue=12&rft.spage=7172&rft.epage=7184&rft.pages=7172-7184&rft.issn=2162-237X&rft.eissn=2162-2388&rft.coden=ITNNAL&rft_id=info:doi/10.1109/TNNLS.2021.3084319&rft_dat=%3Cproquest_RIE%3E2742704051%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2742704051&rft_id=info:pmid/34106862&rft_ieee_id=9450014&rfr_iscdi=true