Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints
There is a large literature on the similarities and differences between biological neural circuits and deep artificial neural networks (DNNs). However, modern training of DNNs relies on several engineering tricks such as data batching, normalization, adaptive optimizers, and precise weight initializ...
Gespeichert in:
Veröffentlicht in: | arXiv.org 2024-10 |
---|---|
Hauptverfasser: | , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | arXiv.org |
container_volume | |
creator | Shervani-Tabar, Navid Marzieh Alireza Mirhoseini Rosenbaum, Robert |
description | There is a large literature on the similarities and differences between biological neural circuits and deep artificial neural networks (DNNs). However, modern training of DNNs relies on several engineering tricks such as data batching, normalization, adaptive optimizers, and precise weight initialization. Despite their critical role in training DNNs, these engineering tricks are often overlooked when drawing parallels between biological and artificial networks, potentially due to a lack of evidence for their direct biological implementation. In this study, we show that Oja's plasticity rule partly overcomes the need for some engineering tricks. Specifically, under difficult, but biologically realistic learning scenarios such as online learning, deep architectures, and sub-optimal weight initialization, Oja's rule can substantially improve the performance of pure backpropagation. Our results demonstrate that simple synaptic plasticity rules can overcome challenges to learning that are typically overcome using less biologically plausible approaches when training DNNs. |
format | Article |
fullrecord | <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_3094565423</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3094565423</sourcerecordid><originalsourceid>FETCH-proquest_journals_30945654233</originalsourceid><addsrcrecordid>eNqNjs0KwjAQhIMgWLTvsODBk1DT1p-zKN68eJcYtzU1Zms2UXx7q_gAnmaYbwamJxKZ57PpspByIFLmJssyOV_IsswTcd03asLQWsXBaBNe4KNFoAd6TTdkYOyssqAvylp0dRdRBcEr44yrwWH8UIfhSf7KEN0ZPZwMWaqN_uzI8bcdeCT6lbKM6U-HYrzdHNa7aevpHpHDsaHoXYeOebYqynlZdM__a70BAQhKbw</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3094565423</pqid></control><display><type>article</type><title>Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints</title><source>Free E- Journals</source><creator>Shervani-Tabar, Navid ; Marzieh Alireza Mirhoseini ; Rosenbaum, Robert</creator><creatorcontrib>Shervani-Tabar, Navid ; Marzieh Alireza Mirhoseini ; Rosenbaum, Robert</creatorcontrib><description>There is a large literature on the similarities and differences between biological neural circuits and deep artificial neural networks (DNNs). However, modern training of DNNs relies on several engineering tricks such as data batching, normalization, adaptive optimizers, and precise weight initialization. Despite their critical role in training DNNs, these engineering tricks are often overlooked when drawing parallels between biological and artificial networks, potentially due to a lack of evidence for their direct biological implementation. In this study, we show that Oja's plasticity rule partly overcomes the need for some engineering tricks. Specifically, under difficult, but biologically realistic learning scenarios such as online learning, deep architectures, and sub-optimal weight initialization, Oja's rule can substantially improve the performance of pure backpropagation. Our results demonstrate that simple synaptic plasticity rules can overcome challenges to learning that are typically overcome using less biologically plausible approaches when training DNNs.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Artificial neural networks ; Back propagation networks ; Biological effects</subject><ispartof>arXiv.org, 2024-10</ispartof><rights>2024. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>780,784</link.rule.ids></links><search><creatorcontrib>Shervani-Tabar, Navid</creatorcontrib><creatorcontrib>Marzieh Alireza Mirhoseini</creatorcontrib><creatorcontrib>Rosenbaum, Robert</creatorcontrib><title>Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints</title><title>arXiv.org</title><description>There is a large literature on the similarities and differences between biological neural circuits and deep artificial neural networks (DNNs). However, modern training of DNNs relies on several engineering tricks such as data batching, normalization, adaptive optimizers, and precise weight initialization. Despite their critical role in training DNNs, these engineering tricks are often overlooked when drawing parallels between biological and artificial networks, potentially due to a lack of evidence for their direct biological implementation. In this study, we show that Oja's plasticity rule partly overcomes the need for some engineering tricks. Specifically, under difficult, but biologically realistic learning scenarios such as online learning, deep architectures, and sub-optimal weight initialization, Oja's rule can substantially improve the performance of pure backpropagation. Our results demonstrate that simple synaptic plasticity rules can overcome challenges to learning that are typically overcome using less biologically plausible approaches when training DNNs.</description><subject>Artificial neural networks</subject><subject>Back propagation networks</subject><subject>Biological effects</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNqNjs0KwjAQhIMgWLTvsODBk1DT1p-zKN68eJcYtzU1Zms2UXx7q_gAnmaYbwamJxKZ57PpspByIFLmJssyOV_IsswTcd03asLQWsXBaBNe4KNFoAd6TTdkYOyssqAvylp0dRdRBcEr44yrwWH8UIfhSf7KEN0ZPZwMWaqN_uzI8bcdeCT6lbKM6U-HYrzdHNa7aevpHpHDsaHoXYeOebYqynlZdM__a70BAQhKbw</recordid><startdate>20241011</startdate><enddate>20241011</enddate><creator>Shervani-Tabar, Navid</creator><creator>Marzieh Alireza Mirhoseini</creator><creator>Rosenbaum, Robert</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20241011</creationdate><title>Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints</title><author>Shervani-Tabar, Navid ; Marzieh Alireza Mirhoseini ; Rosenbaum, Robert</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_30945654233</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Artificial neural networks</topic><topic>Back propagation networks</topic><topic>Biological effects</topic><toplevel>online_resources</toplevel><creatorcontrib>Shervani-Tabar, Navid</creatorcontrib><creatorcontrib>Marzieh Alireza Mirhoseini</creatorcontrib><creatorcontrib>Rosenbaum, Robert</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Access via ProQuest (Open Access)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Shervani-Tabar, Navid</au><au>Marzieh Alireza Mirhoseini</au><au>Rosenbaum, Robert</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints</atitle><jtitle>arXiv.org</jtitle><date>2024-10-11</date><risdate>2024</risdate><eissn>2331-8422</eissn><abstract>There is a large literature on the similarities and differences between biological neural circuits and deep artificial neural networks (DNNs). However, modern training of DNNs relies on several engineering tricks such as data batching, normalization, adaptive optimizers, and precise weight initialization. Despite their critical role in training DNNs, these engineering tricks are often overlooked when drawing parallels between biological and artificial networks, potentially due to a lack of evidence for their direct biological implementation. In this study, we show that Oja's plasticity rule partly overcomes the need for some engineering tricks. Specifically, under difficult, but biologically realistic learning scenarios such as online learning, deep architectures, and sub-optimal weight initialization, Oja's rule can substantially improve the performance of pure backpropagation. Our results demonstrate that simple synaptic plasticity rules can overcome challenges to learning that are typically overcome using less biologically plausible approaches when training DNNs.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | EISSN: 2331-8422 |
ispartof | arXiv.org, 2024-10 |
issn | 2331-8422 |
language | eng |
recordid | cdi_proquest_journals_3094565423 |
source | Free E- Journals |
subjects | Artificial neural networks Back propagation networks Biological effects |
title | Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-23T13%3A44%3A15IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=Oja's%20plasticity%20rule%20overcomes%20several%20challenges%20of%20training%20neural%20networks%20under%20biological%20constraints&rft.jtitle=arXiv.org&rft.au=Shervani-Tabar,%20Navid&rft.date=2024-10-11&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E3094565423%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3094565423&rft_id=info:pmid/&rfr_iscdi=true |