Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very different distributions. \emph{System heterogeneity} refers to the fact that client devices have different system capabilities....
Gespeichert in:
Veröffentlicht in: | arXiv.org 2022-10 |
---|---|
Hauptverfasser: | , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | arXiv.org |
container_volume | |
creator | Nguyen, John Wang, Jianyu Malik, Kshitiz Sanjabi, Maziar Rabbat, Michael |
description | An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very different distributions. \emph{System heterogeneity} refers to the fact that client devices have different system capabilities. A considerable number of federated optimization methods address this challenge. In the literature, empirical evaluations usually start federated training from random initialization. However, in many practical applications of federated learning, the server has access to proxy data for the training task that can be used to pre-train a model before starting federated training. We empirically study the impact of starting from a pre-trained model in federated learning using four standard federated learning benchmark datasets. Unsurprisingly, starting from a pre-trained model reduces the training time required to reach a target error rate and enables the training of more accurate models (up to 40\%) than is possible when starting from random initialization. Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity. We recommend that future work proposing and evaluating federated optimization methods evaluate the performance when starting from random and pre-trained initializations. We also believe this study raises several questions for further work on understanding the role of heterogeneity in federated optimization. |
format | Article |
fullrecord | <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2725724631</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2725724631</sourcerecordid><originalsourceid>FETCH-proquest_journals_27257246313</originalsourceid><addsrcrecordid>eNqNjssKgkAUQIcgSMp_uNBa0Bkf7YIiSQhqEbWUIa86YndsZtz09Rn0Aa3O4pzFmTGPCxEFm5jzBfOt7cIw5GnGk0R47HZv0SA4DTtsFG3hTOBahOI5yIcDXcPFYHA1UpGiBiRVUJBySvbqLZ3SBIogxwqNdFjBCaX5his2r2Vv0f9xydb54bo_BoPRrxGtKzs9GppUyaeTjMepiMR_1QdGPD_y</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2725724631</pqid></control><display><type>article</type><title>Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning</title><source>Open Access: Freely Accessible Journals by multiple vendors</source><creator>Nguyen, John ; Wang, Jianyu ; Malik, Kshitiz ; Sanjabi, Maziar ; Rabbat, Michael</creator><creatorcontrib>Nguyen, John ; Wang, Jianyu ; Malik, Kshitiz ; Sanjabi, Maziar ; Rabbat, Michael</creatorcontrib><description>An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very different distributions. \emph{System heterogeneity} refers to the fact that client devices have different system capabilities. A considerable number of federated optimization methods address this challenge. In the literature, empirical evaluations usually start federated training from random initialization. However, in many practical applications of federated learning, the server has access to proxy data for the training task that can be used to pre-train a model before starting federated training. We empirically study the impact of starting from a pre-trained model in federated learning using four standard federated learning benchmark datasets. Unsurprisingly, starting from a pre-trained model reduces the training time required to reach a target error rate and enables the training of more accurate models (up to 40\%) than is possible when starting from random initialization. Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity. We recommend that future work proposing and evaluating federated optimization methods evaluate the performance when starting from random and pre-trained initializations. We also believe this study raises several questions for further work on understanding the role of heterogeneity in federated optimization.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Federated learning ; Heterogeneity ; Optimization ; Performance evaluation</subject><ispartof>arXiv.org, 2022-10</ispartof><rights>2022. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>780,784</link.rule.ids></links><search><creatorcontrib>Nguyen, John</creatorcontrib><creatorcontrib>Wang, Jianyu</creatorcontrib><creatorcontrib>Malik, Kshitiz</creatorcontrib><creatorcontrib>Sanjabi, Maziar</creatorcontrib><creatorcontrib>Rabbat, Michael</creatorcontrib><title>Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning</title><title>arXiv.org</title><description>An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very different distributions. \emph{System heterogeneity} refers to the fact that client devices have different system capabilities. A considerable number of federated optimization methods address this challenge. In the literature, empirical evaluations usually start federated training from random initialization. However, in many practical applications of federated learning, the server has access to proxy data for the training task that can be used to pre-train a model before starting federated training. We empirically study the impact of starting from a pre-trained model in federated learning using four standard federated learning benchmark datasets. Unsurprisingly, starting from a pre-trained model reduces the training time required to reach a target error rate and enables the training of more accurate models (up to 40\%) than is possible when starting from random initialization. Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity. We recommend that future work proposing and evaluating federated optimization methods evaluate the performance when starting from random and pre-trained initializations. We also believe this study raises several questions for further work on understanding the role of heterogeneity in federated optimization.</description><subject>Federated learning</subject><subject>Heterogeneity</subject><subject>Optimization</subject><subject>Performance evaluation</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNqNjssKgkAUQIcgSMp_uNBa0Bkf7YIiSQhqEbWUIa86YndsZtz09Rn0Aa3O4pzFmTGPCxEFm5jzBfOt7cIw5GnGk0R47HZv0SA4DTtsFG3hTOBahOI5yIcDXcPFYHA1UpGiBiRVUJBySvbqLZ3SBIogxwqNdFjBCaX5his2r2Vv0f9xydb54bo_BoPRrxGtKzs9GppUyaeTjMepiMR_1QdGPD_y</recordid><startdate>20221014</startdate><enddate>20221014</enddate><creator>Nguyen, John</creator><creator>Wang, Jianyu</creator><creator>Malik, Kshitiz</creator><creator>Sanjabi, Maziar</creator><creator>Rabbat, Michael</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20221014</creationdate><title>Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning</title><author>Nguyen, John ; Wang, Jianyu ; Malik, Kshitiz ; Sanjabi, Maziar ; Rabbat, Michael</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_27257246313</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Federated learning</topic><topic>Heterogeneity</topic><topic>Optimization</topic><topic>Performance evaluation</topic><toplevel>online_resources</toplevel><creatorcontrib>Nguyen, John</creatorcontrib><creatorcontrib>Wang, Jianyu</creatorcontrib><creatorcontrib>Malik, Kshitiz</creatorcontrib><creatorcontrib>Sanjabi, Maziar</creatorcontrib><creatorcontrib>Rabbat, Michael</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>AUTh Library subscriptions: ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Nguyen, John</au><au>Wang, Jianyu</au><au>Malik, Kshitiz</au><au>Sanjabi, Maziar</au><au>Rabbat, Michael</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning</atitle><jtitle>arXiv.org</jtitle><date>2022-10-14</date><risdate>2022</risdate><eissn>2331-8422</eissn><abstract>An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very different distributions. \emph{System heterogeneity} refers to the fact that client devices have different system capabilities. A considerable number of federated optimization methods address this challenge. In the literature, empirical evaluations usually start federated training from random initialization. However, in many practical applications of federated learning, the server has access to proxy data for the training task that can be used to pre-train a model before starting federated training. We empirically study the impact of starting from a pre-trained model in federated learning using four standard federated learning benchmark datasets. Unsurprisingly, starting from a pre-trained model reduces the training time required to reach a target error rate and enables the training of more accurate models (up to 40\%) than is possible when starting from random initialization. Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity. We recommend that future work proposing and evaluating federated optimization methods evaluate the performance when starting from random and pre-trained initializations. We also believe this study raises several questions for further work on understanding the role of heterogeneity in federated optimization.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | EISSN: 2331-8422 |
ispartof | arXiv.org, 2022-10 |
issn | 2331-8422 |
language | eng |
recordid | cdi_proquest_journals_2725724631 |
source | Open Access: Freely Accessible Journals by multiple vendors |
subjects | Federated learning Heterogeneity Optimization Performance evaluation |
title | Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T09%3A29%3A45IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=Where%20to%20Begin?%20On%20the%20Impact%20of%20Pre-Training%20and%20Initialization%20in%20Federated%20Learning&rft.jtitle=arXiv.org&rft.au=Nguyen,%20John&rft.date=2022-10-14&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2725724631%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2725724631&rft_id=info:pmid/&rfr_iscdi=true |