Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity
Vision-language pre-training (VLP) has emerged as an effective scheme for multimodal representation learning, but its reliance on large-scale multimodal data poses significant challenges for medical applications. Federated learning (FL) offers a promising solution to scale up the dataset for medical...
Gespeichert in:
Hauptverfasser: | , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | |
container_volume | |
creator | Shuai, Zitao Wu, Chenwei Tang, Zhengxu Shen, Liyue |
description | Vision-language pre-training (VLP) has emerged as an effective scheme for
multimodal representation learning, but its reliance on large-scale multimodal
data poses significant challenges for medical applications. Federated learning
(FL) offers a promising solution to scale up the dataset for medical VLP while
preserving data privacy. However, we observe that client data heterogeneity in
real-world scenarios could cause models to learn biased cross-modal alignment
during local pre-training. This would limit the transferability of the
federally learned representation model on downstream tasks. To address this
challenge, we propose Federated Distributionally Robust Alignment (FedDRA), a
framework for federated VLP that achieves robust vision-language alignment
under heterogeneous conditions. Based on client datasets, we construct a
distribution family that encompasses potential test-time domains, and apply a
distributionally robust framework to optimize the pre-trained model's
performance across this distribution space. This approach bridges the gap
between pre-training samples and downstream applications. To avoid over-fitting
on client-specific information, we use anchor representation from the global
model to guide the local training, and adopt a two-stage approach to first tune
deeper layers before updating the entire network. Extensive experiments on
real-world datasets demonstrate FedDRA's effectiveness in enhancing medical
federated VLP under data heterogeneity. Our method also adapts well to various
medical pre-training methods. |
doi_str_mv | 10.48550/arxiv.2404.03854 |
format | Article |
fullrecord | <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2404_03854</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2404_03854</sourcerecordid><originalsourceid>FETCH-arxiv_primary_2404_038543</originalsourceid><addsrcrecordid>eNqFzrsKwkAQheFtLER9ACvnBRKjScBWvGChIKK2YWImy8BmI5OJmLf3gr3Vaf4DnzHjWRQmizSNpihPfoTzJErCKF6kSd9Ua25UOG-Va4_OdXCq87ZRWDq2viKvUNYCByr4hg62VJCgUgFXbt6PYI_etmgJjkKBCrJnb-Hi3xmsURF2pCS1JU-s3dD0SnQNjX47MJPt5rzaBV9YdheuULrsA8y-wPh_8QKZCkfS</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity</title><source>arXiv.org</source><creator>Shuai, Zitao ; Wu, Chenwei ; Tang, Zhengxu ; Shen, Liyue</creator><creatorcontrib>Shuai, Zitao ; Wu, Chenwei ; Tang, Zhengxu ; Shen, Liyue</creatorcontrib><description>Vision-language pre-training (VLP) has emerged as an effective scheme for
multimodal representation learning, but its reliance on large-scale multimodal
data poses significant challenges for medical applications. Federated learning
(FL) offers a promising solution to scale up the dataset for medical VLP while
preserving data privacy. However, we observe that client data heterogeneity in
real-world scenarios could cause models to learn biased cross-modal alignment
during local pre-training. This would limit the transferability of the
federally learned representation model on downstream tasks. To address this
challenge, we propose Federated Distributionally Robust Alignment (FedDRA), a
framework for federated VLP that achieves robust vision-language alignment
under heterogeneous conditions. Based on client datasets, we construct a
distribution family that encompasses potential test-time domains, and apply a
distributionally robust framework to optimize the pre-trained model's
performance across this distribution space. This approach bridges the gap
between pre-training samples and downstream applications. To avoid over-fitting
on client-specific information, we use anchor representation from the global
model to guide the local training, and adopt a two-stage approach to first tune
deeper layers before updating the entire network. Extensive experiments on
real-world datasets demonstrate FedDRA's effectiveness in enhancing medical
federated VLP under data heterogeneity. Our method also adapts well to various
medical pre-training methods.</description><identifier>DOI: 10.48550/arxiv.2404.03854</identifier><language>eng</language><subject>Computer Science - Computation and Language ; Computer Science - Computer Vision and Pattern Recognition ; Computer Science - Learning</subject><creationdate>2024-04</creationdate><rights>http://creativecommons.org/licenses/by-nc-nd/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2404.03854$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2404.03854$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Shuai, Zitao</creatorcontrib><creatorcontrib>Wu, Chenwei</creatorcontrib><creatorcontrib>Tang, Zhengxu</creatorcontrib><creatorcontrib>Shen, Liyue</creatorcontrib><title>Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity</title><description>Vision-language pre-training (VLP) has emerged as an effective scheme for
multimodal representation learning, but its reliance on large-scale multimodal
data poses significant challenges for medical applications. Federated learning
(FL) offers a promising solution to scale up the dataset for medical VLP while
preserving data privacy. However, we observe that client data heterogeneity in
real-world scenarios could cause models to learn biased cross-modal alignment
during local pre-training. This would limit the transferability of the
federally learned representation model on downstream tasks. To address this
challenge, we propose Federated Distributionally Robust Alignment (FedDRA), a
framework for federated VLP that achieves robust vision-language alignment
under heterogeneous conditions. Based on client datasets, we construct a
distribution family that encompasses potential test-time domains, and apply a
distributionally robust framework to optimize the pre-trained model's
performance across this distribution space. This approach bridges the gap
between pre-training samples and downstream applications. To avoid over-fitting
on client-specific information, we use anchor representation from the global
model to guide the local training, and adopt a two-stage approach to first tune
deeper layers before updating the entire network. Extensive experiments on
real-world datasets demonstrate FedDRA's effectiveness in enhancing medical
federated VLP under data heterogeneity. Our method also adapts well to various
medical pre-training methods.</description><subject>Computer Science - Computation and Language</subject><subject>Computer Science - Computer Vision and Pattern Recognition</subject><subject>Computer Science - Learning</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNqFzrsKwkAQheFtLER9ACvnBRKjScBWvGChIKK2YWImy8BmI5OJmLf3gr3Vaf4DnzHjWRQmizSNpihPfoTzJErCKF6kSd9Ua25UOG-Va4_OdXCq87ZRWDq2viKvUNYCByr4hg62VJCgUgFXbt6PYI_etmgJjkKBCrJnb-Hi3xmsURF2pCS1JU-s3dD0SnQNjX47MJPt5rzaBV9YdheuULrsA8y-wPh_8QKZCkfS</recordid><startdate>20240404</startdate><enddate>20240404</enddate><creator>Shuai, Zitao</creator><creator>Wu, Chenwei</creator><creator>Tang, Zhengxu</creator><creator>Shen, Liyue</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20240404</creationdate><title>Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity</title><author>Shuai, Zitao ; Wu, Chenwei ; Tang, Zhengxu ; Shen, Liyue</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-arxiv_primary_2404_038543</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Science - Computation and Language</topic><topic>Computer Science - Computer Vision and Pattern Recognition</topic><topic>Computer Science - Learning</topic><toplevel>online_resources</toplevel><creatorcontrib>Shuai, Zitao</creatorcontrib><creatorcontrib>Wu, Chenwei</creatorcontrib><creatorcontrib>Tang, Zhengxu</creatorcontrib><creatorcontrib>Shen, Liyue</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Shuai, Zitao</au><au>Wu, Chenwei</au><au>Tang, Zhengxu</au><au>Shen, Liyue</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity</atitle><date>2024-04-04</date><risdate>2024</risdate><abstract>Vision-language pre-training (VLP) has emerged as an effective scheme for
multimodal representation learning, but its reliance on large-scale multimodal
data poses significant challenges for medical applications. Federated learning
(FL) offers a promising solution to scale up the dataset for medical VLP while
preserving data privacy. However, we observe that client data heterogeneity in
real-world scenarios could cause models to learn biased cross-modal alignment
during local pre-training. This would limit the transferability of the
federally learned representation model on downstream tasks. To address this
challenge, we propose Federated Distributionally Robust Alignment (FedDRA), a
framework for federated VLP that achieves robust vision-language alignment
under heterogeneous conditions. Based on client datasets, we construct a
distribution family that encompasses potential test-time domains, and apply a
distributionally robust framework to optimize the pre-trained model's
performance across this distribution space. This approach bridges the gap
between pre-training samples and downstream applications. To avoid over-fitting
on client-specific information, we use anchor representation from the global
model to guide the local training, and adopt a two-stage approach to first tune
deeper layers before updating the entire network. Extensive experiments on
real-world datasets demonstrate FedDRA's effectiveness in enhancing medical
federated VLP under data heterogeneity. Our method also adapts well to various
medical pre-training methods.</abstract><doi>10.48550/arxiv.2404.03854</doi><oa>free_for_read</oa></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | DOI: 10.48550/arxiv.2404.03854 |
ispartof | |
issn | |
language | eng |
recordid | cdi_arxiv_primary_2404_03854 |
source | arXiv.org |
subjects | Computer Science - Computation and Language Computer Science - Computer Vision and Pattern Recognition Computer Science - Learning |
title | Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T23%3A35%3A47IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Distributionally%20Robust%20Alignment%20for%20Medical%20Federated%20Vision-Language%20Pre-training%20Under%20Data%20Heterogeneity&rft.au=Shuai,%20Zitao&rft.date=2024-04-04&rft_id=info:doi/10.48550/arxiv.2404.03854&rft_dat=%3Carxiv_GOX%3E2404_03854%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |