Projective independence tests in high dimensions: the curses and the cures

Summary Testing independence between high-dimensional random vectors is fundamentally different from testing independence between univariate random variables. Taking the projection correlation as an example, it suffers from at least three problems. First, it has a high computational complexity of O{...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Biometrika 2024-09, Vol.111 (3), p.1013-1027
Hauptverfasser: Zhang, Yaowu, Zhu, Liping
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 1027
container_issue 3
container_start_page 1013
container_title Biometrika
container_volume 111
creator Zhang, Yaowu
Zhu, Liping
description Summary Testing independence between high-dimensional random vectors is fundamentally different from testing independence between univariate random variables. Taking the projection correlation as an example, it suffers from at least three problems. First, it has a high computational complexity of O{n3(p+q)}, where n, p and q are the sample size and dimensions of the random vectors; this limits its usefulness substantially when n is extremely large. Second, the asymptotic null distribution of the projection correlation test is rarely tractable; therefore, random permutations are often suggested as a means of approximating the asymptotic null distribution, which further increases the complexity of implementing independence tests. Third, the power performance of the projection correlation test deteriorates in high dimensions. To address these issues, the projection correlation is improved by using a modified weight function, which reduces the complexity to O{n2(p+q)}. We estimate the improved projection correlation with U-statistic theory. Importantly, its asymptotic null distribution is standard normal, thanks to the high dimesnionality of the random vectors. This expedites the implementation of independence tests substantially. To enhance the power performance in high dimensions, we propose incorporating a cross-validation procedure with feature screening into the projection correlation test. The implementation efficacy and power enhancement are confirmed through extensive numerical studies.
doi_str_mv 10.1093/biomet/asad070
format Article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_3103008823</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><oup_id>10.1093/biomet/asad070</oup_id><sourcerecordid>3103008823</sourcerecordid><originalsourceid>FETCH-LOGICAL-c231t-23aa60200b0deb000ba579647d103d9084a29b590347c945e7980a292e536e1d3</originalsourceid><addsrcrecordid>eNqFUD1PwzAUtBBIlMLKbImJIe1z7DgxG6r4VCUYYLac-JW4onGwEyT-Pa5SZpb39E53905HyCWDBQPFl7XzOxyWJhoLJRyRGRNSZLxgcExmACAzLoQ4JWcxbvenLOSMPL8Gv8VmcN9IXWexxzS6BumAcYgJoq37aKl1O-yi8128oUOLtBlDxEhNZ_9OjOfkZGM-I14c9py839-9rR6z9cvD0-p2nTU5Z0OWc2Mk5AA1WKxTkNoUpZKitAy4VVAJk6u6UMBF2ShRYKkqSFCOBZfILJ-Tq8m3D_5rTDH11o-hSy81TxYAVZXzxFpMrCb4GANudB_czoQfzUDv-9JTX_rQVxJcTwI_9v9xfwH2OG1N</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3103008823</pqid></control><display><type>article</type><title>Projective independence tests in high dimensions: the curses and the cures</title><source>Oxford University Press Journals All Titles (1996-Current)</source><creator>Zhang, Yaowu ; Zhu, Liping</creator><creatorcontrib>Zhang, Yaowu ; Zhu, Liping</creatorcontrib><description>Summary Testing independence between high-dimensional random vectors is fundamentally different from testing independence between univariate random variables. Taking the projection correlation as an example, it suffers from at least three problems. First, it has a high computational complexity of O{n3(p+q)}, where n, p and q are the sample size and dimensions of the random vectors; this limits its usefulness substantially when n is extremely large. Second, the asymptotic null distribution of the projection correlation test is rarely tractable; therefore, random permutations are often suggested as a means of approximating the asymptotic null distribution, which further increases the complexity of implementing independence tests. Third, the power performance of the projection correlation test deteriorates in high dimensions. To address these issues, the projection correlation is improved by using a modified weight function, which reduces the complexity to O{n2(p+q)}. We estimate the improved projection correlation with U-statistic theory. Importantly, its asymptotic null distribution is standard normal, thanks to the high dimesnionality of the random vectors. This expedites the implementation of independence tests substantially. To enhance the power performance in high dimensions, we propose incorporating a cross-validation procedure with feature screening into the projection correlation test. The implementation efficacy and power enhancement are confirmed through extensive numerical studies.</description><identifier>ISSN: 0006-3444</identifier><identifier>EISSN: 1464-3510</identifier><identifier>DOI: 10.1093/biomet/asad070</identifier><language>eng</language><publisher>Oxford: Oxford University Press</publisher><subject>Asymptotic methods ; Asymptotic properties ; Complexity ; Correlation ; Electric power distribution ; Permutations ; Random variables ; Weighting functions</subject><ispartof>Biometrika, 2024-09, Vol.111 (3), p.1013-1027</ispartof><rights>The Author(s) 2023. Published by Oxford University Press on behalf of the Biometrika Trust. All rights reserved. For permissions, please email: journals.permissions@oup.com 2023</rights><rights>The Author(s) 2023. Published by Oxford University Press on behalf of the Biometrika Trust. All rights reserved. For permissions, please email: journals.permissions@oup.com</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c231t-23aa60200b0deb000ba579647d103d9084a29b590347c945e7980a292e536e1d3</citedby><cites>FETCH-LOGICAL-c231t-23aa60200b0deb000ba579647d103d9084a29b590347c945e7980a292e536e1d3</cites><orcidid>0000-0001-8926-9791</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,780,784,1584,27924,27925</link.rule.ids></links><search><creatorcontrib>Zhang, Yaowu</creatorcontrib><creatorcontrib>Zhu, Liping</creatorcontrib><title>Projective independence tests in high dimensions: the curses and the cures</title><title>Biometrika</title><description>Summary Testing independence between high-dimensional random vectors is fundamentally different from testing independence between univariate random variables. Taking the projection correlation as an example, it suffers from at least three problems. First, it has a high computational complexity of O{n3(p+q)}, where n, p and q are the sample size and dimensions of the random vectors; this limits its usefulness substantially when n is extremely large. Second, the asymptotic null distribution of the projection correlation test is rarely tractable; therefore, random permutations are often suggested as a means of approximating the asymptotic null distribution, which further increases the complexity of implementing independence tests. Third, the power performance of the projection correlation test deteriorates in high dimensions. To address these issues, the projection correlation is improved by using a modified weight function, which reduces the complexity to O{n2(p+q)}. We estimate the improved projection correlation with U-statistic theory. Importantly, its asymptotic null distribution is standard normal, thanks to the high dimesnionality of the random vectors. This expedites the implementation of independence tests substantially. To enhance the power performance in high dimensions, we propose incorporating a cross-validation procedure with feature screening into the projection correlation test. The implementation efficacy and power enhancement are confirmed through extensive numerical studies.</description><subject>Asymptotic methods</subject><subject>Asymptotic properties</subject><subject>Complexity</subject><subject>Correlation</subject><subject>Electric power distribution</subject><subject>Permutations</subject><subject>Random variables</subject><subject>Weighting functions</subject><issn>0006-3444</issn><issn>1464-3510</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><recordid>eNqFUD1PwzAUtBBIlMLKbImJIe1z7DgxG6r4VCUYYLac-JW4onGwEyT-Pa5SZpb39E53905HyCWDBQPFl7XzOxyWJhoLJRyRGRNSZLxgcExmACAzLoQ4JWcxbvenLOSMPL8Gv8VmcN9IXWexxzS6BumAcYgJoq37aKl1O-yi8128oUOLtBlDxEhNZ_9OjOfkZGM-I14c9py839-9rR6z9cvD0-p2nTU5Z0OWc2Mk5AA1WKxTkNoUpZKitAy4VVAJk6u6UMBF2ShRYKkqSFCOBZfILJ-Tq8m3D_5rTDH11o-hSy81TxYAVZXzxFpMrCb4GANudB_czoQfzUDv-9JTX_rQVxJcTwI_9v9xfwH2OG1N</recordid><startdate>20240901</startdate><enddate>20240901</enddate><creator>Zhang, Yaowu</creator><creator>Zhu, Liping</creator><general>Oxford University Press</general><general>Oxford Publishing Limited (England)</general><scope>AAYXX</scope><scope>CITATION</scope><scope>7QO</scope><scope>8FD</scope><scope>FR3</scope><scope>P64</scope><orcidid>https://orcid.org/0000-0001-8926-9791</orcidid></search><sort><creationdate>20240901</creationdate><title>Projective independence tests in high dimensions: the curses and the cures</title><author>Zhang, Yaowu ; Zhu, Liping</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c231t-23aa60200b0deb000ba579647d103d9084a29b590347c945e7980a292e536e1d3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Asymptotic methods</topic><topic>Asymptotic properties</topic><topic>Complexity</topic><topic>Correlation</topic><topic>Electric power distribution</topic><topic>Permutations</topic><topic>Random variables</topic><topic>Weighting functions</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Zhang, Yaowu</creatorcontrib><creatorcontrib>Zhu, Liping</creatorcontrib><collection>CrossRef</collection><collection>Biotechnology Research Abstracts</collection><collection>Technology Research Database</collection><collection>Engineering Research Database</collection><collection>Biotechnology and BioEngineering Abstracts</collection><jtitle>Biometrika</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Zhang, Yaowu</au><au>Zhu, Liping</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Projective independence tests in high dimensions: the curses and the cures</atitle><jtitle>Biometrika</jtitle><date>2024-09-01</date><risdate>2024</risdate><volume>111</volume><issue>3</issue><spage>1013</spage><epage>1027</epage><pages>1013-1027</pages><issn>0006-3444</issn><eissn>1464-3510</eissn><abstract>Summary Testing independence between high-dimensional random vectors is fundamentally different from testing independence between univariate random variables. Taking the projection correlation as an example, it suffers from at least three problems. First, it has a high computational complexity of O{n3(p+q)}, where n, p and q are the sample size and dimensions of the random vectors; this limits its usefulness substantially when n is extremely large. Second, the asymptotic null distribution of the projection correlation test is rarely tractable; therefore, random permutations are often suggested as a means of approximating the asymptotic null distribution, which further increases the complexity of implementing independence tests. Third, the power performance of the projection correlation test deteriorates in high dimensions. To address these issues, the projection correlation is improved by using a modified weight function, which reduces the complexity to O{n2(p+q)}. We estimate the improved projection correlation with U-statistic theory. Importantly, its asymptotic null distribution is standard normal, thanks to the high dimesnionality of the random vectors. This expedites the implementation of independence tests substantially. To enhance the power performance in high dimensions, we propose incorporating a cross-validation procedure with feature screening into the projection correlation test. The implementation efficacy and power enhancement are confirmed through extensive numerical studies.</abstract><cop>Oxford</cop><pub>Oxford University Press</pub><doi>10.1093/biomet/asad070</doi><tpages>15</tpages><orcidid>https://orcid.org/0000-0001-8926-9791</orcidid></addata></record>
fulltext fulltext
identifier ISSN: 0006-3444
ispartof Biometrika, 2024-09, Vol.111 (3), p.1013-1027
issn 0006-3444
1464-3510
language eng
recordid cdi_proquest_journals_3103008823
source Oxford University Press Journals All Titles (1996-Current)
subjects Asymptotic methods
Asymptotic properties
Complexity
Correlation
Electric power distribution
Permutations
Random variables
Weighting functions
title Projective independence tests in high dimensions: the curses and the cures
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T23%3A57%3A36IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Projective%20independence%20tests%20in%20high%20dimensions:%20the%20curses%20and%20the%20cures&rft.jtitle=Biometrika&rft.au=Zhang,%20Yaowu&rft.date=2024-09-01&rft.volume=111&rft.issue=3&rft.spage=1013&rft.epage=1027&rft.pages=1013-1027&rft.issn=0006-3444&rft.eissn=1464-3510&rft_id=info:doi/10.1093/biomet/asad070&rft_dat=%3Cproquest_cross%3E3103008823%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3103008823&rft_id=info:pmid/&rft_oup_id=10.1093/biomet/asad070&rfr_iscdi=true