Multiclass Classification by Sparse Multinomial Logistic Regression
In this paper we consider high-dimensional multiclass classification by sparse multinomial logistic regression. We propose first a feature selection procedure based on penalized maximum likelihood with a complexity penalty on the model size and derive the nonasymptotic bounds for misclassification e...
Gespeichert in:
| Veröffentlicht in: | IEEE transactions on information theory 2021-07, Vol.67 (7), p.4637-4646 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 4646 |
|---|---|
| container_issue | 7 |
| container_start_page | 4637 |
| container_title | IEEE transactions on information theory |
| container_volume | 67 |
| creator | Abramovich, Felix Grinshtein, Vadim Levy, Tomer |
| description | In this paper we consider high-dimensional multiclass classification by sparse multinomial logistic regression. We propose first a feature selection procedure based on penalized maximum likelihood with a complexity penalty on the model size and derive the nonasymptotic bounds for misclassification excess risk of the resulting classifier. We establish also their tightness by deriving the corresponding minimax lower bounds. In particular, we show that there is a phase transition between small and large number of classes. The bounds can be reduced under the additional low noise condition. To find a penalized maximum likelihood solution with a complexity penalty requires, however, a combinatorial search over all possible models. To design a feature selection procedure computationally feasible for high-dimensional data, we propose multinomial logistic group Lasso and Slope classifiers and show that they also achieve the minimax order. |
| doi_str_mv | 10.1109/TIT.2021.3075137 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_ieee_primary_9410597</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>9410597</ieee_id><sourcerecordid>2541475514</sourcerecordid><originalsourceid>FETCH-LOGICAL-c291t-bf26bdc3c1e65f4af40dbc97efa6049e6a78fe73ff1048f35acde742ad31941f3</originalsourceid><addsrcrecordid>eNo9kE1LAzEQhoMoWKt3wcuC562ZfGw2R1m0FiqC1nPIZpOSst3UZHvovze1xcsMA887LzwI3QOeAWD5tFqsZgQTmFEsOFBxgSbAuShlxdklmmAMdSkZq6_RTUqbfDIOZIKa930_etPrlIrmOL3zRo8-DEV7KL52OiZb_DFD2HrdF8uw9iknik-7jjbzYbhFV073yd6d9xR9v76smrdy-TFfNM_L0hAJY9k6UrWdoQZsxR3TjuGuNVJYpyvMpK20qJ0V1DnArHaUa9NZwYjuKEgGjk7R4-nvLoafvU2j2oR9HHKlIpwBE5wDyxQ-USaGlKJ1ahf9VseDAqyOqlRWpY6q1FlVjjycIt5a-4_nTsyloL-KSmXd</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2541475514</pqid></control><display><type>article</type><title>Multiclass Classification by Sparse Multinomial Logistic Regression</title><source>IEEE Electronic Library (IEL)</source><creator>Abramovich, Felix ; Grinshtein, Vadim ; Levy, Tomer</creator><creatorcontrib>Abramovich, Felix ; Grinshtein, Vadim ; Levy, Tomer</creatorcontrib><description>In this paper we consider high-dimensional multiclass classification by sparse multinomial logistic regression. We propose first a feature selection procedure based on penalized maximum likelihood with a complexity penalty on the model size and derive the nonasymptotic bounds for misclassification excess risk of the resulting classifier. We establish also their tightness by deriving the corresponding minimax lower bounds. In particular, we show that there is a phase transition between small and large number of classes. The bounds can be reduced under the additional low noise condition. To find a penalized maximum likelihood solution with a complexity penalty requires, however, a combinatorial search over all possible models. To design a feature selection procedure computationally feasible for high-dimensional data, we propose multinomial logistic group Lasso and Slope classifiers and show that they also achieve the minimax order.</description><identifier>ISSN: 0018-9448</identifier><identifier>EISSN: 1557-9654</identifier><identifier>DOI: 10.1109/TIT.2021.3075137</identifier><identifier>CODEN: IETTAW</identifier><language>eng</language><publisher>New York: IEEE</publisher><subject>Classification ; Classifiers ; Combinatorial analysis ; Complexity ; Complexity penalty ; Complexity theory ; convex relaxation ; Data models ; Feature extraction ; feature selection ; high-dimensionality ; IEEE Sections ; Logistics ; Low noise ; Lower bounds ; Maximum likelihood estimation ; Minimax technique ; minimaxity ; Minimization ; misclassification excess risk ; Noise reduction ; Phase transitions ; sparsity ; Tightness</subject><ispartof>IEEE transactions on information theory, 2021-07, Vol.67 (7), p.4637-4646</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2021</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c291t-bf26bdc3c1e65f4af40dbc97efa6049e6a78fe73ff1048f35acde742ad31941f3</citedby><cites>FETCH-LOGICAL-c291t-bf26bdc3c1e65f4af40dbc97efa6049e6a78fe73ff1048f35acde742ad31941f3</cites><orcidid>0000-0002-1976-6781 ; 0000-0001-7313-1050 ; 0000-0002-4600-6892</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/9410597$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,796,27924,27925,54758</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/9410597$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Abramovich, Felix</creatorcontrib><creatorcontrib>Grinshtein, Vadim</creatorcontrib><creatorcontrib>Levy, Tomer</creatorcontrib><title>Multiclass Classification by Sparse Multinomial Logistic Regression</title><title>IEEE transactions on information theory</title><addtitle>TIT</addtitle><description>In this paper we consider high-dimensional multiclass classification by sparse multinomial logistic regression. We propose first a feature selection procedure based on penalized maximum likelihood with a complexity penalty on the model size and derive the nonasymptotic bounds for misclassification excess risk of the resulting classifier. We establish also their tightness by deriving the corresponding minimax lower bounds. In particular, we show that there is a phase transition between small and large number of classes. The bounds can be reduced under the additional low noise condition. To find a penalized maximum likelihood solution with a complexity penalty requires, however, a combinatorial search over all possible models. To design a feature selection procedure computationally feasible for high-dimensional data, we propose multinomial logistic group Lasso and Slope classifiers and show that they also achieve the minimax order.</description><subject>Classification</subject><subject>Classifiers</subject><subject>Combinatorial analysis</subject><subject>Complexity</subject><subject>Complexity penalty</subject><subject>Complexity theory</subject><subject>convex relaxation</subject><subject>Data models</subject><subject>Feature extraction</subject><subject>feature selection</subject><subject>high-dimensionality</subject><subject>IEEE Sections</subject><subject>Logistics</subject><subject>Low noise</subject><subject>Lower bounds</subject><subject>Maximum likelihood estimation</subject><subject>Minimax technique</subject><subject>minimaxity</subject><subject>Minimization</subject><subject>misclassification excess risk</subject><subject>Noise reduction</subject><subject>Phase transitions</subject><subject>sparsity</subject><subject>Tightness</subject><issn>0018-9448</issn><issn>1557-9654</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNo9kE1LAzEQhoMoWKt3wcuC562ZfGw2R1m0FiqC1nPIZpOSst3UZHvovze1xcsMA887LzwI3QOeAWD5tFqsZgQTmFEsOFBxgSbAuShlxdklmmAMdSkZq6_RTUqbfDIOZIKa930_etPrlIrmOL3zRo8-DEV7KL52OiZb_DFD2HrdF8uw9iknik-7jjbzYbhFV073yd6d9xR9v76smrdy-TFfNM_L0hAJY9k6UrWdoQZsxR3TjuGuNVJYpyvMpK20qJ0V1DnArHaUa9NZwYjuKEgGjk7R4-nvLoafvU2j2oR9HHKlIpwBE5wDyxQ-USaGlKJ1ahf9VseDAqyOqlRWpY6q1FlVjjycIt5a-4_nTsyloL-KSmXd</recordid><startdate>20210701</startdate><enddate>20210701</enddate><creator>Abramovich, Felix</creator><creator>Grinshtein, Vadim</creator><creator>Levy, Tomer</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>7SP</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><orcidid>https://orcid.org/0000-0002-1976-6781</orcidid><orcidid>https://orcid.org/0000-0001-7313-1050</orcidid><orcidid>https://orcid.org/0000-0002-4600-6892</orcidid></search><sort><creationdate>20210701</creationdate><title>Multiclass Classification by Sparse Multinomial Logistic Regression</title><author>Abramovich, Felix ; Grinshtein, Vadim ; Levy, Tomer</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c291t-bf26bdc3c1e65f4af40dbc97efa6049e6a78fe73ff1048f35acde742ad31941f3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Classification</topic><topic>Classifiers</topic><topic>Combinatorial analysis</topic><topic>Complexity</topic><topic>Complexity penalty</topic><topic>Complexity theory</topic><topic>convex relaxation</topic><topic>Data models</topic><topic>Feature extraction</topic><topic>feature selection</topic><topic>high-dimensionality</topic><topic>IEEE Sections</topic><topic>Logistics</topic><topic>Low noise</topic><topic>Lower bounds</topic><topic>Maximum likelihood estimation</topic><topic>Minimax technique</topic><topic>minimaxity</topic><topic>Minimization</topic><topic>misclassification excess risk</topic><topic>Noise reduction</topic><topic>Phase transitions</topic><topic>sparsity</topic><topic>Tightness</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Abramovich, Felix</creatorcontrib><creatorcontrib>Grinshtein, Vadim</creatorcontrib><creatorcontrib>Levy, Tomer</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Electronics & Communications Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><jtitle>IEEE transactions on information theory</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Abramovich, Felix</au><au>Grinshtein, Vadim</au><au>Levy, Tomer</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Multiclass Classification by Sparse Multinomial Logistic Regression</atitle><jtitle>IEEE transactions on information theory</jtitle><stitle>TIT</stitle><date>2021-07-01</date><risdate>2021</risdate><volume>67</volume><issue>7</issue><spage>4637</spage><epage>4646</epage><pages>4637-4646</pages><issn>0018-9448</issn><eissn>1557-9654</eissn><coden>IETTAW</coden><abstract>In this paper we consider high-dimensional multiclass classification by sparse multinomial logistic regression. We propose first a feature selection procedure based on penalized maximum likelihood with a complexity penalty on the model size and derive the nonasymptotic bounds for misclassification excess risk of the resulting classifier. We establish also their tightness by deriving the corresponding minimax lower bounds. In particular, we show that there is a phase transition between small and large number of classes. The bounds can be reduced under the additional low noise condition. To find a penalized maximum likelihood solution with a complexity penalty requires, however, a combinatorial search over all possible models. To design a feature selection procedure computationally feasible for high-dimensional data, we propose multinomial logistic group Lasso and Slope classifiers and show that they also achieve the minimax order.</abstract><cop>New York</cop><pub>IEEE</pub><doi>10.1109/TIT.2021.3075137</doi><tpages>10</tpages><orcidid>https://orcid.org/0000-0002-1976-6781</orcidid><orcidid>https://orcid.org/0000-0001-7313-1050</orcidid><orcidid>https://orcid.org/0000-0002-4600-6892</orcidid></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | ISSN: 0018-9448 |
| ispartof | IEEE transactions on information theory, 2021-07, Vol.67 (7), p.4637-4646 |
| issn | 0018-9448 1557-9654 |
| language | eng |
| recordid | cdi_ieee_primary_9410597 |
| source | IEEE Electronic Library (IEL) |
| subjects | Classification Classifiers Combinatorial analysis Complexity Complexity penalty Complexity theory convex relaxation Data models Feature extraction feature selection high-dimensionality IEEE Sections Logistics Low noise Lower bounds Maximum likelihood estimation Minimax technique minimaxity Minimization misclassification excess risk Noise reduction Phase transitions sparsity Tightness |
| title | Multiclass Classification by Sparse Multinomial Logistic Regression |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-26T10%3A57%3A28IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Multiclass%20Classification%20by%20Sparse%20Multinomial%20Logistic%20Regression&rft.jtitle=IEEE%20transactions%20on%20information%20theory&rft.au=Abramovich,%20Felix&rft.date=2021-07-01&rft.volume=67&rft.issue=7&rft.spage=4637&rft.epage=4646&rft.pages=4637-4646&rft.issn=0018-9448&rft.eissn=1557-9654&rft.coden=IETTAW&rft_id=info:doi/10.1109/TIT.2021.3075137&rft_dat=%3Cproquest_RIE%3E2541475514%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2541475514&rft_id=info:pmid/&rft_ieee_id=9410597&rfr_iscdi=true |