Multi-Granularity Anchor-Contrastive Representation Learning for Semi-Supervised Skeleton-Based Action Recognition

In the semi-supervised skeleton-based action recognition task, obtaining more discriminative information from both labeled and unlabeled data is a challenging problem. As the current mainstream approach, contrastive learning can learn more representations of augmented data, which can be considered a...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on pattern analysis and machine intelligence 2023-06, Vol.45 (6), p.7559-7576
Hauptverfasser: Shu, Xiangbo, Xu, Binqian, Zhang, Liyan, Tang, Jinhui
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 7576
container_issue 6
container_start_page 7559
container_title IEEE transactions on pattern analysis and machine intelligence
container_volume 45
creator Shu, Xiangbo
Xu, Binqian
Zhang, Liyan
Tang, Jinhui
description In the semi-supervised skeleton-based action recognition task, obtaining more discriminative information from both labeled and unlabeled data is a challenging problem. As the current mainstream approach, contrastive learning can learn more representations of augmented data, which can be considered as the pretext task of action recognition. However, such a method still confronts three main limitations: 1) It usually learns global-granularity features that cannot well reflect the local motion information. 2) The positive/negative pairs are usually pre-defined, some of which are ambiguous. 3) It generally measures the distance between positive/negative pairs only within the same granularity, which neglects the contrasting between the cross-granularity positive and negative pairs. Toward these limitations, we propose a novel Multi-granularity Anchor-Contrastive representation Learning (dubbed as MAC-Learning) to learn multi-granularity representations by conducting inter- and intra-granularity contrastive pretext tasks on the learnable and structural-link skeletons among three types of granularities covering local, context, and global views. To avoid the disturbance of ambiguous pairs from noisy and outlier samples, we design a more reliable Multi-granularity Anchor-Contrastive Loss (dubbed as MAC-Loss) that measures the agreement/disagreement between high-confidence soft-positive/negative pairs based on the anchor graph instead of the hard-positive/negative pairs in the conventional contrastive loss. Extensive experiments on both NTU RGB+D and Northwestern-UCLA datasets show that the proposed MAC-Learning outperforms existing competitive methods in semi-supervised skeleton-based action recognition tasks.
doi_str_mv 10.1109/TPAMI.2022.3222871
format Article
fullrecord <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_pubmed_primary_36395133</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>9954217</ieee_id><sourcerecordid>2738191352</sourcerecordid><originalsourceid>FETCH-LOGICAL-c351t-49b2b526f97154da0dbfbd63b8f89e03a620717594dde3d704ea4e8cf04303953</originalsourceid><addsrcrecordid>eNpdkcFuGyEQhlGVqnHTvkAjRSv10gsuMMsuHB2rTSM5ahWnZ8TuzqYka3CAtZS37zp2c-iJQXz_DKOPkE-czTln-uvdr8XN9VwwIeYghFA1f0NmXIOmIEGfkBnjlaBKCXVK3qf0wBgvJYN35BQq0JIDzEi8GYfs6FW0fhxsdPm5WPj2T4h0GXyONmW3w-IWtxET-myzC75YoY3e-fuiD7FY48bR9bjFuHMJu2L9iAPm4Oml3V8X7UvkFttw792-_kDe9nZI-PF4npHf37_dLX_Q1c-r6-ViRVuQPNNSN6KRoup1zWXZWdY1fdNV0KheaWRgK8FqXktddh1CV7MSbYmq7VkJbFoPzsiXQ99tDE8jpmw2LrU4DNZjGJMRNSiuOUgxoZ__Qx_CGP30OyMU00qpacxEiQPVxpBSxN5so9vY-Gw4M3sj5sWI2RsxRyNT6OLYemw22L1G_imYgPMD4BDx9VlrWQpew19r8pCr</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2809888594</pqid></control><display><type>article</type><title>Multi-Granularity Anchor-Contrastive Representation Learning for Semi-Supervised Skeleton-Based Action Recognition</title><source>IEEE Electronic Library (IEL)</source><creator>Shu, Xiangbo ; Xu, Binqian ; Zhang, Liyan ; Tang, Jinhui</creator><creatorcontrib>Shu, Xiangbo ; Xu, Binqian ; Zhang, Liyan ; Tang, Jinhui</creatorcontrib><description>In the semi-supervised skeleton-based action recognition task, obtaining more discriminative information from both labeled and unlabeled data is a challenging problem. As the current mainstream approach, contrastive learning can learn more representations of augmented data, which can be considered as the pretext task of action recognition. However, such a method still confronts three main limitations: 1) It usually learns global-granularity features that cannot well reflect the local motion information. 2) The positive/negative pairs are usually pre-defined, some of which are ambiguous. 3) It generally measures the distance between positive/negative pairs only within the same granularity, which neglects the contrasting between the cross-granularity positive and negative pairs. Toward these limitations, we propose a novel Multi-granularity Anchor-Contrastive representation Learning (dubbed as MAC-Learning) to learn multi-granularity representations by conducting inter- and intra-granularity contrastive pretext tasks on the learnable and structural-link skeletons among three types of granularities covering local, context, and global views. To avoid the disturbance of ambiguous pairs from noisy and outlier samples, we design a more reliable Multi-granularity Anchor-Contrastive Loss (dubbed as MAC-Loss) that measures the agreement/disagreement between high-confidence soft-positive/negative pairs based on the anchor graph instead of the hard-positive/negative pairs in the conventional contrastive loss. Extensive experiments on both NTU RGB+D and Northwestern-UCLA datasets show that the proposed MAC-Learning outperforms existing competitive methods in semi-supervised skeleton-based action recognition tasks.</description><identifier>ISSN: 0162-8828</identifier><identifier>EISSN: 1939-3539</identifier><identifier>EISSN: 2160-9292</identifier><identifier>DOI: 10.1109/TPAMI.2022.3222871</identifier><identifier>PMID: 36395133</identifier><identifier>CODEN: ITPIDJ</identifier><language>eng</language><publisher>United States: IEEE</publisher><subject>Action recognition ; Activity recognition ; anchor graph ; contrastive learning ; Data augmentation ; Data models ; Joints ; Loss measurement ; Machine learning ; Outliers (statistics) ; Pattern recognition ; Representations ; Semantics ; semi-supervised ; Skeleton ; Task analysis</subject><ispartof>IEEE transactions on pattern analysis and machine intelligence, 2023-06, Vol.45 (6), p.7559-7576</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c351t-49b2b526f97154da0dbfbd63b8f89e03a620717594dde3d704ea4e8cf04303953</citedby><cites>FETCH-LOGICAL-c351t-49b2b526f97154da0dbfbd63b8f89e03a620717594dde3d704ea4e8cf04303953</cites><orcidid>0000-0003-4902-4663 ; 0000-0001-9008-222X ; 0000-0002-1549-3317 ; 0000-0003-3904-1400</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/9954217$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,796,27924,27925,54758</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/9954217$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/36395133$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Shu, Xiangbo</creatorcontrib><creatorcontrib>Xu, Binqian</creatorcontrib><creatorcontrib>Zhang, Liyan</creatorcontrib><creatorcontrib>Tang, Jinhui</creatorcontrib><title>Multi-Granularity Anchor-Contrastive Representation Learning for Semi-Supervised Skeleton-Based Action Recognition</title><title>IEEE transactions on pattern analysis and machine intelligence</title><addtitle>TPAMI</addtitle><addtitle>IEEE Trans Pattern Anal Mach Intell</addtitle><description>In the semi-supervised skeleton-based action recognition task, obtaining more discriminative information from both labeled and unlabeled data is a challenging problem. As the current mainstream approach, contrastive learning can learn more representations of augmented data, which can be considered as the pretext task of action recognition. However, such a method still confronts three main limitations: 1) It usually learns global-granularity features that cannot well reflect the local motion information. 2) The positive/negative pairs are usually pre-defined, some of which are ambiguous. 3) It generally measures the distance between positive/negative pairs only within the same granularity, which neglects the contrasting between the cross-granularity positive and negative pairs. Toward these limitations, we propose a novel Multi-granularity Anchor-Contrastive representation Learning (dubbed as MAC-Learning) to learn multi-granularity representations by conducting inter- and intra-granularity contrastive pretext tasks on the learnable and structural-link skeletons among three types of granularities covering local, context, and global views. To avoid the disturbance of ambiguous pairs from noisy and outlier samples, we design a more reliable Multi-granularity Anchor-Contrastive Loss (dubbed as MAC-Loss) that measures the agreement/disagreement between high-confidence soft-positive/negative pairs based on the anchor graph instead of the hard-positive/negative pairs in the conventional contrastive loss. Extensive experiments on both NTU RGB+D and Northwestern-UCLA datasets show that the proposed MAC-Learning outperforms existing competitive methods in semi-supervised skeleton-based action recognition tasks.</description><subject>Action recognition</subject><subject>Activity recognition</subject><subject>anchor graph</subject><subject>contrastive learning</subject><subject>Data augmentation</subject><subject>Data models</subject><subject>Joints</subject><subject>Loss measurement</subject><subject>Machine learning</subject><subject>Outliers (statistics)</subject><subject>Pattern recognition</subject><subject>Representations</subject><subject>Semantics</subject><subject>semi-supervised</subject><subject>Skeleton</subject><subject>Task analysis</subject><issn>0162-8828</issn><issn>1939-3539</issn><issn>2160-9292</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNpdkcFuGyEQhlGVqnHTvkAjRSv10gsuMMsuHB2rTSM5ahWnZ8TuzqYka3CAtZS37zp2c-iJQXz_DKOPkE-czTln-uvdr8XN9VwwIeYghFA1f0NmXIOmIEGfkBnjlaBKCXVK3qf0wBgvJYN35BQq0JIDzEi8GYfs6FW0fhxsdPm5WPj2T4h0GXyONmW3w-IWtxET-myzC75YoY3e-fuiD7FY48bR9bjFuHMJu2L9iAPm4Oml3V8X7UvkFttw792-_kDe9nZI-PF4npHf37_dLX_Q1c-r6-ViRVuQPNNSN6KRoup1zWXZWdY1fdNV0KheaWRgK8FqXktddh1CV7MSbYmq7VkJbFoPzsiXQ99tDE8jpmw2LrU4DNZjGJMRNSiuOUgxoZ__Qx_CGP30OyMU00qpacxEiQPVxpBSxN5so9vY-Gw4M3sj5sWI2RsxRyNT6OLYemw22L1G_imYgPMD4BDx9VlrWQpew19r8pCr</recordid><startdate>20230601</startdate><enddate>20230601</enddate><creator>Shu, Xiangbo</creator><creator>Xu, Binqian</creator><creator>Zhang, Liyan</creator><creator>Tang, Jinhui</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>7SP</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>7X8</scope><orcidid>https://orcid.org/0000-0003-4902-4663</orcidid><orcidid>https://orcid.org/0000-0001-9008-222X</orcidid><orcidid>https://orcid.org/0000-0002-1549-3317</orcidid><orcidid>https://orcid.org/0000-0003-3904-1400</orcidid></search><sort><creationdate>20230601</creationdate><title>Multi-Granularity Anchor-Contrastive Representation Learning for Semi-Supervised Skeleton-Based Action Recognition</title><author>Shu, Xiangbo ; Xu, Binqian ; Zhang, Liyan ; Tang, Jinhui</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c351t-49b2b526f97154da0dbfbd63b8f89e03a620717594dde3d704ea4e8cf04303953</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Action recognition</topic><topic>Activity recognition</topic><topic>anchor graph</topic><topic>contrastive learning</topic><topic>Data augmentation</topic><topic>Data models</topic><topic>Joints</topic><topic>Loss measurement</topic><topic>Machine learning</topic><topic>Outliers (statistics)</topic><topic>Pattern recognition</topic><topic>Representations</topic><topic>Semantics</topic><topic>semi-supervised</topic><topic>Skeleton</topic><topic>Task analysis</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Shu, Xiangbo</creatorcontrib><creatorcontrib>Xu, Binqian</creatorcontrib><creatorcontrib>Zhang, Liyan</creatorcontrib><creatorcontrib>Tang, Jinhui</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Electronics &amp; Communications Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>MEDLINE - Academic</collection><jtitle>IEEE transactions on pattern analysis and machine intelligence</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Shu, Xiangbo</au><au>Xu, Binqian</au><au>Zhang, Liyan</au><au>Tang, Jinhui</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Multi-Granularity Anchor-Contrastive Representation Learning for Semi-Supervised Skeleton-Based Action Recognition</atitle><jtitle>IEEE transactions on pattern analysis and machine intelligence</jtitle><stitle>TPAMI</stitle><addtitle>IEEE Trans Pattern Anal Mach Intell</addtitle><date>2023-06-01</date><risdate>2023</risdate><volume>45</volume><issue>6</issue><spage>7559</spage><epage>7576</epage><pages>7559-7576</pages><issn>0162-8828</issn><eissn>1939-3539</eissn><eissn>2160-9292</eissn><coden>ITPIDJ</coden><abstract>In the semi-supervised skeleton-based action recognition task, obtaining more discriminative information from both labeled and unlabeled data is a challenging problem. As the current mainstream approach, contrastive learning can learn more representations of augmented data, which can be considered as the pretext task of action recognition. However, such a method still confronts three main limitations: 1) It usually learns global-granularity features that cannot well reflect the local motion information. 2) The positive/negative pairs are usually pre-defined, some of which are ambiguous. 3) It generally measures the distance between positive/negative pairs only within the same granularity, which neglects the contrasting between the cross-granularity positive and negative pairs. Toward these limitations, we propose a novel Multi-granularity Anchor-Contrastive representation Learning (dubbed as MAC-Learning) to learn multi-granularity representations by conducting inter- and intra-granularity contrastive pretext tasks on the learnable and structural-link skeletons among three types of granularities covering local, context, and global views. To avoid the disturbance of ambiguous pairs from noisy and outlier samples, we design a more reliable Multi-granularity Anchor-Contrastive Loss (dubbed as MAC-Loss) that measures the agreement/disagreement between high-confidence soft-positive/negative pairs based on the anchor graph instead of the hard-positive/negative pairs in the conventional contrastive loss. Extensive experiments on both NTU RGB+D and Northwestern-UCLA datasets show that the proposed MAC-Learning outperforms existing competitive methods in semi-supervised skeleton-based action recognition tasks.</abstract><cop>United States</cop><pub>IEEE</pub><pmid>36395133</pmid><doi>10.1109/TPAMI.2022.3222871</doi><tpages>18</tpages><orcidid>https://orcid.org/0000-0003-4902-4663</orcidid><orcidid>https://orcid.org/0000-0001-9008-222X</orcidid><orcidid>https://orcid.org/0000-0002-1549-3317</orcidid><orcidid>https://orcid.org/0000-0003-3904-1400</orcidid></addata></record>
fulltext fulltext_linktorsrc
identifier ISSN: 0162-8828
ispartof IEEE transactions on pattern analysis and machine intelligence, 2023-06, Vol.45 (6), p.7559-7576
issn 0162-8828
1939-3539
2160-9292
language eng
recordid cdi_pubmed_primary_36395133
source IEEE Electronic Library (IEL)
subjects Action recognition
Activity recognition
anchor graph
contrastive learning
Data augmentation
Data models
Joints
Loss measurement
Machine learning
Outliers (statistics)
Pattern recognition
Representations
Semantics
semi-supervised
Skeleton
Task analysis
title Multi-Granularity Anchor-Contrastive Representation Learning for Semi-Supervised Skeleton-Based Action Recognition
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-23T12%3A16%3A43IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Multi-Granularity%20Anchor-Contrastive%20Representation%20Learning%20for%20Semi-Supervised%20Skeleton-Based%20Action%20Recognition&rft.jtitle=IEEE%20transactions%20on%20pattern%20analysis%20and%20machine%20intelligence&rft.au=Shu,%20Xiangbo&rft.date=2023-06-01&rft.volume=45&rft.issue=6&rft.spage=7559&rft.epage=7576&rft.pages=7559-7576&rft.issn=0162-8828&rft.eissn=1939-3539&rft.coden=ITPIDJ&rft_id=info:doi/10.1109/TPAMI.2022.3222871&rft_dat=%3Cproquest_RIE%3E2738191352%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2809888594&rft_id=info:pmid/36395133&rft_ieee_id=9954217&rfr_iscdi=true