IDF-Sign: Addressing Inconsistent Depth Features for Dynamic Sign Word Recognition
Inconsistent hand and body features pose barriers to sign language recognition and translation leading to unsatisfactory models. Existing recognition models are built up on the spatial-temporal depth S_{p} features. Finding suitable expert features for the S_{p} model is challenging especially f...
Gespeichert in:
Veröffentlicht in: | IEEE access 2023, Vol.11, p.88511-88526 |
---|---|
Hauptverfasser: | , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | 88526 |
---|---|
container_issue | |
container_start_page | 88511 |
container_title | IEEE access |
container_volume | 11 |
creator | Abdullahi, Sunusi Bala Chamnongthai, Kosin |
description | Inconsistent hand and body features pose barriers to sign language recognition and translation leading to unsatisfactory models. Existing recognition models are built up on the spatial-temporal depth S_{p} features. Finding suitable expert features for the S_{p} model is challenging especially for dynamic sign words because many inconsistent features exist across hand motions and shapes. In this article, we propose IDF-Sign: an efficient and consistent S_{p} model from a spatial-temporal multivariate pairwise consistency feature ranking (PairCFR) approach. The temporal features are obtained by computing the 3D position vector of skeletal hand joint coordinates, while the spatial features were obtained by taking every ten spatial coordinates in the 3D video frames and averaging it and doing so until the end of the frames. The PairCFR was used to rank and select the best S_{p} model features at different feature thresholds. We employed a threshold selection to compute a mid-point value of each ranked feature according to its weight. The receiver operating characteristics (ROC) scheme was employed to identify the relationship between the sensitive parameters and the S_{p} features, and the obtained values were utilized as modeling inputs. To verify the IDF-Sign, we design a real-life experiment with a leap motion sensor (LMS) consisting of ten signers with a total of ninety dynamic sign words. LMS provides the depth videos, since depth videos are too dense for the S_{p} model to treat directly, we read the depth videos in comma-separated files in real time. Extensive IDF-Sign evaluations using machine learning on ASL, GSL, DSG, and ASL-similar datasets prove the Optimized Forest achieved an average recognition performance of 95%, 78%, 65.07%, and 95% of the top-1, respectively. |
doi_str_mv | 10.1109/ACCESS.2023.3305255 |
format | Article |
fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2857104560</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>10216946</ieee_id><doaj_id>oai_doaj_org_article_08c7157de981402ca472366bd74ea476</doaj_id><sourcerecordid>2857104560</sourcerecordid><originalsourceid>FETCH-LOGICAL-c409t-77abddeb01abb2c9e3ece2bcdc148b2df241e175412414900a3f4c2cf48a50a73</originalsourceid><addsrcrecordid>eNpNkU9LAzEQxRdRUKqfQA8Bz1vzd7PrrbRWC4JgFY8hm8zWFE1qsj3025u6RZpLHjPz3gz8iuKa4DEhuLmbTKcPy-WYYsrGjGFBhTgpLiipmpIJVp0e6fPiKqU1zq_OJSEvitfFbF4u3crfo4m1EVJyfoUW3gSfXOrB92gGm_4TzUH329xHXYhotvP62xm0N6KPEC16BRNW3vUu-MvirNNfCa4O_6h4nz-8TZ_K55fHxXTyXBqOm76UUrfWQouJbltqGmBggLbGGsLrltqOcgJECk6y4A3GmnXcUNPxWgusJRsViyHXBr1Wm-i-ddypoJ36K4S4Ujr2znyBwrWRREgLTU04pkZzSVlVtVZyyLrKWbdD1iaGny2kXq3DNvp8vqK1kARzUeE8xYYpE0NKEbr_rQSrPQs1sFB7FurAIrtuBpcDgCPHngqv2C9_ZoQ0</addsrcrecordid><sourcetype>Open Website</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2857104560</pqid></control><display><type>article</type><title>IDF-Sign: Addressing Inconsistent Depth Features for Dynamic Sign Word Recognition</title><source>IEEE Open Access Journals</source><source>DOAJ Directory of Open Access Journals</source><source>EZB-FREE-00999 freely available EZB journals</source><creator>Abdullahi, Sunusi Bala ; Chamnongthai, Kosin</creator><creatorcontrib>Abdullahi, Sunusi Bala ; Chamnongthai, Kosin</creatorcontrib><description><![CDATA[Inconsistent hand and body features pose barriers to sign language recognition and translation leading to unsatisfactory models. Existing recognition models are built up on the spatial-temporal depth <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> features. Finding suitable expert features for the <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model is challenging especially for dynamic sign words because many inconsistent features exist across hand motions and shapes. In this article, we propose IDF-Sign: an efficient and consistent <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model from a spatial-temporal multivariate pairwise consistency feature ranking (PairCFR) approach. The temporal features are obtained by computing the 3D position vector of skeletal hand joint coordinates, while the spatial features were obtained by taking every ten spatial coordinates in the 3D video frames and averaging it and doing so until the end of the frames. The PairCFR was used to rank and select the best <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model features at different feature thresholds. We employed a threshold selection to compute a mid-point value of each ranked feature according to its weight. The receiver operating characteristics (ROC) scheme was employed to identify the relationship between the sensitive parameters and the <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> features, and the obtained values were utilized as modeling inputs. To verify the IDF-Sign, we design a real-life experiment with a leap motion sensor (LMS) consisting of ten signers with a total of ninety dynamic sign words. LMS provides the depth videos, since depth videos are too dense for the <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model to treat directly, we read the depth videos in comma-separated files in real time. Extensive IDF-Sign evaluations using machine learning on ASL, GSL, DSG, and ASL-similar datasets prove the Optimized Forest achieved an average recognition performance of 95%, 78%, 65.07%, and 95% of the top-1, respectively.]]></description><identifier>ISSN: 2169-3536</identifier><identifier>EISSN: 2169-3536</identifier><identifier>DOI: 10.1109/ACCESS.2023.3305255</identifier><identifier>CODEN: IAECCG</identifier><language>eng</language><publisher>Piscataway: IEEE</publisher><subject>3D video processing ; Assistive technologies ; Automatic sign language recognition ; Computational modeling ; depth sensor ; Feature extraction ; feature selection ; Frames (data processing) ; Gesture recognition ; hand gesture ; Hidden Markov models ; Image processing ; Machine learning ; Motion sensors ; Parameter identification ; Parameter sensitivity ; Pattern recognition ; Sign language ; Solid modeling ; Streaming media ; Three-dimensional displays ; Video</subject><ispartof>IEEE access, 2023, Vol.11, p.88511-88526</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c409t-77abddeb01abb2c9e3ece2bcdc148b2df241e175412414900a3f4c2cf48a50a73</citedby><cites>FETCH-LOGICAL-c409t-77abddeb01abb2c9e3ece2bcdc148b2df241e175412414900a3f4c2cf48a50a73</cites><orcidid>0000-0003-1509-5754 ; 0000-0003-1898-7352</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/10216946$$EHTML$$P50$$Gieee$$Hfree_for_read</linktohtml><link.rule.ids>314,780,784,864,2102,4024,27633,27923,27924,27925,54933</link.rule.ids></links><search><creatorcontrib>Abdullahi, Sunusi Bala</creatorcontrib><creatorcontrib>Chamnongthai, Kosin</creatorcontrib><title>IDF-Sign: Addressing Inconsistent Depth Features for Dynamic Sign Word Recognition</title><title>IEEE access</title><addtitle>Access</addtitle><description><![CDATA[Inconsistent hand and body features pose barriers to sign language recognition and translation leading to unsatisfactory models. Existing recognition models are built up on the spatial-temporal depth <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> features. Finding suitable expert features for the <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model is challenging especially for dynamic sign words because many inconsistent features exist across hand motions and shapes. In this article, we propose IDF-Sign: an efficient and consistent <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model from a spatial-temporal multivariate pairwise consistency feature ranking (PairCFR) approach. The temporal features are obtained by computing the 3D position vector of skeletal hand joint coordinates, while the spatial features were obtained by taking every ten spatial coordinates in the 3D video frames and averaging it and doing so until the end of the frames. The PairCFR was used to rank and select the best <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model features at different feature thresholds. We employed a threshold selection to compute a mid-point value of each ranked feature according to its weight. The receiver operating characteristics (ROC) scheme was employed to identify the relationship between the sensitive parameters and the <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> features, and the obtained values were utilized as modeling inputs. To verify the IDF-Sign, we design a real-life experiment with a leap motion sensor (LMS) consisting of ten signers with a total of ninety dynamic sign words. LMS provides the depth videos, since depth videos are too dense for the <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model to treat directly, we read the depth videos in comma-separated files in real time. Extensive IDF-Sign evaluations using machine learning on ASL, GSL, DSG, and ASL-similar datasets prove the Optimized Forest achieved an average recognition performance of 95%, 78%, 65.07%, and 95% of the top-1, respectively.]]></description><subject>3D video processing</subject><subject>Assistive technologies</subject><subject>Automatic sign language recognition</subject><subject>Computational modeling</subject><subject>depth sensor</subject><subject>Feature extraction</subject><subject>feature selection</subject><subject>Frames (data processing)</subject><subject>Gesture recognition</subject><subject>hand gesture</subject><subject>Hidden Markov models</subject><subject>Image processing</subject><subject>Machine learning</subject><subject>Motion sensors</subject><subject>Parameter identification</subject><subject>Parameter sensitivity</subject><subject>Pattern recognition</subject><subject>Sign language</subject><subject>Solid modeling</subject><subject>Streaming media</subject><subject>Three-dimensional displays</subject><subject>Video</subject><issn>2169-3536</issn><issn>2169-3536</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>ESBDL</sourceid><sourceid>RIE</sourceid><sourceid>DOA</sourceid><recordid>eNpNkU9LAzEQxRdRUKqfQA8Bz1vzd7PrrbRWC4JgFY8hm8zWFE1qsj3025u6RZpLHjPz3gz8iuKa4DEhuLmbTKcPy-WYYsrGjGFBhTgpLiipmpIJVp0e6fPiKqU1zq_OJSEvitfFbF4u3crfo4m1EVJyfoUW3gSfXOrB92gGm_4TzUH329xHXYhotvP62xm0N6KPEC16BRNW3vUu-MvirNNfCa4O_6h4nz-8TZ_K55fHxXTyXBqOm76UUrfWQouJbltqGmBggLbGGsLrltqOcgJECk6y4A3GmnXcUNPxWgusJRsViyHXBr1Wm-i-ddypoJ36K4S4Ujr2znyBwrWRREgLTU04pkZzSVlVtVZyyLrKWbdD1iaGny2kXq3DNvp8vqK1kARzUeE8xYYpE0NKEbr_rQSrPQs1sFB7FurAIrtuBpcDgCPHngqv2C9_ZoQ0</recordid><startdate>2023</startdate><enddate>2023</enddate><creator>Abdullahi, Sunusi Bala</creator><creator>Chamnongthai, Kosin</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>ESBDL</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>7SP</scope><scope>7SR</scope><scope>8BQ</scope><scope>8FD</scope><scope>JG9</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>DOA</scope><orcidid>https://orcid.org/0000-0003-1509-5754</orcidid><orcidid>https://orcid.org/0000-0003-1898-7352</orcidid></search><sort><creationdate>2023</creationdate><title>IDF-Sign: Addressing Inconsistent Depth Features for Dynamic Sign Word Recognition</title><author>Abdullahi, Sunusi Bala ; Chamnongthai, Kosin</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c409t-77abddeb01abb2c9e3ece2bcdc148b2df241e175412414900a3f4c2cf48a50a73</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>3D video processing</topic><topic>Assistive technologies</topic><topic>Automatic sign language recognition</topic><topic>Computational modeling</topic><topic>depth sensor</topic><topic>Feature extraction</topic><topic>feature selection</topic><topic>Frames (data processing)</topic><topic>Gesture recognition</topic><topic>hand gesture</topic><topic>Hidden Markov models</topic><topic>Image processing</topic><topic>Machine learning</topic><topic>Motion sensors</topic><topic>Parameter identification</topic><topic>Parameter sensitivity</topic><topic>Pattern recognition</topic><topic>Sign language</topic><topic>Solid modeling</topic><topic>Streaming media</topic><topic>Three-dimensional displays</topic><topic>Video</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Abdullahi, Sunusi Bala</creatorcontrib><creatorcontrib>Chamnongthai, Kosin</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005–Present</collection><collection>IEEE Open Access Journals</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Electronics & Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>DOAJ Directory of Open Access Journals</collection><jtitle>IEEE access</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Abdullahi, Sunusi Bala</au><au>Chamnongthai, Kosin</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>IDF-Sign: Addressing Inconsistent Depth Features for Dynamic Sign Word Recognition</atitle><jtitle>IEEE access</jtitle><stitle>Access</stitle><date>2023</date><risdate>2023</risdate><volume>11</volume><spage>88511</spage><epage>88526</epage><pages>88511-88526</pages><issn>2169-3536</issn><eissn>2169-3536</eissn><coden>IAECCG</coden><abstract><![CDATA[Inconsistent hand and body features pose barriers to sign language recognition and translation leading to unsatisfactory models. Existing recognition models are built up on the spatial-temporal depth <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> features. Finding suitable expert features for the <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model is challenging especially for dynamic sign words because many inconsistent features exist across hand motions and shapes. In this article, we propose IDF-Sign: an efficient and consistent <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model from a spatial-temporal multivariate pairwise consistency feature ranking (PairCFR) approach. The temporal features are obtained by computing the 3D position vector of skeletal hand joint coordinates, while the spatial features were obtained by taking every ten spatial coordinates in the 3D video frames and averaging it and doing so until the end of the frames. The PairCFR was used to rank and select the best <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model features at different feature thresholds. We employed a threshold selection to compute a mid-point value of each ranked feature according to its weight. The receiver operating characteristics (ROC) scheme was employed to identify the relationship between the sensitive parameters and the <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> features, and the obtained values were utilized as modeling inputs. To verify the IDF-Sign, we design a real-life experiment with a leap motion sensor (LMS) consisting of ten signers with a total of ninety dynamic sign words. LMS provides the depth videos, since depth videos are too dense for the <inline-formula> <tex-math notation="LaTeX">S_{p} </tex-math></inline-formula> model to treat directly, we read the depth videos in comma-separated files in real time. Extensive IDF-Sign evaluations using machine learning on ASL, GSL, DSG, and ASL-similar datasets prove the Optimized Forest achieved an average recognition performance of 95%, 78%, 65.07%, and 95% of the top-1, respectively.]]></abstract><cop>Piscataway</cop><pub>IEEE</pub><doi>10.1109/ACCESS.2023.3305255</doi><tpages>16</tpages><orcidid>https://orcid.org/0000-0003-1509-5754</orcidid><orcidid>https://orcid.org/0000-0003-1898-7352</orcidid><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | ISSN: 2169-3536 |
ispartof | IEEE access, 2023, Vol.11, p.88511-88526 |
issn | 2169-3536 2169-3536 |
language | eng |
recordid | cdi_proquest_journals_2857104560 |
source | IEEE Open Access Journals; DOAJ Directory of Open Access Journals; EZB-FREE-00999 freely available EZB journals |
subjects | 3D video processing Assistive technologies Automatic sign language recognition Computational modeling depth sensor Feature extraction feature selection Frames (data processing) Gesture recognition hand gesture Hidden Markov models Image processing Machine learning Motion sensors Parameter identification Parameter sensitivity Pattern recognition Sign language Solid modeling Streaming media Three-dimensional displays Video |
title | IDF-Sign: Addressing Inconsistent Depth Features for Dynamic Sign Word Recognition |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T14%3A55%3A22IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=IDF-Sign:%20Addressing%20Inconsistent%20Depth%20Features%20for%20Dynamic%20Sign%20Word%20Recognition&rft.jtitle=IEEE%20access&rft.au=Abdullahi,%20Sunusi%20Bala&rft.date=2023&rft.volume=11&rft.spage=88511&rft.epage=88526&rft.pages=88511-88526&rft.issn=2169-3536&rft.eissn=2169-3536&rft.coden=IAECCG&rft_id=info:doi/10.1109/ACCESS.2023.3305255&rft_dat=%3Cproquest_cross%3E2857104560%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2857104560&rft_id=info:pmid/&rft_ieee_id=10216946&rft_doaj_id=oai_doaj_org_article_08c7157de981402ca472366bd74ea476&rfr_iscdi=true |