Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework
During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three i...
Gespeichert in:
Veröffentlicht in: | IET control theory & applications 2024-12, Vol.18 (18), p.2742-2765 |
---|---|
Hauptverfasser: | , , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | 2765 |
---|---|
container_issue | 18 |
container_start_page | 2742 |
container_title | IET control theory & applications |
container_volume | 18 |
creator | Wu, Ao Jin, Yang Lv, Maolong Li, Huanyu Li, Leyan Yang, Rennong |
description | During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality.
A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs. |
doi_str_mv | 10.1049/cth2.12754 |
format | Article |
fullrecord | <record><control><sourceid>wiley_cross</sourceid><recordid>TN_cdi_crossref_primary_10_1049_cth2_12754</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>CTH212754</sourcerecordid><originalsourceid>FETCH-LOGICAL-c1984-36b6c9f7b53aae7587cb0d9894adc6998864ca3ba4d9cb4f47943e330d7acd893</originalsourceid><addsrcrecordid>eNp9kL1OwzAUhS0EEqWw8ASekQJ2bMcxW1UBRarEUuboxnZaQ-JEtkvVjUfgGXkS-oMYmc4dvnuk8yF0TcktJVzd6bTKb2kuBT9BIyoFzcpC5Kd_N-fn6CLGN0KEKLgYoThxQQdoEl6tO_Dfn18d6JXzFjufbACdXO8xxOhiAp-wsdEt_T2eYN9_2BZ36za5rjfQYgMJ8BB6bXe0X2LwBsMwtE7DoaQJ0NlNH94v0VkDbbRXvzlGr48Pi-ksm788PU8n80xTVfKMFXWhVSNrwQCsFKXUNTGqVByMLpQqd3M0sBq4UbrmDZeKM8sYMRK0KRUbo5tjrw59jME21RBcB2FbUVLtdVV7XdVB1w6mR3jjWrv9h6ymi1l-_PkBihpxDQ</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework</title><source>Wiley Online Library Open Access</source><source>DOAJ Directory of Open Access Journals</source><source>Wiley Online Library Journals Frontfile Complete</source><source>Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals</source><creator>Wu, Ao ; Jin, Yang ; Lv, Maolong ; Li, Huanyu ; Li, Leyan ; Yang, Rennong</creator><creatorcontrib>Wu, Ao ; Jin, Yang ; Lv, Maolong ; Li, Huanyu ; Li, Leyan ; Yang, Rennong</creatorcontrib><description>During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality.
A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs.</description><identifier>ISSN: 1751-8644</identifier><identifier>EISSN: 1751-8652</identifier><identifier>DOI: 10.1049/cth2.12754</identifier><language>eng</language><subject>data analysis ; human computer interaction ; military aircraft</subject><ispartof>IET control theory & applications, 2024-12, Vol.18 (18), p.2742-2765</ispartof><rights>2024 The Author(s). published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c1984-36b6c9f7b53aae7587cb0d9894adc6998864ca3ba4d9cb4f47943e330d7acd893</cites><orcidid>0000-0003-0654-8790</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://onlinelibrary.wiley.com/doi/pdf/10.1049%2Fcth2.12754$$EPDF$$P50$$Gwiley$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://onlinelibrary.wiley.com/doi/full/10.1049%2Fcth2.12754$$EHTML$$P50$$Gwiley$$Hfree_for_read</linktohtml><link.rule.ids>314,776,780,860,1411,11541,27901,27902,45550,45551,46027,46451</link.rule.ids></links><search><creatorcontrib>Wu, Ao</creatorcontrib><creatorcontrib>Jin, Yang</creatorcontrib><creatorcontrib>Lv, Maolong</creatorcontrib><creatorcontrib>Li, Huanyu</creatorcontrib><creatorcontrib>Li, Leyan</creatorcontrib><creatorcontrib>Yang, Rennong</creatorcontrib><title>Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework</title><title>IET control theory & applications</title><description>During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality.
A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs.</description><subject>data analysis</subject><subject>human computer interaction</subject><subject>military aircraft</subject><issn>1751-8644</issn><issn>1751-8652</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>24P</sourceid><recordid>eNp9kL1OwzAUhS0EEqWw8ASekQJ2bMcxW1UBRarEUuboxnZaQ-JEtkvVjUfgGXkS-oMYmc4dvnuk8yF0TcktJVzd6bTKb2kuBT9BIyoFzcpC5Kd_N-fn6CLGN0KEKLgYoThxQQdoEl6tO_Dfn18d6JXzFjufbACdXO8xxOhiAp-wsdEt_T2eYN9_2BZ36za5rjfQYgMJ8BB6bXe0X2LwBsMwtE7DoaQJ0NlNH94v0VkDbbRXvzlGr48Pi-ksm788PU8n80xTVfKMFXWhVSNrwQCsFKXUNTGqVByMLpQqd3M0sBq4UbrmDZeKM8sYMRK0KRUbo5tjrw59jME21RBcB2FbUVLtdVV7XdVB1w6mR3jjWrv9h6ymi1l-_PkBihpxDQ</recordid><startdate>202412</startdate><enddate>202412</enddate><creator>Wu, Ao</creator><creator>Jin, Yang</creator><creator>Lv, Maolong</creator><creator>Li, Huanyu</creator><creator>Li, Leyan</creator><creator>Yang, Rennong</creator><scope>24P</scope><scope>AAYXX</scope><scope>CITATION</scope><orcidid>https://orcid.org/0000-0003-0654-8790</orcidid></search><sort><creationdate>202412</creationdate><title>Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework</title><author>Wu, Ao ; Jin, Yang ; Lv, Maolong ; Li, Huanyu ; Li, Leyan ; Yang, Rennong</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c1984-36b6c9f7b53aae7587cb0d9894adc6998864ca3ba4d9cb4f47943e330d7acd893</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>data analysis</topic><topic>human computer interaction</topic><topic>military aircraft</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Wu, Ao</creatorcontrib><creatorcontrib>Jin, Yang</creatorcontrib><creatorcontrib>Lv, Maolong</creatorcontrib><creatorcontrib>Li, Huanyu</creatorcontrib><creatorcontrib>Li, Leyan</creatorcontrib><creatorcontrib>Yang, Rennong</creatorcontrib><collection>Wiley Online Library Open Access</collection><collection>CrossRef</collection><jtitle>IET control theory & applications</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Wu, Ao</au><au>Jin, Yang</au><au>Lv, Maolong</au><au>Li, Huanyu</au><au>Li, Leyan</au><au>Yang, Rennong</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework</atitle><jtitle>IET control theory & applications</jtitle><date>2024-12</date><risdate>2024</risdate><volume>18</volume><issue>18</issue><spage>2742</spage><epage>2765</epage><pages>2742-2765</pages><issn>1751-8644</issn><eissn>1751-8652</eissn><abstract>During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality.
A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs.</abstract><doi>10.1049/cth2.12754</doi><tpages>24</tpages><orcidid>https://orcid.org/0000-0003-0654-8790</orcidid><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | ISSN: 1751-8644 |
ispartof | IET control theory & applications, 2024-12, Vol.18 (18), p.2742-2765 |
issn | 1751-8644 1751-8652 |
language | eng |
recordid | cdi_crossref_primary_10_1049_cth2_12754 |
source | Wiley Online Library Open Access; DOAJ Directory of Open Access Journals; Wiley Online Library Journals Frontfile Complete; Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals |
subjects | data analysis human computer interaction military aircraft |
title | Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-20T04%3A47%3A42IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-wiley_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Aircraft%20human%E2%80%90machine%20interaction%20assistant%20design:%20A%20novel%20multimodal%20data%20processing%20and%20application%20framework&rft.jtitle=IET%20control%20theory%20&%20applications&rft.au=Wu,%20Ao&rft.date=2024-12&rft.volume=18&rft.issue=18&rft.spage=2742&rft.epage=2765&rft.pages=2742-2765&rft.issn=1751-8644&rft.eissn=1751-8652&rft_id=info:doi/10.1049/cth2.12754&rft_dat=%3Cwiley_cross%3ECTH212754%3C/wiley_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |