Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework

During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three i...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IET control theory & applications 2024-12, Vol.18 (18), p.2742-2765
Hauptverfasser: Wu, Ao, Jin, Yang, Lv, Maolong, Li, Huanyu, Li, Leyan, Yang, Rennong
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 2765
container_issue 18
container_start_page 2742
container_title IET control theory & applications
container_volume 18
creator Wu, Ao
Jin, Yang
Lv, Maolong
Li, Huanyu
Li, Leyan
Yang, Rennong
description During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality. A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs.
doi_str_mv 10.1049/cth2.12754
format Article
fullrecord <record><control><sourceid>wiley_cross</sourceid><recordid>TN_cdi_crossref_primary_10_1049_cth2_12754</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>CTH212754</sourcerecordid><originalsourceid>FETCH-LOGICAL-c1984-36b6c9f7b53aae7587cb0d9894adc6998864ca3ba4d9cb4f47943e330d7acd893</originalsourceid><addsrcrecordid>eNp9kL1OwzAUhS0EEqWw8ASekQJ2bMcxW1UBRarEUuboxnZaQ-JEtkvVjUfgGXkS-oMYmc4dvnuk8yF0TcktJVzd6bTKb2kuBT9BIyoFzcpC5Kd_N-fn6CLGN0KEKLgYoThxQQdoEl6tO_Dfn18d6JXzFjufbACdXO8xxOhiAp-wsdEt_T2eYN9_2BZ36za5rjfQYgMJ8BB6bXe0X2LwBsMwtE7DoaQJ0NlNH94v0VkDbbRXvzlGr48Pi-ksm788PU8n80xTVfKMFXWhVSNrwQCsFKXUNTGqVByMLpQqd3M0sBq4UbrmDZeKM8sYMRK0KRUbo5tjrw59jME21RBcB2FbUVLtdVV7XdVB1w6mR3jjWrv9h6ymi1l-_PkBihpxDQ</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework</title><source>Wiley Online Library Open Access</source><source>DOAJ Directory of Open Access Journals</source><source>Wiley Online Library Journals Frontfile Complete</source><source>Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals</source><creator>Wu, Ao ; Jin, Yang ; Lv, Maolong ; Li, Huanyu ; Li, Leyan ; Yang, Rennong</creator><creatorcontrib>Wu, Ao ; Jin, Yang ; Lv, Maolong ; Li, Huanyu ; Li, Leyan ; Yang, Rennong</creatorcontrib><description>During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality. A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs.</description><identifier>ISSN: 1751-8644</identifier><identifier>EISSN: 1751-8652</identifier><identifier>DOI: 10.1049/cth2.12754</identifier><language>eng</language><subject>data analysis ; human computer interaction ; military aircraft</subject><ispartof>IET control theory &amp; applications, 2024-12, Vol.18 (18), p.2742-2765</ispartof><rights>2024 The Author(s). published by John Wiley &amp; Sons Ltd on behalf of The Institution of Engineering and Technology.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c1984-36b6c9f7b53aae7587cb0d9894adc6998864ca3ba4d9cb4f47943e330d7acd893</cites><orcidid>0000-0003-0654-8790</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://onlinelibrary.wiley.com/doi/pdf/10.1049%2Fcth2.12754$$EPDF$$P50$$Gwiley$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://onlinelibrary.wiley.com/doi/full/10.1049%2Fcth2.12754$$EHTML$$P50$$Gwiley$$Hfree_for_read</linktohtml><link.rule.ids>314,776,780,860,1411,11541,27901,27902,45550,45551,46027,46451</link.rule.ids></links><search><creatorcontrib>Wu, Ao</creatorcontrib><creatorcontrib>Jin, Yang</creatorcontrib><creatorcontrib>Lv, Maolong</creatorcontrib><creatorcontrib>Li, Huanyu</creatorcontrib><creatorcontrib>Li, Leyan</creatorcontrib><creatorcontrib>Yang, Rennong</creatorcontrib><title>Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework</title><title>IET control theory &amp; applications</title><description>During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality. A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs.</description><subject>data analysis</subject><subject>human computer interaction</subject><subject>military aircraft</subject><issn>1751-8644</issn><issn>1751-8652</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>24P</sourceid><recordid>eNp9kL1OwzAUhS0EEqWw8ASekQJ2bMcxW1UBRarEUuboxnZaQ-JEtkvVjUfgGXkS-oMYmc4dvnuk8yF0TcktJVzd6bTKb2kuBT9BIyoFzcpC5Kd_N-fn6CLGN0KEKLgYoThxQQdoEl6tO_Dfn18d6JXzFjufbACdXO8xxOhiAp-wsdEt_T2eYN9_2BZ36za5rjfQYgMJ8BB6bXe0X2LwBsMwtE7DoaQJ0NlNH94v0VkDbbRXvzlGr48Pi-ksm788PU8n80xTVfKMFXWhVSNrwQCsFKXUNTGqVByMLpQqd3M0sBq4UbrmDZeKM8sYMRK0KRUbo5tjrw59jME21RBcB2FbUVLtdVV7XdVB1w6mR3jjWrv9h6ymi1l-_PkBihpxDQ</recordid><startdate>202412</startdate><enddate>202412</enddate><creator>Wu, Ao</creator><creator>Jin, Yang</creator><creator>Lv, Maolong</creator><creator>Li, Huanyu</creator><creator>Li, Leyan</creator><creator>Yang, Rennong</creator><scope>24P</scope><scope>AAYXX</scope><scope>CITATION</scope><orcidid>https://orcid.org/0000-0003-0654-8790</orcidid></search><sort><creationdate>202412</creationdate><title>Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework</title><author>Wu, Ao ; Jin, Yang ; Lv, Maolong ; Li, Huanyu ; Li, Leyan ; Yang, Rennong</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c1984-36b6c9f7b53aae7587cb0d9894adc6998864ca3ba4d9cb4f47943e330d7acd893</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>data analysis</topic><topic>human computer interaction</topic><topic>military aircraft</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Wu, Ao</creatorcontrib><creatorcontrib>Jin, Yang</creatorcontrib><creatorcontrib>Lv, Maolong</creatorcontrib><creatorcontrib>Li, Huanyu</creatorcontrib><creatorcontrib>Li, Leyan</creatorcontrib><creatorcontrib>Yang, Rennong</creatorcontrib><collection>Wiley Online Library Open Access</collection><collection>CrossRef</collection><jtitle>IET control theory &amp; applications</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Wu, Ao</au><au>Jin, Yang</au><au>Lv, Maolong</au><au>Li, Huanyu</au><au>Li, Leyan</au><au>Yang, Rennong</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework</atitle><jtitle>IET control theory &amp; applications</jtitle><date>2024-12</date><risdate>2024</risdate><volume>18</volume><issue>18</issue><spage>2742</spage><epage>2765</epage><pages>2742-2765</pages><issn>1751-8644</issn><eissn>1751-8652</eissn><abstract>During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality. A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs.</abstract><doi>10.1049/cth2.12754</doi><tpages>24</tpages><orcidid>https://orcid.org/0000-0003-0654-8790</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 1751-8644
ispartof IET control theory & applications, 2024-12, Vol.18 (18), p.2742-2765
issn 1751-8644
1751-8652
language eng
recordid cdi_crossref_primary_10_1049_cth2_12754
source Wiley Online Library Open Access; DOAJ Directory of Open Access Journals; Wiley Online Library Journals Frontfile Complete; Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals
subjects data analysis
human computer interaction
military aircraft
title Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-20T04%3A47%3A42IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-wiley_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Aircraft%20human%E2%80%90machine%20interaction%20assistant%20design:%20A%20novel%20multimodal%20data%20processing%20and%20application%20framework&rft.jtitle=IET%20control%20theory%20&%20applications&rft.au=Wu,%20Ao&rft.date=2024-12&rft.volume=18&rft.issue=18&rft.spage=2742&rft.epage=2765&rft.pages=2742-2765&rft.issn=1751-8644&rft.eissn=1751-8652&rft_id=info:doi/10.1049/cth2.12754&rft_dat=%3Cwiley_cross%3ECTH212754%3C/wiley_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true