The Acoustically Emotion-Aware Conversational Agent With Speech Emotion Recognition and Empathetic Responses
Emotion is important for the conversational user interface. In prior research, conversational agents (CAs) employ natural language process techniques to create affective interaction based on text. However, the use of acoustic features of speech for voice-based CAs is under exploration. This work pre...
Gespeichert in:
| Veröffentlicht in: | IEEE transactions on affective computing 2023-01, Vol.14 (1), p.17-30 |
|---|---|
| Hauptverfasser: | , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 30 |
|---|---|
| container_issue | 1 |
| container_start_page | 17 |
| container_title | IEEE transactions on affective computing |
| container_volume | 14 |
| creator | Hu, Jiaxiong Huang, Yun Hu, Xiaozhu Xu, Yingqing |
| description | Emotion is important for the conversational user interface. In prior research, conversational agents (CAs) employ natural language process techniques to create affective interaction based on text. However, the use of acoustic features of speech for voice-based CAs is under exploration. This work presents an acoustically emotion-aware CA that enables speech emotion recognition and stylizes responses with empathetic feedback and interjections. We conducted an experiment with 75 participants to evaluate their perceived emotional intelligence (PEI) after interacting with the CA. Our results show that the acoustical emotion-awareness increased the participants' PEI of the CA, and the empathetic responses from the CA helped alleviate some participants' negative emotions. Our work provides implications for designing future CAs with better PEI. |
| doi_str_mv | 10.1109/TAFFC.2022.3205919 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_ieee_primary_9888003</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>9888003</ieee_id><sourcerecordid>2780983645</sourcerecordid><originalsourceid>FETCH-LOGICAL-c295t-c5653d3c4432dbf0e38d7d9d81677a31700b1b40dfbf98e5db3787418b1306193</originalsourceid><addsrcrecordid>eNpNkE9Lw0AQxRdRsNR-Ab0EPKfunyS7ewyhVaEgaMXjstlMmpQ0G3dTpd_epK3iXGaYee8x_BC6JXhOCJYP63S5zOYUUzpnFMeSyAs0ITKSIcNRfPlvvkYz77d4KMZYQvkENesKgtTYve9ro5vmECx2tq9tG6bf2kGQ2fYLnNfjSjdBuoG2Dz7qvgreOgBT_cqDVzB209bHWbfFsO90X8GQOpx8Z1sP_gZdlbrxMDv3KXpfLtbZU7h6eXzO0lVoqIz70MRJzApmoojRIi8xMFHwQhaCJJxrRjjGOckjXJR5KQXERc644BEROWE4IZJN0f0pt3P2cw--V1u7d8P_XlEusBQsieJBRU8q46z3DkrVuXqn3UERrEaw6ghWjWDVGexgujuZagD4M0ghxMj0Bxi6dJQ</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2780983645</pqid></control><display><type>article</type><title>The Acoustically Emotion-Aware Conversational Agent With Speech Emotion Recognition and Empathetic Responses</title><source>IEEE Electronic Library (IEL)</source><creator>Hu, Jiaxiong ; Huang, Yun ; Hu, Xiaozhu ; Xu, Yingqing</creator><creatorcontrib>Hu, Jiaxiong ; Huang, Yun ; Hu, Xiaozhu ; Xu, Yingqing</creatorcontrib><description>Emotion is important for the conversational user interface. In prior research, conversational agents (CAs) employ natural language process techniques to create affective interaction based on text. However, the use of acoustic features of speech for voice-based CAs is under exploration. This work presents an acoustically emotion-aware CA that enables speech emotion recognition and stylizes responses with empathetic feedback and interjections. We conducted an experiment with 75 participants to evaluate their perceived emotional intelligence (PEI) after interacting with the CA. Our results show that the acoustical emotion-awareness increased the participants' PEI of the CA, and the empathetic responses from the CA helped alleviate some participants' negative emotions. Our work provides implications for designing future CAs with better PEI.</description><identifier>ISSN: 1949-3045</identifier><identifier>EISSN: 1949-3045</identifier><identifier>DOI: 10.1109/TAFFC.2022.3205919</identifier><identifier>CODEN: ITACBQ</identifier><language>eng</language><publisher>Piscataway: IEEE</publisher><subject>Artificial intelligence ; Convolutional neural networks ; Electronic mail ; emotion in human-computer interaction ; Emotion recognition ; Emotions ; Games ; Human-centered computing ; influencing human emotional states ; intelligent agents ; Sentiment analysis ; Speech recognition</subject><ispartof>IEEE transactions on affective computing, 2023-01, Vol.14 (1), p.17-30</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c295t-c5653d3c4432dbf0e38d7d9d81677a31700b1b40dfbf98e5db3787418b1306193</citedby><cites>FETCH-LOGICAL-c295t-c5653d3c4432dbf0e38d7d9d81677a31700b1b40dfbf98e5db3787418b1306193</cites><orcidid>0000-0001-9988-6101</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/9888003$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,776,780,792,27901,27902,54733</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/9888003$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Hu, Jiaxiong</creatorcontrib><creatorcontrib>Huang, Yun</creatorcontrib><creatorcontrib>Hu, Xiaozhu</creatorcontrib><creatorcontrib>Xu, Yingqing</creatorcontrib><title>The Acoustically Emotion-Aware Conversational Agent With Speech Emotion Recognition and Empathetic Responses</title><title>IEEE transactions on affective computing</title><addtitle>TAFFC</addtitle><description>Emotion is important for the conversational user interface. In prior research, conversational agents (CAs) employ natural language process techniques to create affective interaction based on text. However, the use of acoustic features of speech for voice-based CAs is under exploration. This work presents an acoustically emotion-aware CA that enables speech emotion recognition and stylizes responses with empathetic feedback and interjections. We conducted an experiment with 75 participants to evaluate their perceived emotional intelligence (PEI) after interacting with the CA. Our results show that the acoustical emotion-awareness increased the participants' PEI of the CA, and the empathetic responses from the CA helped alleviate some participants' negative emotions. Our work provides implications for designing future CAs with better PEI.</description><subject>Artificial intelligence</subject><subject>Convolutional neural networks</subject><subject>Electronic mail</subject><subject>emotion in human-computer interaction</subject><subject>Emotion recognition</subject><subject>Emotions</subject><subject>Games</subject><subject>Human-centered computing</subject><subject>influencing human emotional states</subject><subject>intelligent agents</subject><subject>Sentiment analysis</subject><subject>Speech recognition</subject><issn>1949-3045</issn><issn>1949-3045</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNpNkE9Lw0AQxRdRsNR-Ab0EPKfunyS7ewyhVaEgaMXjstlMmpQ0G3dTpd_epK3iXGaYee8x_BC6JXhOCJYP63S5zOYUUzpnFMeSyAs0ITKSIcNRfPlvvkYz77d4KMZYQvkENesKgtTYve9ro5vmECx2tq9tG6bf2kGQ2fYLnNfjSjdBuoG2Dz7qvgreOgBT_cqDVzB209bHWbfFsO90X8GQOpx8Z1sP_gZdlbrxMDv3KXpfLtbZU7h6eXzO0lVoqIz70MRJzApmoojRIi8xMFHwQhaCJJxrRjjGOckjXJR5KQXERc644BEROWE4IZJN0f0pt3P2cw--V1u7d8P_XlEusBQsieJBRU8q46z3DkrVuXqn3UERrEaw6ghWjWDVGexgujuZagD4M0ghxMj0Bxi6dJQ</recordid><startdate>202301</startdate><enddate>202301</enddate><creator>Hu, Jiaxiong</creator><creator>Huang, Yun</creator><creator>Hu, Xiaozhu</creator><creator>Xu, Yingqing</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><orcidid>https://orcid.org/0000-0001-9988-6101</orcidid></search><sort><creationdate>202301</creationdate><title>The Acoustically Emotion-Aware Conversational Agent With Speech Emotion Recognition and Empathetic Responses</title><author>Hu, Jiaxiong ; Huang, Yun ; Hu, Xiaozhu ; Xu, Yingqing</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c295t-c5653d3c4432dbf0e38d7d9d81677a31700b1b40dfbf98e5db3787418b1306193</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Artificial intelligence</topic><topic>Convolutional neural networks</topic><topic>Electronic mail</topic><topic>emotion in human-computer interaction</topic><topic>Emotion recognition</topic><topic>Emotions</topic><topic>Games</topic><topic>Human-centered computing</topic><topic>influencing human emotional states</topic><topic>intelligent agents</topic><topic>Sentiment analysis</topic><topic>Speech recognition</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Hu, Jiaxiong</creatorcontrib><creatorcontrib>Huang, Yun</creatorcontrib><creatorcontrib>Hu, Xiaozhu</creatorcontrib><creatorcontrib>Xu, Yingqing</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><jtitle>IEEE transactions on affective computing</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Hu, Jiaxiong</au><au>Huang, Yun</au><au>Hu, Xiaozhu</au><au>Xu, Yingqing</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>The Acoustically Emotion-Aware Conversational Agent With Speech Emotion Recognition and Empathetic Responses</atitle><jtitle>IEEE transactions on affective computing</jtitle><stitle>TAFFC</stitle><date>2023-01</date><risdate>2023</risdate><volume>14</volume><issue>1</issue><spage>17</spage><epage>30</epage><pages>17-30</pages><issn>1949-3045</issn><eissn>1949-3045</eissn><coden>ITACBQ</coden><abstract>Emotion is important for the conversational user interface. In prior research, conversational agents (CAs) employ natural language process techniques to create affective interaction based on text. However, the use of acoustic features of speech for voice-based CAs is under exploration. This work presents an acoustically emotion-aware CA that enables speech emotion recognition and stylizes responses with empathetic feedback and interjections. We conducted an experiment with 75 participants to evaluate their perceived emotional intelligence (PEI) after interacting with the CA. Our results show that the acoustical emotion-awareness increased the participants' PEI of the CA, and the empathetic responses from the CA helped alleviate some participants' negative emotions. Our work provides implications for designing future CAs with better PEI.</abstract><cop>Piscataway</cop><pub>IEEE</pub><doi>10.1109/TAFFC.2022.3205919</doi><tpages>14</tpages><orcidid>https://orcid.org/0000-0001-9988-6101</orcidid></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | ISSN: 1949-3045 |
| ispartof | IEEE transactions on affective computing, 2023-01, Vol.14 (1), p.17-30 |
| issn | 1949-3045 1949-3045 |
| language | eng |
| recordid | cdi_ieee_primary_9888003 |
| source | IEEE Electronic Library (IEL) |
| subjects | Artificial intelligence Convolutional neural networks Electronic mail emotion in human-computer interaction Emotion recognition Emotions Games Human-centered computing influencing human emotional states intelligent agents Sentiment analysis Speech recognition |
| title | The Acoustically Emotion-Aware Conversational Agent With Speech Emotion Recognition and Empathetic Responses |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-01T00%3A02%3A22IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=The%20Acoustically%20Emotion-Aware%20Conversational%20Agent%20With%20Speech%20Emotion%20Recognition%20and%20Empathetic%20Responses&rft.jtitle=IEEE%20transactions%20on%20affective%20computing&rft.au=Hu,%20Jiaxiong&rft.date=2023-01&rft.volume=14&rft.issue=1&rft.spage=17&rft.epage=30&rft.pages=17-30&rft.issn=1949-3045&rft.eissn=1949-3045&rft.coden=ITACBQ&rft_id=info:doi/10.1109/TAFFC.2022.3205919&rft_dat=%3Cproquest_RIE%3E2780983645%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2780983645&rft_id=info:pmid/&rft_ieee_id=9888003&rfr_iscdi=true |