Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization
Emotion is a key element in user-generated video. However, it is difficult to understand emotions conveyed in such videos due to the complex and unstructured nature of user-generated content and the sparsity of video frames expressing emotion. In this paper, for the first time, we propose a techniqu...
Gespeichert in:
| Veröffentlicht in: | IEEE transactions on affective computing 2018-04, Vol.9 (2), p.255-270 |
|---|---|
| Hauptverfasser: | , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 270 |
|---|---|
| container_issue | 2 |
| container_start_page | 255 |
| container_title | IEEE transactions on affective computing |
| container_volume | 9 |
| creator | Baohan Xu Yanwei Fu Yu-Gang Jiang Boyang Li Sigal, Leonid |
| description | Emotion is a key element in user-generated video. However, it is difficult to understand emotions conveyed in such videos due to the complex and unstructured nature of user-generated content and the sparsity of video frames expressing emotion. In this paper, for the first time, we propose a technique for transferring knowledge from heterogeneous external sources, including image and textual data, to facilitate three related tasks in understanding video emotion: emotion recognition, emotion attribution and emotion-oriented summarization. Specifically, our framework (1) learns a video encoding from an auxiliary emotional image dataset in order to improve supervised video emotion recognition, and (2) transfers knowledge from an auxiliary textual corpora for zero-shot recognition of emotion classes unseen during training. The proposed technique for knowledge transfer facilitates novel applications of emotion attribution and emotion-oriented summarization. A comprehensive set of experiments on multiple datasets demonstrate the effectiveness of our framework. |
| doi_str_mv | 10.1109/TAFFC.2016.2622690 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_ieee_primary_7723914</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>7723914</ieee_id><sourcerecordid>2174523664</sourcerecordid><originalsourceid>FETCH-LOGICAL-c295t-8bcd3ffeda2b7469eea56053192a7879ad4b9fa5d793bae688eeaf08ebaa05193</originalsourceid><addsrcrecordid>eNpNkN9LwzAQx4MoOOb-AX0J-GpnfrRJ8zjG5sSBoNMnIaTtdWRsyUxaRP96222I93LH3fd7d3wQuqZkTClR96vJfD4dM0LFmAnGhCJnaEBVqhJO0uz8X32JRjFuSBecc8HkAH0soIHg1-DAtxE_Of-1hWoNeBWMizUEbB1-txV4PNv5xnqHX6D0a2f7-g5PmibYoj0MjKvwa7vbmWB_TN-5Qhe12UYYnfIQvc1nq-kiWT4_PE4ny6RkKmuSvCgrXtdQGVbIVCgAkwmScaqYkblUpkoLVZuskooXBkSed4qa5FAYQzKq-BDdHvfug_9sITZ649vgupOaUZlmjAuRdip2VJXBxxig1vtgu2e_NSW6B6kPIHUPUp9Adqabo8kCwJ9BSsYVTfkvNdJwxg</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2174523664</pqid></control><display><type>article</type><title>Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization</title><source>IEEE Electronic Library (IEL)</source><creator>Baohan Xu ; Yanwei Fu ; Yu-Gang Jiang ; Boyang Li ; Sigal, Leonid</creator><creatorcontrib>Baohan Xu ; Yanwei Fu ; Yu-Gang Jiang ; Boyang Li ; Sigal, Leonid</creatorcontrib><description>Emotion is a key element in user-generated video. However, it is difficult to understand emotions conveyed in such videos due to the complex and unstructured nature of user-generated content and the sparsity of video frames expressing emotion. In this paper, for the first time, we propose a technique for transferring knowledge from heterogeneous external sources, including image and textual data, to facilitate three related tasks in understanding video emotion: emotion recognition, emotion attribution and emotion-oriented summarization. Specifically, our framework (1) learns a video encoding from an auxiliary emotional image dataset in order to improve supervised video emotion recognition, and (2) transfers knowledge from an auxiliary textual corpora for zero-shot recognition of emotion classes unseen during training. The proposed technique for knowledge transfer facilitates novel applications of emotion attribution and emotion-oriented summarization. A comprehensive set of experiments on multiple datasets demonstrate the effectiveness of our framework.</description><identifier>ISSN: 1949-3045</identifier><identifier>EISSN: 1949-3045</identifier><identifier>DOI: 10.1109/TAFFC.2016.2622690</identifier><identifier>CODEN: ITACBQ</identifier><language>eng</language><publisher>Piscataway: IEEE</publisher><subject>Emotion recognition ; Feature extraction ; Image recognition ; Knowledge management ; Knowledge transfer ; Object recognition ; Semantics ; summarization ; Training ; transfer learning ; User generated content ; Video data ; Video emotion recognition ; Visualization ; zero-shot learning</subject><ispartof>IEEE transactions on affective computing, 2018-04, Vol.9 (2), p.255-270</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2018</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c295t-8bcd3ffeda2b7469eea56053192a7879ad4b9fa5d793bae688eeaf08ebaa05193</citedby><cites>FETCH-LOGICAL-c295t-8bcd3ffeda2b7469eea56053192a7879ad4b9fa5d793bae688eeaf08ebaa05193</cites><orcidid>0000-0002-1907-8567 ; 0000-0003-4046-1189 ; 0000-0002-6595-6893</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/7723914$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,777,781,793,27905,27906,54739</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/7723914$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Baohan Xu</creatorcontrib><creatorcontrib>Yanwei Fu</creatorcontrib><creatorcontrib>Yu-Gang Jiang</creatorcontrib><creatorcontrib>Boyang Li</creatorcontrib><creatorcontrib>Sigal, Leonid</creatorcontrib><title>Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization</title><title>IEEE transactions on affective computing</title><addtitle>T-AFFC</addtitle><description>Emotion is a key element in user-generated video. However, it is difficult to understand emotions conveyed in such videos due to the complex and unstructured nature of user-generated content and the sparsity of video frames expressing emotion. In this paper, for the first time, we propose a technique for transferring knowledge from heterogeneous external sources, including image and textual data, to facilitate three related tasks in understanding video emotion: emotion recognition, emotion attribution and emotion-oriented summarization. Specifically, our framework (1) learns a video encoding from an auxiliary emotional image dataset in order to improve supervised video emotion recognition, and (2) transfers knowledge from an auxiliary textual corpora for zero-shot recognition of emotion classes unseen during training. The proposed technique for knowledge transfer facilitates novel applications of emotion attribution and emotion-oriented summarization. A comprehensive set of experiments on multiple datasets demonstrate the effectiveness of our framework.</description><subject>Emotion recognition</subject><subject>Feature extraction</subject><subject>Image recognition</subject><subject>Knowledge management</subject><subject>Knowledge transfer</subject><subject>Object recognition</subject><subject>Semantics</subject><subject>summarization</subject><subject>Training</subject><subject>transfer learning</subject><subject>User generated content</subject><subject>Video data</subject><subject>Video emotion recognition</subject><subject>Visualization</subject><subject>zero-shot learning</subject><issn>1949-3045</issn><issn>1949-3045</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2018</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNpNkN9LwzAQx4MoOOb-AX0J-GpnfrRJ8zjG5sSBoNMnIaTtdWRsyUxaRP96222I93LH3fd7d3wQuqZkTClR96vJfD4dM0LFmAnGhCJnaEBVqhJO0uz8X32JRjFuSBecc8HkAH0soIHg1-DAtxE_Of-1hWoNeBWMizUEbB1-txV4PNv5xnqHX6D0a2f7-g5PmibYoj0MjKvwa7vbmWB_TN-5Qhe12UYYnfIQvc1nq-kiWT4_PE4ny6RkKmuSvCgrXtdQGVbIVCgAkwmScaqYkblUpkoLVZuskooXBkSed4qa5FAYQzKq-BDdHvfug_9sITZ649vgupOaUZlmjAuRdip2VJXBxxig1vtgu2e_NSW6B6kPIHUPUp9Adqabo8kCwJ9BSsYVTfkvNdJwxg</recordid><startdate>20180401</startdate><enddate>20180401</enddate><creator>Baohan Xu</creator><creator>Yanwei Fu</creator><creator>Yu-Gang Jiang</creator><creator>Boyang Li</creator><creator>Sigal, Leonid</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><orcidid>https://orcid.org/0000-0002-1907-8567</orcidid><orcidid>https://orcid.org/0000-0003-4046-1189</orcidid><orcidid>https://orcid.org/0000-0002-6595-6893</orcidid></search><sort><creationdate>20180401</creationdate><title>Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization</title><author>Baohan Xu ; Yanwei Fu ; Yu-Gang Jiang ; Boyang Li ; Sigal, Leonid</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c295t-8bcd3ffeda2b7469eea56053192a7879ad4b9fa5d793bae688eeaf08ebaa05193</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2018</creationdate><topic>Emotion recognition</topic><topic>Feature extraction</topic><topic>Image recognition</topic><topic>Knowledge management</topic><topic>Knowledge transfer</topic><topic>Object recognition</topic><topic>Semantics</topic><topic>summarization</topic><topic>Training</topic><topic>transfer learning</topic><topic>User generated content</topic><topic>Video data</topic><topic>Video emotion recognition</topic><topic>Visualization</topic><topic>zero-shot learning</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Baohan Xu</creatorcontrib><creatorcontrib>Yanwei Fu</creatorcontrib><creatorcontrib>Yu-Gang Jiang</creatorcontrib><creatorcontrib>Boyang Li</creatorcontrib><creatorcontrib>Sigal, Leonid</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><jtitle>IEEE transactions on affective computing</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Baohan Xu</au><au>Yanwei Fu</au><au>Yu-Gang Jiang</au><au>Boyang Li</au><au>Sigal, Leonid</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization</atitle><jtitle>IEEE transactions on affective computing</jtitle><stitle>T-AFFC</stitle><date>2018-04-01</date><risdate>2018</risdate><volume>9</volume><issue>2</issue><spage>255</spage><epage>270</epage><pages>255-270</pages><issn>1949-3045</issn><eissn>1949-3045</eissn><coden>ITACBQ</coden><abstract>Emotion is a key element in user-generated video. However, it is difficult to understand emotions conveyed in such videos due to the complex and unstructured nature of user-generated content and the sparsity of video frames expressing emotion. In this paper, for the first time, we propose a technique for transferring knowledge from heterogeneous external sources, including image and textual data, to facilitate three related tasks in understanding video emotion: emotion recognition, emotion attribution and emotion-oriented summarization. Specifically, our framework (1) learns a video encoding from an auxiliary emotional image dataset in order to improve supervised video emotion recognition, and (2) transfers knowledge from an auxiliary textual corpora for zero-shot recognition of emotion classes unseen during training. The proposed technique for knowledge transfer facilitates novel applications of emotion attribution and emotion-oriented summarization. A comprehensive set of experiments on multiple datasets demonstrate the effectiveness of our framework.</abstract><cop>Piscataway</cop><pub>IEEE</pub><doi>10.1109/TAFFC.2016.2622690</doi><tpages>16</tpages><orcidid>https://orcid.org/0000-0002-1907-8567</orcidid><orcidid>https://orcid.org/0000-0003-4046-1189</orcidid><orcidid>https://orcid.org/0000-0002-6595-6893</orcidid></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | ISSN: 1949-3045 |
| ispartof | IEEE transactions on affective computing, 2018-04, Vol.9 (2), p.255-270 |
| issn | 1949-3045 1949-3045 |
| language | eng |
| recordid | cdi_ieee_primary_7723914 |
| source | IEEE Electronic Library (IEL) |
| subjects | Emotion recognition Feature extraction Image recognition Knowledge management Knowledge transfer Object recognition Semantics summarization Training transfer learning User generated content Video data Video emotion recognition Visualization zero-shot learning |
| title | Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-18T08%3A42%3A28IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Heterogeneous%20Knowledge%20Transfer%20in%20Video%20Emotion%20Recognition,%20Attribution%20and%20Summarization&rft.jtitle=IEEE%20transactions%20on%20affective%20computing&rft.au=Baohan%20Xu&rft.date=2018-04-01&rft.volume=9&rft.issue=2&rft.spage=255&rft.epage=270&rft.pages=255-270&rft.issn=1949-3045&rft.eissn=1949-3045&rft.coden=ITACBQ&rft_id=info:doi/10.1109/TAFFC.2016.2622690&rft_dat=%3Cproquest_RIE%3E2174523664%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2174523664&rft_id=info:pmid/&rft_ieee_id=7723914&rfr_iscdi=true |