Visual Speech Enhancement

When video is shot in noisy environment, the voice of a speaker seen in the video can be enhanced using the visible mouth movements, reducing background noise. While most existing methods use audio-only inputs, improved performance is obtained with our visual speech enhancement, based on an audio-vi...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Veröffentlicht in:	arXiv.org 2018-06
Hauptverfasser:	Gabbay, Aviv, Shamir, Asaph, Peleg, Shmuel
Format:	Artikel
Sprache:	eng
Schlagworte:	Audio data Audio equipment Background noise Lipreading Neural networks Noise Noise reduction Speech processing Voice
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

container_end_page
container_issue
container_start_page
container_title	arXiv.org
container_volume
creator	Gabbay, Aviv Shamir, Asaph Peleg, Shmuel
description	When video is shot in noisy environment, the voice of a speaker seen in the video can be enhanced using the visible mouth movements, reducing background noise. While most existing methods use audio-only inputs, improved performance is obtained with our visual speech enhancement, based on an audio-visual neural network. We include in the training data videos to which we added the voice of the target speaker as background noise. Since the audio input is not sufficient to separate the voice of a speaker from his own voice, the trained model better exploits the visual input and generalizes well to different noise types. The proposed model outperforms prior audio visual methods on two public lipreading datasets. It is also the first to be demonstrated on a dataset not designed for lipreading, such as the weekly addresses of Barack Obama.
format	Article
fullrecord	<record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2073899850</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2073899850</sourcerecordid><originalsourceid>FETCH-proquest_journals_20738998503</originalsourceid><addsrcrecordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mSQDMssLk3MUQguSE1NzlBwzctIzEtOzU3NK-FhYE1LzClO5YXS3AzKbq4hzh66BUX5haWpxSXxWfmlRXlAqXgjA3NjC0tLC1MDY-JUAQCKhSp2</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2073899850</pqid></control><display><type>article</type><title>Visual Speech Enhancement</title><source>Freely Accessible Journals</source><creator>Gabbay, Aviv ; Shamir, Asaph ; Peleg, Shmuel</creator><creatorcontrib>Gabbay, Aviv ; Shamir, Asaph ; Peleg, Shmuel</creatorcontrib><description>When video is shot in noisy environment, the voice of a speaker seen in the video can be enhanced using the visible mouth movements, reducing background noise. While most existing methods use audio-only inputs, improved performance is obtained with our visual speech enhancement, based on an audio-visual neural network. We include in the training data videos to which we added the voice of the target speaker as background noise. Since the audio input is not sufficient to separate the voice of a speaker from his own voice, the trained model better exploits the visual input and generalizes well to different noise types. The proposed model outperforms prior audio visual methods on two public lipreading datasets. It is also the first to be demonstrated on a dataset not designed for lipreading, such as the weekly addresses of Barack Obama.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Audio data ; Audio equipment ; Background noise ; Lipreading ; Neural networks ; Noise ; Noise reduction ; Speech processing ; Voice</subject><ispartof>arXiv.org, 2018-06</ispartof><rights>2018. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>776,780</link.rule.ids></links><search><creatorcontrib>Gabbay, Aviv</creatorcontrib><creatorcontrib>Shamir, Asaph</creatorcontrib><creatorcontrib>Peleg, Shmuel</creatorcontrib><title>Visual Speech Enhancement</title><title>arXiv.org</title><description>When video is shot in noisy environment, the voice of a speaker seen in the video can be enhanced using the visible mouth movements, reducing background noise. While most existing methods use audio-only inputs, improved performance is obtained with our visual speech enhancement, based on an audio-visual neural network. We include in the training data videos to which we added the voice of the target speaker as background noise. Since the audio input is not sufficient to separate the voice of a speaker from his own voice, the trained model better exploits the visual input and generalizes well to different noise types. The proposed model outperforms prior audio visual methods on two public lipreading datasets. It is also the first to be demonstrated on a dataset not designed for lipreading, such as the weekly addresses of Barack Obama.</description><subject>Audio data</subject><subject>Audio equipment</subject><subject>Background noise</subject><subject>Lipreading</subject><subject>Neural networks</subject><subject>Noise</subject><subject>Noise reduction</subject><subject>Speech processing</subject><subject>Voice</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2018</creationdate><recordtype>article</recordtype><sourceid>BENPR</sourceid><recordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mSQDMssLk3MUQguSE1NzlBwzctIzEtOzU3NK-FhYE1LzClO5YXS3AzKbq4hzh66BUX5haWpxSXxWfmlRXlAqXgjA3NjC0tLC1MDY-JUAQCKhSp2</recordid><startdate>20180613</startdate><enddate>20180613</enddate><creator>Gabbay, Aviv</creator><creator>Shamir, Asaph</creator><creator>Peleg, Shmuel</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20180613</creationdate><title>Visual Speech Enhancement</title><author>Gabbay, Aviv ; Shamir, Asaph ; Peleg, Shmuel</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_20738998503</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2018</creationdate><topic>Audio data</topic><topic>Audio equipment</topic><topic>Background noise</topic><topic>Lipreading</topic><topic>Neural networks</topic><topic>Noise</topic><topic>Noise reduction</topic><topic>Speech processing</topic><topic>Voice</topic><toplevel>online_resources</toplevel><creatorcontrib>Gabbay, Aviv</creatorcontrib><creatorcontrib>Shamir, Asaph</creatorcontrib><creatorcontrib>Peleg, Shmuel</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Gabbay, Aviv</au><au>Shamir, Asaph</au><au>Peleg, Shmuel</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>Visual Speech Enhancement</atitle><jtitle>arXiv.org</jtitle><date>2018-06-13</date><risdate>2018</risdate><eissn>2331-8422</eissn><abstract>When video is shot in noisy environment, the voice of a speaker seen in the video can be enhanced using the visible mouth movements, reducing background noise. While most existing methods use audio-only inputs, improved performance is obtained with our visual speech enhancement, based on an audio-visual neural network. We include in the training data videos to which we added the voice of the target speaker as background noise. Since the audio input is not sufficient to separate the voice of a speaker from his own voice, the trained model better exploits the visual input and generalizes well to different noise types. The proposed model outperforms prior audio visual methods on two public lipreading datasets. It is also the first to be demonstrated on a dataset not designed for lipreading, such as the weekly addresses of Barack Obama.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext	fulltext
identifier	EISSN: 2331-8422
ispartof	arXiv.org, 2018-06
issn	2331-8422
language	eng
recordid	cdi_proquest_journals_2073899850
source	Freely Accessible Journals
subjects	Audio data Audio equipment Background noise Lipreading Neural networks Noise Noise reduction Speech processing Voice
title	Visual Speech Enhancement
url	https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-08T06%3A49%3A18IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=Visual%20Speech%20Enhancement&rft.jtitle=arXiv.org&rft.au=Gabbay,%20Aviv&rft.date=2018-06-13&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2073899850%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2073899850&rft_id=info:pmid/&rfr_iscdi=true