Semantic Face Hallucination: Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes

Given a tiny face image, existing face hallucination methods aim at super-resolving its high-resolution (HR) counterpart by learning a mapping from an exemplary dataset. Since a low-resolution (LR) input patch may correspond to many HR candidate patches, this ambiguity may lead to distorted HR facia...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on pattern analysis and machine intelligence 2020-11, Vol.42 (11), p.2926-2943
Hauptverfasser: Yu, Xin, Fernando, Basura, Hartley, Richard, Porikli, Fatih
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 2943
container_issue 11
container_start_page 2926
container_title IEEE transactions on pattern analysis and machine intelligence
container_volume 42
creator Yu, Xin
Fernando, Basura
Hartley, Richard
Porikli, Fatih
description Given a tiny face image, existing face hallucination methods aim at super-resolving its high-resolution (HR) counterpart by learning a mapping from an exemplary dataset. Since a low-resolution (LR) input patch may correspond to many HR candidate patches, this ambiguity may lead to distorted HR facial details and wrong attributes such as gender reversal and rejuvenation. An LR input contains low-frequency facial components of its HR version while its residual face image, defined as the difference between the HR ground-truth and interpolated LR images, contains the missing high-frequency facial details. We demonstrate that supplementing residual images or feature maps with additional facial attribute information can significantly reduce the ambiguity in face super-resolution. To explore this idea, we develop an attribute-embedded upsampling network, which consists of an upsampling network and a discriminative network. The upsampling network is composed of an autoencoder with skip-connections, which incorporates facial attribute vectors into the residual features of LR inputs at the bottleneck of the autoencoder, and deconvolutional layers used for upsampling. The discriminative network is designed to examine whether super-resolved faces contain the desired attributes or not and then its loss is used for updating the upsampling network. In this manner, we can super-resolve tiny (16×16 pixels) unaligned face images with a large upscaling factor of 8× while reducing the uncertainty of one-to-many mappings remarkably. By conducting extensive evaluations on a large-scale dataset, we demonstrate that our method achieves superior face hallucination results and outperforms the state-of-the-art.
doi_str_mv 10.1109/TPAMI.2019.2916881
format Article
fullrecord <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_crossref_primary_10_1109_TPAMI_2019_2916881</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>8713923</ieee_id><sourcerecordid>2232059903</sourcerecordid><originalsourceid>FETCH-LOGICAL-c351t-dff16994af1e89c3c858e311f043207bbd026e9a3de6c065d21651e71aa7180e3</originalsourceid><addsrcrecordid>eNpdkctKAzEUhoMoWi8voCADbtxMzUnmkrgr4qVQUbxtQ5o5oylzqZOM4tubOtWFq8DJ9_2c5CfkEOgYgMqzp_vJ7XTMKMgxk5AJARtkBJLLmKdcbpIRhYzFQjCxQ3adW1AKSUr5NtnhQU-TPB-RxSPWuvHWRFfaYHSjq6o3ttHets159NgvsYsf0LXVh21eoxfsvqJZ-zmM-hU0eNNav6KLPq1_W0nLCmtsvA70xPvOznuPbp9slbpyeLA-98jz1eXTxU08u7ueXkxmseEp-LgoS8ikTHQJKKThRqQCOUBJE85oPp8XlGUoNS8wMzRLCwZZCpiD1jkIinyPnA65y65979F5VVtnsKp0g23vFGMhJ5WS8oCe_EMXbd81YTvFkkQkSS4oBIoNlOla5zos1bKzdXicAqpWTaifJtSqCbVuIkjH6-h-XmPxp_x-fQCOBsAi4t-1yIFLxvk3PYqNFQ</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2448447801</pqid></control><display><type>article</type><title>Semantic Face Hallucination: Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes</title><source>IEEE Electronic Library (IEL)</source><creator>Yu, Xin ; Fernando, Basura ; Hartley, Richard ; Porikli, Fatih</creator><creatorcontrib>Yu, Xin ; Fernando, Basura ; Hartley, Richard ; Porikli, Fatih</creatorcontrib><description>Given a tiny face image, existing face hallucination methods aim at super-resolving its high-resolution (HR) counterpart by learning a mapping from an exemplary dataset. Since a low-resolution (LR) input patch may correspond to many HR candidate patches, this ambiguity may lead to distorted HR facial details and wrong attributes such as gender reversal and rejuvenation. An LR input contains low-frequency facial components of its HR version while its residual face image, defined as the difference between the HR ground-truth and interpolated LR images, contains the missing high-frequency facial details. We demonstrate that supplementing residual images or feature maps with additional facial attribute information can significantly reduce the ambiguity in face super-resolution. To explore this idea, we develop an attribute-embedded upsampling network, which consists of an upsampling network and a discriminative network. The upsampling network is composed of an autoencoder with skip-connections, which incorporates facial attribute vectors into the residual features of LR inputs at the bottleneck of the autoencoder, and deconvolutional layers used for upsampling. The discriminative network is designed to examine whether super-resolved faces contain the desired attributes or not and then its loss is used for updating the upsampling network. In this manner, we can super-resolve tiny (16×16 pixels) unaligned face images with a large upscaling factor of 8× while reducing the uncertainty of one-to-many mappings remarkably. By conducting extensive evaluations on a large-scale dataset, we demonstrate that our method achieves superior face hallucination results and outperforms the state-of-the-art.</description><identifier>ISSN: 0162-8828</identifier><identifier>EISSN: 1939-3539</identifier><identifier>EISSN: 2160-9292</identifier><identifier>DOI: 10.1109/TPAMI.2019.2916881</identifier><identifier>PMID: 31095477</identifier><identifier>CODEN: ITPIDJ</identifier><language>eng</language><publisher>United States: IEEE</publisher><subject>Ambiguity ; attribute ; Datasets ; Face ; Facial features ; Feature extraction ; Feature maps ; hallucination ; Hallucinations ; Image reconstruction ; Image resolution ; Semantics ; Spatial resolution ; super-resolution</subject><ispartof>IEEE transactions on pattern analysis and machine intelligence, 2020-11, Vol.42 (11), p.2926-2943</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2020</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c351t-dff16994af1e89c3c858e311f043207bbd026e9a3de6c065d21651e71aa7180e3</citedby><cites>FETCH-LOGICAL-c351t-dff16994af1e89c3c858e311f043207bbd026e9a3de6c065d21651e71aa7180e3</cites><orcidid>0000-0002-0269-5649 ; 0000-0002-1520-4466 ; 0000-0002-5005-0191 ; 0000-0002-6920-9916</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/8713923$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,796,27924,27925,54758</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/8713923$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/31095477$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Yu, Xin</creatorcontrib><creatorcontrib>Fernando, Basura</creatorcontrib><creatorcontrib>Hartley, Richard</creatorcontrib><creatorcontrib>Porikli, Fatih</creatorcontrib><title>Semantic Face Hallucination: Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes</title><title>IEEE transactions on pattern analysis and machine intelligence</title><addtitle>TPAMI</addtitle><addtitle>IEEE Trans Pattern Anal Mach Intell</addtitle><description>Given a tiny face image, existing face hallucination methods aim at super-resolving its high-resolution (HR) counterpart by learning a mapping from an exemplary dataset. Since a low-resolution (LR) input patch may correspond to many HR candidate patches, this ambiguity may lead to distorted HR facial details and wrong attributes such as gender reversal and rejuvenation. An LR input contains low-frequency facial components of its HR version while its residual face image, defined as the difference between the HR ground-truth and interpolated LR images, contains the missing high-frequency facial details. We demonstrate that supplementing residual images or feature maps with additional facial attribute information can significantly reduce the ambiguity in face super-resolution. To explore this idea, we develop an attribute-embedded upsampling network, which consists of an upsampling network and a discriminative network. The upsampling network is composed of an autoencoder with skip-connections, which incorporates facial attribute vectors into the residual features of LR inputs at the bottleneck of the autoencoder, and deconvolutional layers used for upsampling. The discriminative network is designed to examine whether super-resolved faces contain the desired attributes or not and then its loss is used for updating the upsampling network. In this manner, we can super-resolve tiny (16×16 pixels) unaligned face images with a large upscaling factor of 8× while reducing the uncertainty of one-to-many mappings remarkably. By conducting extensive evaluations on a large-scale dataset, we demonstrate that our method achieves superior face hallucination results and outperforms the state-of-the-art.</description><subject>Ambiguity</subject><subject>attribute</subject><subject>Datasets</subject><subject>Face</subject><subject>Facial features</subject><subject>Feature extraction</subject><subject>Feature maps</subject><subject>hallucination</subject><subject>Hallucinations</subject><subject>Image reconstruction</subject><subject>Image resolution</subject><subject>Semantics</subject><subject>Spatial resolution</subject><subject>super-resolution</subject><issn>0162-8828</issn><issn>1939-3539</issn><issn>2160-9292</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2020</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNpdkctKAzEUhoMoWi8voCADbtxMzUnmkrgr4qVQUbxtQ5o5oylzqZOM4tubOtWFq8DJ9_2c5CfkEOgYgMqzp_vJ7XTMKMgxk5AJARtkBJLLmKdcbpIRhYzFQjCxQ3adW1AKSUr5NtnhQU-TPB-RxSPWuvHWRFfaYHSjq6o3ttHets159NgvsYsf0LXVh21eoxfsvqJZ-zmM-hU0eNNav6KLPq1_W0nLCmtsvA70xPvOznuPbp9slbpyeLA-98jz1eXTxU08u7ueXkxmseEp-LgoS8ikTHQJKKThRqQCOUBJE85oPp8XlGUoNS8wMzRLCwZZCpiD1jkIinyPnA65y65979F5VVtnsKp0g23vFGMhJ5WS8oCe_EMXbd81YTvFkkQkSS4oBIoNlOla5zos1bKzdXicAqpWTaifJtSqCbVuIkjH6-h-XmPxp_x-fQCOBsAi4t-1yIFLxvk3PYqNFQ</recordid><startdate>20201101</startdate><enddate>20201101</enddate><creator>Yu, Xin</creator><creator>Fernando, Basura</creator><creator>Hartley, Richard</creator><creator>Porikli, Fatih</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>7SP</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>7X8</scope><orcidid>https://orcid.org/0000-0002-0269-5649</orcidid><orcidid>https://orcid.org/0000-0002-1520-4466</orcidid><orcidid>https://orcid.org/0000-0002-5005-0191</orcidid><orcidid>https://orcid.org/0000-0002-6920-9916</orcidid></search><sort><creationdate>20201101</creationdate><title>Semantic Face Hallucination: Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes</title><author>Yu, Xin ; Fernando, Basura ; Hartley, Richard ; Porikli, Fatih</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c351t-dff16994af1e89c3c858e311f043207bbd026e9a3de6c065d21651e71aa7180e3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2020</creationdate><topic>Ambiguity</topic><topic>attribute</topic><topic>Datasets</topic><topic>Face</topic><topic>Facial features</topic><topic>Feature extraction</topic><topic>Feature maps</topic><topic>hallucination</topic><topic>Hallucinations</topic><topic>Image reconstruction</topic><topic>Image resolution</topic><topic>Semantics</topic><topic>Spatial resolution</topic><topic>super-resolution</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Yu, Xin</creatorcontrib><creatorcontrib>Fernando, Basura</creatorcontrib><creatorcontrib>Hartley, Richard</creatorcontrib><creatorcontrib>Porikli, Fatih</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Electronics &amp; Communications Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>MEDLINE - Academic</collection><jtitle>IEEE transactions on pattern analysis and machine intelligence</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Yu, Xin</au><au>Fernando, Basura</au><au>Hartley, Richard</au><au>Porikli, Fatih</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Semantic Face Hallucination: Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes</atitle><jtitle>IEEE transactions on pattern analysis and machine intelligence</jtitle><stitle>TPAMI</stitle><addtitle>IEEE Trans Pattern Anal Mach Intell</addtitle><date>2020-11-01</date><risdate>2020</risdate><volume>42</volume><issue>11</issue><spage>2926</spage><epage>2943</epage><pages>2926-2943</pages><issn>0162-8828</issn><eissn>1939-3539</eissn><eissn>2160-9292</eissn><coden>ITPIDJ</coden><abstract>Given a tiny face image, existing face hallucination methods aim at super-resolving its high-resolution (HR) counterpart by learning a mapping from an exemplary dataset. Since a low-resolution (LR) input patch may correspond to many HR candidate patches, this ambiguity may lead to distorted HR facial details and wrong attributes such as gender reversal and rejuvenation. An LR input contains low-frequency facial components of its HR version while its residual face image, defined as the difference between the HR ground-truth and interpolated LR images, contains the missing high-frequency facial details. We demonstrate that supplementing residual images or feature maps with additional facial attribute information can significantly reduce the ambiguity in face super-resolution. To explore this idea, we develop an attribute-embedded upsampling network, which consists of an upsampling network and a discriminative network. The upsampling network is composed of an autoencoder with skip-connections, which incorporates facial attribute vectors into the residual features of LR inputs at the bottleneck of the autoencoder, and deconvolutional layers used for upsampling. The discriminative network is designed to examine whether super-resolved faces contain the desired attributes or not and then its loss is used for updating the upsampling network. In this manner, we can super-resolve tiny (16×16 pixels) unaligned face images with a large upscaling factor of 8× while reducing the uncertainty of one-to-many mappings remarkably. By conducting extensive evaluations on a large-scale dataset, we demonstrate that our method achieves superior face hallucination results and outperforms the state-of-the-art.</abstract><cop>United States</cop><pub>IEEE</pub><pmid>31095477</pmid><doi>10.1109/TPAMI.2019.2916881</doi><tpages>18</tpages><orcidid>https://orcid.org/0000-0002-0269-5649</orcidid><orcidid>https://orcid.org/0000-0002-1520-4466</orcidid><orcidid>https://orcid.org/0000-0002-5005-0191</orcidid><orcidid>https://orcid.org/0000-0002-6920-9916</orcidid></addata></record>
fulltext fulltext_linktorsrc
identifier ISSN: 0162-8828
ispartof IEEE transactions on pattern analysis and machine intelligence, 2020-11, Vol.42 (11), p.2926-2943
issn 0162-8828
1939-3539
2160-9292
language eng
recordid cdi_crossref_primary_10_1109_TPAMI_2019_2916881
source IEEE Electronic Library (IEL)
subjects Ambiguity
attribute
Datasets
Face
Facial features
Feature extraction
Feature maps
hallucination
Hallucinations
Image reconstruction
Image resolution
Semantics
Spatial resolution
super-resolution
title Semantic Face Hallucination: Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T14%3A36%3A29IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Semantic%20Face%20Hallucination:%20Super-Resolving%20Very%20Low-Resolution%20Face%20Images%20with%20Supplementary%20Attributes&rft.jtitle=IEEE%20transactions%20on%20pattern%20analysis%20and%20machine%20intelligence&rft.au=Yu,%20Xin&rft.date=2020-11-01&rft.volume=42&rft.issue=11&rft.spage=2926&rft.epage=2943&rft.pages=2926-2943&rft.issn=0162-8828&rft.eissn=1939-3539&rft.coden=ITPIDJ&rft_id=info:doi/10.1109/TPAMI.2019.2916881&rft_dat=%3Cproquest_RIE%3E2232059903%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2448447801&rft_id=info:pmid/31095477&rft_ieee_id=8713923&rfr_iscdi=true