Lightweight Deep Exemplar Colorization via Semantic Attention-Guided Laplacian Pyramid
Exemplar-based colorization aims to generate plausible colors for a grayscale image with the guidance of a color reference image. The main challenging problem is finding the correct semantic correspondence between the target image and the reference image. However, the colors of the object and backgr...
Gespeichert in:
Veröffentlicht in: | IEEE transactions on visualization and computer graphics 2024-05, Vol.PP, p.1-12 |
---|---|
Hauptverfasser: | , , , , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | 12 |
---|---|
container_issue | |
container_start_page | 1 |
container_title | IEEE transactions on visualization and computer graphics |
container_volume | PP |
creator | Zou, Chengyi Wan, Shuai Blanch, Marc Gorriz Murn, Luka Mrak, Marta Sock, Juil Yang, Fei Herranz, Luis |
description | Exemplar-based colorization aims to generate plausible colors for a grayscale image with the guidance of a color reference image. The main challenging problem is finding the correct semantic correspondence between the target image and the reference image. However, the colors of the object and background are often confused in the existing methods. Besides, these methods usually use simple encoder-decoder architectures or pyramid structures to extract features and lack appropriate fusion mechanisms, which results in the loss of high-frequency information or high complexity. To address these problems, this paper proposes a lightweight semantic attention-guided Laplacian pyramid network (SAGLP-Net) for deep exemplar-based colorization, exploiting the inherent multi-scale properties of color representations. They are exploited through a Laplacian pyramid, and semantic information is introduced as high-level guidance to align the object and background information. Specially, a semantic guided non-local attention fusion module is designed to exploit the long-range dependency and fuse the local and global features. Moreover, a Laplacian pyramid fusion module based on criss-cross attention is proposed to fuse high frequency components in the large-scale domain. An unsupervised multi-scale multi-loss training strategy is further introduced for network training, which combines pixel loss, color histogram loss, total variance regularisation, and adversarial loss. Experimental results demonstrate that our colorization method achieves better subjective and objective performance with lower complexity than the state-of-the-art methods. |
doi_str_mv | 10.1109/TVCG.2024.3398791 |
format | Article |
fullrecord | <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_proquest_miscellaneous_3053979793</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>10526459</ieee_id><sourcerecordid>3053979793</sourcerecordid><originalsourceid>FETCH-LOGICAL-c189t-73e4742775da162ce9aff50012b85bd1ac6db5be5326eaca7524afac0eae89153</originalsourceid><addsrcrecordid>eNpNkFtLw0AQhRdRbK3-AEEkj76k7iWbzT6WWKsQULD2NUw2E13JzVzU-utNaBUZmBmGcw7DR8g5o3PGqL5eb8LVnFPuzYXQgdLsgEyZ9phLJfUPh50q5XKf-xNy0rZvlDLPC_QxmYhAca44nZJNZF9eu08cu3ODWDvLLyzqHBonrPKqsd_Q2ap0Piw4T1hA2VnjLLoOy_HsrnqbYupEMDiMhdJ53DZQ2PSUHGWQt3i2nzPyfLtch3du9LC6DxeRa1igO1cJ9JTHlZIpMJ8b1JBlcviTJ4FMUgbGTxOZoBTcRzCgJPcgA0MRMNBMihm52uXWTfXeY9vFhW0N5jmUWPVtLKgUWg0lBinbSU1TtW2DWVw3toBmGzMajzjjEWc84oz3OAfP5T6-TwpM_xy__AbBxU5gEfFfoOS-J7X4AWPnemc</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3053979793</pqid></control><display><type>article</type><title>Lightweight Deep Exemplar Colorization via Semantic Attention-Guided Laplacian Pyramid</title><source>IEEE Electronic Library (IEL)</source><creator>Zou, Chengyi ; Wan, Shuai ; Blanch, Marc Gorriz ; Murn, Luka ; Mrak, Marta ; Sock, Juil ; Yang, Fei ; Herranz, Luis</creator><creatorcontrib>Zou, Chengyi ; Wan, Shuai ; Blanch, Marc Gorriz ; Murn, Luka ; Mrak, Marta ; Sock, Juil ; Yang, Fei ; Herranz, Luis</creatorcontrib><description>Exemplar-based colorization aims to generate plausible colors for a grayscale image with the guidance of a color reference image. The main challenging problem is finding the correct semantic correspondence between the target image and the reference image. However, the colors of the object and background are often confused in the existing methods. Besides, these methods usually use simple encoder-decoder architectures or pyramid structures to extract features and lack appropriate fusion mechanisms, which results in the loss of high-frequency information or high complexity. To address these problems, this paper proposes a lightweight semantic attention-guided Laplacian pyramid network (SAGLP-Net) for deep exemplar-based colorization, exploiting the inherent multi-scale properties of color representations. They are exploited through a Laplacian pyramid, and semantic information is introduced as high-level guidance to align the object and background information. Specially, a semantic guided non-local attention fusion module is designed to exploit the long-range dependency and fuse the local and global features. Moreover, a Laplacian pyramid fusion module based on criss-cross attention is proposed to fuse high frequency components in the large-scale domain. An unsupervised multi-scale multi-loss training strategy is further introduced for network training, which combines pixel loss, color histogram loss, total variance regularisation, and adversarial loss. Experimental results demonstrate that our colorization method achieves better subjective and objective performance with lower complexity than the state-of-the-art methods.</description><identifier>ISSN: 1077-2626</identifier><identifier>EISSN: 1941-0506</identifier><identifier>DOI: 10.1109/TVCG.2024.3398791</identifier><identifier>PMID: 38722720</identifier><identifier>CODEN: ITVGEA</identifier><language>eng</language><publisher>United States: IEEE</publisher><subject>attention mechanism ; Exemplar-based colorization ; Feature extraction ; Gray-scale ; Image color analysis ; Laplace equations ; Laplacian pyramid network ; semantic guidance ; Semantics ; Task analysis ; Transformers</subject><ispartof>IEEE transactions on visualization and computer graphics, 2024-05, Vol.PP, p.1-12</ispartof><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><orcidid>0000-0001-8617-149X ; 0000-0002-5780-4777 ; 0000-0003-4099-6511 ; 0000-0002-7777-0252 ; 0000-0002-7022-3395 ; 0009-0001-9186-6592 ; 0000-0002-7636-4981 ; 0000-0001-9041-647X</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/10526459$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,796,27924,27925,54758</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/10526459$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/38722720$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Zou, Chengyi</creatorcontrib><creatorcontrib>Wan, Shuai</creatorcontrib><creatorcontrib>Blanch, Marc Gorriz</creatorcontrib><creatorcontrib>Murn, Luka</creatorcontrib><creatorcontrib>Mrak, Marta</creatorcontrib><creatorcontrib>Sock, Juil</creatorcontrib><creatorcontrib>Yang, Fei</creatorcontrib><creatorcontrib>Herranz, Luis</creatorcontrib><title>Lightweight Deep Exemplar Colorization via Semantic Attention-Guided Laplacian Pyramid</title><title>IEEE transactions on visualization and computer graphics</title><addtitle>TVCG</addtitle><addtitle>IEEE Trans Vis Comput Graph</addtitle><description>Exemplar-based colorization aims to generate plausible colors for a grayscale image with the guidance of a color reference image. The main challenging problem is finding the correct semantic correspondence between the target image and the reference image. However, the colors of the object and background are often confused in the existing methods. Besides, these methods usually use simple encoder-decoder architectures or pyramid structures to extract features and lack appropriate fusion mechanisms, which results in the loss of high-frequency information or high complexity. To address these problems, this paper proposes a lightweight semantic attention-guided Laplacian pyramid network (SAGLP-Net) for deep exemplar-based colorization, exploiting the inherent multi-scale properties of color representations. They are exploited through a Laplacian pyramid, and semantic information is introduced as high-level guidance to align the object and background information. Specially, a semantic guided non-local attention fusion module is designed to exploit the long-range dependency and fuse the local and global features. Moreover, a Laplacian pyramid fusion module based on criss-cross attention is proposed to fuse high frequency components in the large-scale domain. An unsupervised multi-scale multi-loss training strategy is further introduced for network training, which combines pixel loss, color histogram loss, total variance regularisation, and adversarial loss. Experimental results demonstrate that our colorization method achieves better subjective and objective performance with lower complexity than the state-of-the-art methods.</description><subject>attention mechanism</subject><subject>Exemplar-based colorization</subject><subject>Feature extraction</subject><subject>Gray-scale</subject><subject>Image color analysis</subject><subject>Laplace equations</subject><subject>Laplacian pyramid network</subject><subject>semantic guidance</subject><subject>Semantics</subject><subject>Task analysis</subject><subject>Transformers</subject><issn>1077-2626</issn><issn>1941-0506</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNpNkFtLw0AQhRdRbK3-AEEkj76k7iWbzT6WWKsQULD2NUw2E13JzVzU-utNaBUZmBmGcw7DR8g5o3PGqL5eb8LVnFPuzYXQgdLsgEyZ9phLJfUPh50q5XKf-xNy0rZvlDLPC_QxmYhAca44nZJNZF9eu08cu3ODWDvLLyzqHBonrPKqsd_Q2ap0Piw4T1hA2VnjLLoOy_HsrnqbYupEMDiMhdJ53DZQ2PSUHGWQt3i2nzPyfLtch3du9LC6DxeRa1igO1cJ9JTHlZIpMJ8b1JBlcviTJ4FMUgbGTxOZoBTcRzCgJPcgA0MRMNBMihm52uXWTfXeY9vFhW0N5jmUWPVtLKgUWg0lBinbSU1TtW2DWVw3toBmGzMajzjjEWc84oz3OAfP5T6-TwpM_xy__AbBxU5gEfFfoOS-J7X4AWPnemc</recordid><startdate>20240509</startdate><enddate>20240509</enddate><creator>Zou, Chengyi</creator><creator>Wan, Shuai</creator><creator>Blanch, Marc Gorriz</creator><creator>Murn, Luka</creator><creator>Mrak, Marta</creator><creator>Sock, Juil</creator><creator>Yang, Fei</creator><creator>Herranz, Luis</creator><general>IEEE</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7X8</scope><orcidid>https://orcid.org/0000-0001-8617-149X</orcidid><orcidid>https://orcid.org/0000-0002-5780-4777</orcidid><orcidid>https://orcid.org/0000-0003-4099-6511</orcidid><orcidid>https://orcid.org/0000-0002-7777-0252</orcidid><orcidid>https://orcid.org/0000-0002-7022-3395</orcidid><orcidid>https://orcid.org/0009-0001-9186-6592</orcidid><orcidid>https://orcid.org/0000-0002-7636-4981</orcidid><orcidid>https://orcid.org/0000-0001-9041-647X</orcidid></search><sort><creationdate>20240509</creationdate><title>Lightweight Deep Exemplar Colorization via Semantic Attention-Guided Laplacian Pyramid</title><author>Zou, Chengyi ; Wan, Shuai ; Blanch, Marc Gorriz ; Murn, Luka ; Mrak, Marta ; Sock, Juil ; Yang, Fei ; Herranz, Luis</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c189t-73e4742775da162ce9aff50012b85bd1ac6db5be5326eaca7524afac0eae89153</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>attention mechanism</topic><topic>Exemplar-based colorization</topic><topic>Feature extraction</topic><topic>Gray-scale</topic><topic>Image color analysis</topic><topic>Laplace equations</topic><topic>Laplacian pyramid network</topic><topic>semantic guidance</topic><topic>Semantics</topic><topic>Task analysis</topic><topic>Transformers</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Zou, Chengyi</creatorcontrib><creatorcontrib>Wan, Shuai</creatorcontrib><creatorcontrib>Blanch, Marc Gorriz</creatorcontrib><creatorcontrib>Murn, Luka</creatorcontrib><creatorcontrib>Mrak, Marta</creatorcontrib><creatorcontrib>Sock, Juil</creatorcontrib><creatorcontrib>Yang, Fei</creatorcontrib><creatorcontrib>Herranz, Luis</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>MEDLINE - Academic</collection><jtitle>IEEE transactions on visualization and computer graphics</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Zou, Chengyi</au><au>Wan, Shuai</au><au>Blanch, Marc Gorriz</au><au>Murn, Luka</au><au>Mrak, Marta</au><au>Sock, Juil</au><au>Yang, Fei</au><au>Herranz, Luis</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Lightweight Deep Exemplar Colorization via Semantic Attention-Guided Laplacian Pyramid</atitle><jtitle>IEEE transactions on visualization and computer graphics</jtitle><stitle>TVCG</stitle><addtitle>IEEE Trans Vis Comput Graph</addtitle><date>2024-05-09</date><risdate>2024</risdate><volume>PP</volume><spage>1</spage><epage>12</epage><pages>1-12</pages><issn>1077-2626</issn><eissn>1941-0506</eissn><coden>ITVGEA</coden><abstract>Exemplar-based colorization aims to generate plausible colors for a grayscale image with the guidance of a color reference image. The main challenging problem is finding the correct semantic correspondence between the target image and the reference image. However, the colors of the object and background are often confused in the existing methods. Besides, these methods usually use simple encoder-decoder architectures or pyramid structures to extract features and lack appropriate fusion mechanisms, which results in the loss of high-frequency information or high complexity. To address these problems, this paper proposes a lightweight semantic attention-guided Laplacian pyramid network (SAGLP-Net) for deep exemplar-based colorization, exploiting the inherent multi-scale properties of color representations. They are exploited through a Laplacian pyramid, and semantic information is introduced as high-level guidance to align the object and background information. Specially, a semantic guided non-local attention fusion module is designed to exploit the long-range dependency and fuse the local and global features. Moreover, a Laplacian pyramid fusion module based on criss-cross attention is proposed to fuse high frequency components in the large-scale domain. An unsupervised multi-scale multi-loss training strategy is further introduced for network training, which combines pixel loss, color histogram loss, total variance regularisation, and adversarial loss. Experimental results demonstrate that our colorization method achieves better subjective and objective performance with lower complexity than the state-of-the-art methods.</abstract><cop>United States</cop><pub>IEEE</pub><pmid>38722720</pmid><doi>10.1109/TVCG.2024.3398791</doi><tpages>12</tpages><orcidid>https://orcid.org/0000-0001-8617-149X</orcidid><orcidid>https://orcid.org/0000-0002-5780-4777</orcidid><orcidid>https://orcid.org/0000-0003-4099-6511</orcidid><orcidid>https://orcid.org/0000-0002-7777-0252</orcidid><orcidid>https://orcid.org/0000-0002-7022-3395</orcidid><orcidid>https://orcid.org/0009-0001-9186-6592</orcidid><orcidid>https://orcid.org/0000-0002-7636-4981</orcidid><orcidid>https://orcid.org/0000-0001-9041-647X</orcidid></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | ISSN: 1077-2626 |
ispartof | IEEE transactions on visualization and computer graphics, 2024-05, Vol.PP, p.1-12 |
issn | 1077-2626 1941-0506 |
language | eng |
recordid | cdi_proquest_miscellaneous_3053979793 |
source | IEEE Electronic Library (IEL) |
subjects | attention mechanism Exemplar-based colorization Feature extraction Gray-scale Image color analysis Laplace equations Laplacian pyramid network semantic guidance Semantics Task analysis Transformers |
title | Lightweight Deep Exemplar Colorization via Semantic Attention-Guided Laplacian Pyramid |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T22%3A35%3A02IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Lightweight%20Deep%20Exemplar%20Colorization%20via%20Semantic%20Attention-Guided%20Laplacian%20Pyramid&rft.jtitle=IEEE%20transactions%20on%20visualization%20and%20computer%20graphics&rft.au=Zou,%20Chengyi&rft.date=2024-05-09&rft.volume=PP&rft.spage=1&rft.epage=12&rft.pages=1-12&rft.issn=1077-2626&rft.eissn=1941-0506&rft.coden=ITVGEA&rft_id=info:doi/10.1109/TVCG.2024.3398791&rft_dat=%3Cproquest_RIE%3E3053979793%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3053979793&rft_id=info:pmid/38722720&rft_ieee_id=10526459&rfr_iscdi=true |