End-to-End Learning of Video Super-Resolution with Motion Compensation

Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this p...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Makansi, Osama, Ilg, Eddy, Brox, Thomas
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Makansi, Osama
Ilg, Eddy
Brox, Thomas
description Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.
doi_str_mv 10.48550/arxiv.1707.00471
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_1707_00471</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>1707_00471</sourcerecordid><originalsourceid>FETCH-LOGICAL-a671-815b16ec0dfe722357ce22fc8de204fbbeefc453b506fdf1b136af3c26d7a30a3</originalsourceid><addsrcrecordid>eNotj7tOwzAYRr0woJYHYMIv4OBLbGetohaQgpDaijXy5XdrqbUjJ-Xy9tDAdL5vOdJB6J7Rqm6kpI-mfMWPimmqK0przW7RZp08mTL5Be7AlBTTAeeA36OHjHeXAQrZwphPlynmhD_jdMSved5tPg-QRnM9S3QTzGmEu38u0H6z3rfPpHt7emlXHTFKM9IwaZkCR30AzbmQ2gHnwTUeOK2DtQDB1VJYSVXwgVkmlAnCceW1EdSIBXr4084h_VDi2ZTv_hrUz0HiB-xNRqM</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>End-to-End Learning of Video Super-Resolution with Motion Compensation</title><source>arXiv.org</source><creator>Makansi, Osama ; Ilg, Eddy ; Brox, Thomas</creator><creatorcontrib>Makansi, Osama ; Ilg, Eddy ; Brox, Thomas</creatorcontrib><description>Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.</description><identifier>DOI: 10.48550/arxiv.1707.00471</identifier><language>eng</language><subject>Computer Science - Computer Vision and Pattern Recognition</subject><creationdate>2017-07</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/1707.00471$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.1707.00471$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Makansi, Osama</creatorcontrib><creatorcontrib>Ilg, Eddy</creatorcontrib><creatorcontrib>Brox, Thomas</creatorcontrib><title>End-to-End Learning of Video Super-Resolution with Motion Compensation</title><description>Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.</description><subject>Computer Science - Computer Vision and Pattern Recognition</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2017</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotj7tOwzAYRr0woJYHYMIv4OBLbGetohaQgpDaijXy5XdrqbUjJ-Xy9tDAdL5vOdJB6J7Rqm6kpI-mfMWPimmqK0przW7RZp08mTL5Be7AlBTTAeeA36OHjHeXAQrZwphPlynmhD_jdMSved5tPg-QRnM9S3QTzGmEu38u0H6z3rfPpHt7emlXHTFKM9IwaZkCR30AzbmQ2gHnwTUeOK2DtQDB1VJYSVXwgVkmlAnCceW1EdSIBXr4084h_VDi2ZTv_hrUz0HiB-xNRqM</recordid><startdate>20170703</startdate><enddate>20170703</enddate><creator>Makansi, Osama</creator><creator>Ilg, Eddy</creator><creator>Brox, Thomas</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20170703</creationdate><title>End-to-End Learning of Video Super-Resolution with Motion Compensation</title><author>Makansi, Osama ; Ilg, Eddy ; Brox, Thomas</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a671-815b16ec0dfe722357ce22fc8de204fbbeefc453b506fdf1b136af3c26d7a30a3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2017</creationdate><topic>Computer Science - Computer Vision and Pattern Recognition</topic><toplevel>online_resources</toplevel><creatorcontrib>Makansi, Osama</creatorcontrib><creatorcontrib>Ilg, Eddy</creatorcontrib><creatorcontrib>Brox, Thomas</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Makansi, Osama</au><au>Ilg, Eddy</au><au>Brox, Thomas</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>End-to-End Learning of Video Super-Resolution with Motion Compensation</atitle><date>2017-07-03</date><risdate>2017</risdate><abstract>Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.</abstract><doi>10.48550/arxiv.1707.00471</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.1707.00471
ispartof
issn
language eng
recordid cdi_arxiv_primary_1707_00471
source arXiv.org
subjects Computer Science - Computer Vision and Pattern Recognition
title End-to-End Learning of Video Super-Resolution with Motion Compensation
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T13%3A05%3A39IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=End-to-End%20Learning%20of%20Video%20Super-Resolution%20with%20Motion%20Compensation&rft.au=Makansi,%20Osama&rft.date=2017-07-03&rft_id=info:doi/10.48550/arxiv.1707.00471&rft_dat=%3Carxiv_GOX%3E1707_00471%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true