Space-time video super-resolution via multi-scale feature interpolation and temporal feature fusion
The goal of Space-Time Video Super-Resolution (STVSR) is to simultaneously increase the spatial resolution and frame rate of low-resolution, low-frame-rate video. In response to the problem that the STVSR method does not fully consider the spatio-temporal correlation between successive video frames,...
Gespeichert in:
Veröffentlicht in: | Signal, image and video processing image and video processing, 2024-11, Vol.18 (11), p.8279-8291 |
---|---|
Hauptverfasser: | , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | 8291 |
---|---|
container_issue | 11 |
container_start_page | 8279 |
container_title | Signal, image and video processing |
container_volume | 18 |
creator | Yang, Caisong Kong, Guangqian Duan, Xun Long, Huiyun Zhao, Jian |
description | The goal of Space-Time Video Super-Resolution (STVSR) is to simultaneously increase the spatial resolution and frame rate of low-resolution, low-frame-rate video. In response to the problem that the STVSR method does not fully consider the spatio-temporal correlation between successive video frames, which makes the video frame reconstruction results unsatisfactory, and the problem that the inference speed of large models is slow. This paper proposes a STVSR method based on Multi-Scale Feature Interpolation and Temporal Feature Fusion (MSITF). First, feature interpolation is performed in the low-resolution feature space to obtain the features corresponding to the missing frames. The feature is then enhanced using deformable convolution with the aim of obtaining a more accurate feature of the missing frames. Finally, the temporal alignment and global context learning of sequence frame features are performed by a temporal feature fusion module to fully extract and utilize the useful spatio-temporal information in adjacent frames, resulting in better quality of the reconstructed video frames. Extensive experiments on the benchmark datasets Vid4 and Vimeo-90k show that the proposed method achieves better qualitative and quantitative performance, with PSNR and SSIM on the Vid4 dataset improving by 0.8% and 1.9%, respectively, over the state-of-the-art two-stage method AdaCof+TTVSR, and MSITF improved by 1.2% and 2.5%, respectively, compared to single-stage method RSTT. The number of parameters decreased by 80.4% and 8.2% compared to the AdaCof+TTVSR and RSTT, respectively.We release our code at
https://github.com/carpenterChina/MSITF. |
doi_str_mv | 10.1007/s11760-024-03469-7 |
format | Article |
fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_3104475733</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3104475733</sourcerecordid><originalsourceid>FETCH-LOGICAL-c200t-6fb5306d3cd09e878dfc2452d020344be819dbf789dde39199b8be2117bc941b3</originalsourceid><addsrcrecordid>eNp9kE1LxDAQhoMouKz7BzwVPEfz0TbJURa_YMGDeg5pMpUubVOTVPDfG7ey3pzLDDPPO8O8CF1Sck0JETeRUlETTFiJCS9rhcUJWlFZc0wFpafHmvBztIlxT3JwJmQtV8i-TMYCTt0AxWfnwBdxniDgANH3c-r8mNumGOY-dTha00PRgklzgKIbE4TJ9-ZAmdEVCYbJB9MfkXaOeXaBzlrTR9j85jV6u7973T7i3fPD0_Z2hy0jJOG6bSpOasetIwqkkK61rKyYIyy_VTYgqXJNK6RyDriiSjWyAZafb6wqacPX6GrZOwX_MUNMeu_nMOaTmlNSlqISnGeKLZQNPsYArZ5CN5jwpSnRP37qxU-d_dQHP7XIIr6IYobHdwh_q_9RfQN5Nnmm</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3104475733</pqid></control><display><type>article</type><title>Space-time video super-resolution via multi-scale feature interpolation and temporal feature fusion</title><source>Springer Journals</source><creator>Yang, Caisong ; Kong, Guangqian ; Duan, Xun ; Long, Huiyun ; Zhao, Jian</creator><creatorcontrib>Yang, Caisong ; Kong, Guangqian ; Duan, Xun ; Long, Huiyun ; Zhao, Jian</creatorcontrib><description>The goal of Space-Time Video Super-Resolution (STVSR) is to simultaneously increase the spatial resolution and frame rate of low-resolution, low-frame-rate video. In response to the problem that the STVSR method does not fully consider the spatio-temporal correlation between successive video frames, which makes the video frame reconstruction results unsatisfactory, and the problem that the inference speed of large models is slow. This paper proposes a STVSR method based on Multi-Scale Feature Interpolation and Temporal Feature Fusion (MSITF). First, feature interpolation is performed in the low-resolution feature space to obtain the features corresponding to the missing frames. The feature is then enhanced using deformable convolution with the aim of obtaining a more accurate feature of the missing frames. Finally, the temporal alignment and global context learning of sequence frame features are performed by a temporal feature fusion module to fully extract and utilize the useful spatio-temporal information in adjacent frames, resulting in better quality of the reconstructed video frames. Extensive experiments on the benchmark datasets Vid4 and Vimeo-90k show that the proposed method achieves better qualitative and quantitative performance, with PSNR and SSIM on the Vid4 dataset improving by 0.8% and 1.9%, respectively, over the state-of-the-art two-stage method AdaCof+TTVSR, and MSITF improved by 1.2% and 2.5%, respectively, compared to single-stage method RSTT. The number of parameters decreased by 80.4% and 8.2% compared to the AdaCof+TTVSR and RSTT, respectively.We release our code at
https://github.com/carpenterChina/MSITF.</description><identifier>ISSN: 1863-1703</identifier><identifier>EISSN: 1863-1711</identifier><identifier>DOI: 10.1007/s11760-024-03469-7</identifier><language>eng</language><publisher>London: Springer London</publisher><subject>Computer Imaging ; Computer Science ; Datasets ; Formability ; Frames (data processing) ; Image Processing and Computer Vision ; Interpolation ; Multimedia Information Systems ; Original Paper ; Pattern Recognition and Graphics ; Signal,Image and Speech Processing ; Spacetime ; Spatial resolution ; Spatiotemporal data ; Vision</subject><ispartof>Signal, image and video processing, 2024-11, Vol.18 (11), p.8279-8291</ispartof><rights>The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2024. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c200t-6fb5306d3cd09e878dfc2452d020344be819dbf789dde39199b8be2117bc941b3</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://link.springer.com/content/pdf/10.1007/s11760-024-03469-7$$EPDF$$P50$$Gspringer$$H</linktopdf><linktohtml>$$Uhttps://link.springer.com/10.1007/s11760-024-03469-7$$EHTML$$P50$$Gspringer$$H</linktohtml><link.rule.ids>315,781,785,27926,27927,41490,42559,51321</link.rule.ids></links><search><creatorcontrib>Yang, Caisong</creatorcontrib><creatorcontrib>Kong, Guangqian</creatorcontrib><creatorcontrib>Duan, Xun</creatorcontrib><creatorcontrib>Long, Huiyun</creatorcontrib><creatorcontrib>Zhao, Jian</creatorcontrib><title>Space-time video super-resolution via multi-scale feature interpolation and temporal feature fusion</title><title>Signal, image and video processing</title><addtitle>SIViP</addtitle><description>The goal of Space-Time Video Super-Resolution (STVSR) is to simultaneously increase the spatial resolution and frame rate of low-resolution, low-frame-rate video. In response to the problem that the STVSR method does not fully consider the spatio-temporal correlation between successive video frames, which makes the video frame reconstruction results unsatisfactory, and the problem that the inference speed of large models is slow. This paper proposes a STVSR method based on Multi-Scale Feature Interpolation and Temporal Feature Fusion (MSITF). First, feature interpolation is performed in the low-resolution feature space to obtain the features corresponding to the missing frames. The feature is then enhanced using deformable convolution with the aim of obtaining a more accurate feature of the missing frames. Finally, the temporal alignment and global context learning of sequence frame features are performed by a temporal feature fusion module to fully extract and utilize the useful spatio-temporal information in adjacent frames, resulting in better quality of the reconstructed video frames. Extensive experiments on the benchmark datasets Vid4 and Vimeo-90k show that the proposed method achieves better qualitative and quantitative performance, with PSNR and SSIM on the Vid4 dataset improving by 0.8% and 1.9%, respectively, over the state-of-the-art two-stage method AdaCof+TTVSR, and MSITF improved by 1.2% and 2.5%, respectively, compared to single-stage method RSTT. The number of parameters decreased by 80.4% and 8.2% compared to the AdaCof+TTVSR and RSTT, respectively.We release our code at
https://github.com/carpenterChina/MSITF.</description><subject>Computer Imaging</subject><subject>Computer Science</subject><subject>Datasets</subject><subject>Formability</subject><subject>Frames (data processing)</subject><subject>Image Processing and Computer Vision</subject><subject>Interpolation</subject><subject>Multimedia Information Systems</subject><subject>Original Paper</subject><subject>Pattern Recognition and Graphics</subject><subject>Signal,Image and Speech Processing</subject><subject>Spacetime</subject><subject>Spatial resolution</subject><subject>Spatiotemporal data</subject><subject>Vision</subject><issn>1863-1703</issn><issn>1863-1711</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><recordid>eNp9kE1LxDAQhoMouKz7BzwVPEfz0TbJURa_YMGDeg5pMpUubVOTVPDfG7ey3pzLDDPPO8O8CF1Sck0JETeRUlETTFiJCS9rhcUJWlFZc0wFpafHmvBztIlxT3JwJmQtV8i-TMYCTt0AxWfnwBdxniDgANH3c-r8mNumGOY-dTha00PRgklzgKIbE4TJ9-ZAmdEVCYbJB9MfkXaOeXaBzlrTR9j85jV6u7973T7i3fPD0_Z2hy0jJOG6bSpOasetIwqkkK61rKyYIyy_VTYgqXJNK6RyDriiSjWyAZafb6wqacPX6GrZOwX_MUNMeu_nMOaTmlNSlqISnGeKLZQNPsYArZ5CN5jwpSnRP37qxU-d_dQHP7XIIr6IYobHdwh_q_9RfQN5Nnmm</recordid><startdate>20241101</startdate><enddate>20241101</enddate><creator>Yang, Caisong</creator><creator>Kong, Guangqian</creator><creator>Duan, Xun</creator><creator>Long, Huiyun</creator><creator>Zhao, Jian</creator><general>Springer London</general><general>Springer Nature B.V</general><scope>AAYXX</scope><scope>CITATION</scope></search><sort><creationdate>20241101</creationdate><title>Space-time video super-resolution via multi-scale feature interpolation and temporal feature fusion</title><author>Yang, Caisong ; Kong, Guangqian ; Duan, Xun ; Long, Huiyun ; Zhao, Jian</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c200t-6fb5306d3cd09e878dfc2452d020344be819dbf789dde39199b8be2117bc941b3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Imaging</topic><topic>Computer Science</topic><topic>Datasets</topic><topic>Formability</topic><topic>Frames (data processing)</topic><topic>Image Processing and Computer Vision</topic><topic>Interpolation</topic><topic>Multimedia Information Systems</topic><topic>Original Paper</topic><topic>Pattern Recognition and Graphics</topic><topic>Signal,Image and Speech Processing</topic><topic>Spacetime</topic><topic>Spatial resolution</topic><topic>Spatiotemporal data</topic><topic>Vision</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Yang, Caisong</creatorcontrib><creatorcontrib>Kong, Guangqian</creatorcontrib><creatorcontrib>Duan, Xun</creatorcontrib><creatorcontrib>Long, Huiyun</creatorcontrib><creatorcontrib>Zhao, Jian</creatorcontrib><collection>CrossRef</collection><jtitle>Signal, image and video processing</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Yang, Caisong</au><au>Kong, Guangqian</au><au>Duan, Xun</au><au>Long, Huiyun</au><au>Zhao, Jian</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Space-time video super-resolution via multi-scale feature interpolation and temporal feature fusion</atitle><jtitle>Signal, image and video processing</jtitle><stitle>SIViP</stitle><date>2024-11-01</date><risdate>2024</risdate><volume>18</volume><issue>11</issue><spage>8279</spage><epage>8291</epage><pages>8279-8291</pages><issn>1863-1703</issn><eissn>1863-1711</eissn><abstract>The goal of Space-Time Video Super-Resolution (STVSR) is to simultaneously increase the spatial resolution and frame rate of low-resolution, low-frame-rate video. In response to the problem that the STVSR method does not fully consider the spatio-temporal correlation between successive video frames, which makes the video frame reconstruction results unsatisfactory, and the problem that the inference speed of large models is slow. This paper proposes a STVSR method based on Multi-Scale Feature Interpolation and Temporal Feature Fusion (MSITF). First, feature interpolation is performed in the low-resolution feature space to obtain the features corresponding to the missing frames. The feature is then enhanced using deformable convolution with the aim of obtaining a more accurate feature of the missing frames. Finally, the temporal alignment and global context learning of sequence frame features are performed by a temporal feature fusion module to fully extract and utilize the useful spatio-temporal information in adjacent frames, resulting in better quality of the reconstructed video frames. Extensive experiments on the benchmark datasets Vid4 and Vimeo-90k show that the proposed method achieves better qualitative and quantitative performance, with PSNR and SSIM on the Vid4 dataset improving by 0.8% and 1.9%, respectively, over the state-of-the-art two-stage method AdaCof+TTVSR, and MSITF improved by 1.2% and 2.5%, respectively, compared to single-stage method RSTT. The number of parameters decreased by 80.4% and 8.2% compared to the AdaCof+TTVSR and RSTT, respectively.We release our code at
https://github.com/carpenterChina/MSITF.</abstract><cop>London</cop><pub>Springer London</pub><doi>10.1007/s11760-024-03469-7</doi><tpages>13</tpages></addata></record> |
fulltext | fulltext |
identifier | ISSN: 1863-1703 |
ispartof | Signal, image and video processing, 2024-11, Vol.18 (11), p.8279-8291 |
issn | 1863-1703 1863-1711 |
language | eng |
recordid | cdi_proquest_journals_3104475733 |
source | Springer Journals |
subjects | Computer Imaging Computer Science Datasets Formability Frames (data processing) Image Processing and Computer Vision Interpolation Multimedia Information Systems Original Paper Pattern Recognition and Graphics Signal,Image and Speech Processing Spacetime Spatial resolution Spatiotemporal data Vision |
title | Space-time video super-resolution via multi-scale feature interpolation and temporal feature fusion |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-17T21%3A02%3A02IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Space-time%20video%20super-resolution%20via%20multi-scale%20feature%20interpolation%20and%20temporal%20feature%20fusion&rft.jtitle=Signal,%20image%20and%20video%20processing&rft.au=Yang,%20Caisong&rft.date=2024-11-01&rft.volume=18&rft.issue=11&rft.spage=8279&rft.epage=8291&rft.pages=8279-8291&rft.issn=1863-1703&rft.eissn=1863-1711&rft_id=info:doi/10.1007/s11760-024-03469-7&rft_dat=%3Cproquest_cross%3E3104475733%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3104475733&rft_id=info:pmid/&rfr_iscdi=true |