[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Reference-based image super-resolution with attention extraction and pooling of residuals

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Reference-based image super-resolution aims to improve the clarity of input low-resolution (LR) images by leveraging additional high-resolution reference (Ref) images. Although existing methods effectively address the problems associated with super-resolution reconstruction based on reference images, there are still significant challenges in effectively bridging the gap between LR and Ref and enhancing the finesse of texture details. This paper presents MARP-SR, which utilizes attention and residual pooling techniques to bridge their resolution gaps while enhancing texture details. The key innovations include the pooled residual feature enhancement (PRFE) module and the attention extraction feature (AEF) module. PRFE employs residual learning through convolution and pooling to boost texture and salient high-frequency features. AEF utilizes attention mechanisms to extract multi-scale features from LR and Ref images, aligning their resolutions. Experiments on benchmark datasets (CUFED5, Urban100, Manga109, and WR-SR) demonstrate that our proposed MARP-SR achieves state-of-the-art performance, generating more realistic and natural-looking super-resolution results than previous methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The CUFED5 dataset was developed by Zhang et al. [54] for training and testing reference-based image super-resolution and is publicly available for download. The WR-SR dataset was developed by Jiang et al. [22] as a benchmark for reference-based image super-resolution and can be downloaded publicly. The Urban100 dataset was developed by Huang et al. [20] for super-resolution benchmarking and is available for public download. The Manga109 dataset was developed by Matsui et al. [36] for super-resolution benchmarking and is available for open download.

References

  1. Ahmad W et al (2022) A new generative adversarial network for medical images super resolution. Sci Rep 12(1):9533

    Article  MathSciNet  Google Scholar 

  2. Yang C et al (2022) Review of image super-resolution reconstruction algorithms based on deep learning. J Front Comput Sci Technol 16(9):1990

    Google Scholar 

  3. Cao J et al (2022) Reference-based image super-resolution with deformable attention transformer. European Conference on Computer Vision. Springer, Berlin, pp 325–342

    Google Scholar 

  4. Chiche BN et al (2022) Stable long-term recurrent video superresolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 837–846

  5. Conde MV et al (2022) Swin2SR: Swinv2 transformer for compressed image super-resolution and restoration. European Conference on Computer Vision. Springer, Berlin, pp 669–687

    Google Scholar 

  6. Cornebise J, Orsolić I, Kalaitzis F (2022) Open high-resolution satellite imagery: the worldstrat dataset-with application to super-resolution. Adv Neural Inf Process Syst 35:25979–25991

    Google Scholar 

  7. Cristani M et al (2004) Distilling information with super-resolution for video surveillance. In: Proceedings of the ACM 2nd International Workshop on Video Surveillance and Sensor Networks. pp 2–11

  8. Dai J et al (2017) Deformable convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp 764–773

  9. Demiray BZ, Sit M, Demir I (2021) D-SRGAN: DEM superresolution with generative adversarial networks. SN Comput Sci 2:1–11

    Article  Google Scholar 

  10. Deng J et al (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 248–255

  11. Dong C et al (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  MathSciNet  Google Scholar 

  12. Dou H et al (2020) PCA-SRGAN: incremental orthogonal projection discrimination for face super-resolution. In: Proceedings of the 28th ACM International Conference on Multimedia, pp 1891–1899

  13. Feng C-M et al (2021) Task transformer network for joint MRI reconstruction and super-resolution. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VI 24. Springer, pp 307– 317

  14. Georgescu M-I et al (2023) Multimodal multi-head convolutional attention with various kernel sizes for medical image super-resolution. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp 2195–2205

  15. Greenspan H (2009) Super-resolution in medical imaging. Comput J 52(1):43–63

    Article  Google Scholar 

  16. Gulrajani I et al (2017) Improved training of wasserstein gans. Adv Neural Inf Process Syst 30

  17. Haris M, Shakhnarovich G, Ukita N (2018) Deep backprojection networks for super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1664–1673

  18. He K et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778

  19. Huang H et al (2021) Lightweight image super-resolution with hierarchical and differentiable neural architecture search. arXiv preprint arXiv:2105.03939

  20. Huang J-B, Singh A, Ahuja N (2015) Single image superresolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 5197–5206

  21. Ignatov A et al (2017) Dslr-quality photos on mobile devices with deep convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3277–3285

  22. Jiang Y et al (2021) Robust reference-based super-resolution via c2-matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2103–2112

  23. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for realtime style transfer and super-resolution. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. Springer, pp 694–711

  24. Kim J, Lee JK, Lee KM (2016) Accurate image superresolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1646–1654

  25. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1637–1645

  26. Lai W-S et al (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 624–632

  27. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  28. Ledig C et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4681–4690

  29. Lian S, Zhou H, Sun Y (2019) FG-SRGAN: a feature-guided superresolution generative adversarial network for unpaired image super-resolution. In: Advances in Neural Networks–ISNN 2019: 16th International Symposium on Neural Networks, ISNN 2019, Moscow, Russia, July 10–12, 2019, Proceedings, Part I 16. Springer, pp 151–161

  30. Liang J et al (2021) Swinir: image restoration using swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 1833–1844

  31. Lim B et al (2017) Enhanced deep residual networks for single image superresolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 136–144

  32. Liu C et al (2022) Learning trajectory-aware transformer for video superresolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5687–5696

  33. Liu J, Tang J, Wu G (2020) Residual feature distillation network for lightweight image super-resolution. In: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer, pp 41–55

  34. Lu L et al (2021) Masa-sr: matching acceleration and spatial adaptation for reference-based image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6368–6377

  35. Lu T et al (2019) Satellite image super-resolution via multi-scale residual deep neural network. Remote Sens 11(13):1588

    Article  Google Scholar 

  36. Matsui Y et al (2017) Sketch-based manga retrieval using manga109 dataset. Multimed Tools Appl 76:21811–21838

    Article  Google Scholar 

  37. Rasti P et al (2016) Convolutional neural network super resolution for face recognition in surveillance monitoring. In: Articulated Motion and Deformable Objects: 9th International Conference, AMDO 2016, Palma de Mallorca, Spain, July 13-15, 2016, Proceedings 9. Springer, pp 175–184

  38. Sajjadi MSM, Scholkopf B, Hirsch M (2017) Enhancenet: single image super-resolution through automated texture synthesis. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4491–4500

  39. Shermeyer J, Van Etten A (2019) The effects of super-resolution on object detection performance in satellite imagery. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops

  40. Shim G, Park J, Kweon IS (2020) Robust reference-based superresolution with similarity-aware deformable convolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 8425–8434

  41. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  42. Sun J, Zongben X, Shum H-Y (2008) Image super-resolution using gradient profile prior. In: IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp 1–8

  43. Vaswani A et al (2017) Attention is all you need. Adv Neural Inf Process Syst 30

  44. Wang L et al (2019) Learning for video super-resolution through HR optical flow estimation. In: Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part I 14. Springer, pp 514–529

  45. Wang X et al (2018) Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops

  46. Xia B et al (2022) Coarse-to-fine embedded patchmatch and multi-scale dynamic aggregation for reference-based super-resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol 36, issue 3, pp 2768–2776

  47. Xiong Y et al (2020) Improved SRGAN for remote sensing image superresolution across locations and sensors. Remote Sens 12(8):1263

    Article  Google Scholar 

  48. Yan X et al (2020) Towards content-independent multi-reference super-resolution: Adaptive pattern matching and feature aggregation. In: Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16. Springer, pp 52–68

  49. Yang F et al (2020) Learning texture transformer network for image superresolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5791–5800

  50. Yang J, Lin Z, Cohen S (2013) Fast image super-resolution based on in-place example regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1059–1066

  51. Image JY et al (2008) Super-resolution as sparse representation of raw image patches. In: IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp 1–8

  52. Zhang L et al (2010) A super-resolution reconstruction algorithm for surveillance images. Signal Process 90(3):848–859

    Article  Google Scholar 

  53. Zhang W et al (2019) Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 3096–3105

  54. Zhang Z et al (2019) Image super-resolution by neural texture transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7982–7991

  55. Zhao M et al (2018) Cross-scale reference-based light field super-resolution. IEEE Trans Comput Imaging 4(3):406–418

    Article  Google Scholar 

  56. Zheng H et al (2018) Crossnet: an end-to-end reference-based super resolution network using cross-scale warping. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 88–104

  57. Zheng H et al (2017) Learning cross-scale correspondence and patch-based synthesis for reference-based super-resolution. In: BMVC, vol 1, p 2

Download references

Acknowledgements

This work is supported by the Sichuan Science and Tech nology program (Grant Nos. 2024YFG0001, 2023ZHCG0018, 2023NSFSC0470, 2020JDTD0020, 2022YFG0026, 2021YFG0018, 2023NSFSC0469) and partially supported by Opening Foundation of Agile and Intelligent Computing Key Laboratory of Sichuan Province and CUIT Science and Technology Innovation Capacity Enhancement Program Project under Grant KYTD202330.

Author information

Author notes

  1. Mingqing Wang, Zhipeng Yang, Xiaojie Li and Xi Wu have contributed equally to this work.

Authors and Affiliations

  1. College of Computer Science & Technology, Chengdu University of Information Technology, Chengdu, China

    Junkun Du, Zhipeng Yang, Xiaojie Li & Xi Wu

  2. Ministry of Education Key Laboratory for Earth System Modeling, Department of Earth System Science, Tsinghua University, Beijing, China

    Mingqing Wang

  3. College of Integrated Health Sciences, AI Plus Institute, University at Albany, State University of New York (SUNY), Albany, NY, USA

    Xin Wang

Authors
  1. Junkun Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingqing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhipeng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaojie Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xi Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J.D. (Junkun Du) designed the research methods, conducted experiments, and performed data analysis, providing the primary conceptual framework and experimental support for the entire study. M.W. (Mingqing Wang) assisted in experimental design, collected and processed experimental data, and made contributions to the creation and organization of figures and tables. Xi.W. (Xi Wu) provided critical research background and related work and revised and supplemented the discussion and conclusions sections of the paper. Z.Y. (Zhipeng Yang) assisted in experimental design and provided research suggestions, contributing to the literature review section. X.L. (Xiaojie Li), as one of the corresponding authors, was responsible for writing the main sections of the paper, including the introduction, methods, and results and revised and proofread the entire manuscript. X.W. (Xin Wang), as one of the corresponding authors, provided statistical analysis and interpretation of experimental data and made contributions to the discussion section. All authors reviewed and approved the final version of the manuscript.

Corresponding authors

Correspondence to Xiaojie Li or Xi Wu.

Ethics declarations

Conflict of interest

The authors declare that they have no competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical approval

This declaration is not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Du, J., Wang, M., Wang, X. et al. Reference-based image super-resolution with attention extraction and pooling of residuals. J Supercomput 81, 240 (2025). https://doi.org/10.1007/s11227-024-06587-8

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-024-06587-8

Keywords

Search

Navigation