More Web Proxy on the site http://driver.im/

Article

Simple Baselines for Image Restoration

Authors:

Jian SunAuthors Info & Claims

Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII

Pages 17 - 33

https://doi.org/10.1007/978-3-031-20071-7_2

Published: 23 October 2022 Publication History

Abstract

Although there have been significant advances in the field of image restoration recently, the system complexity of the state-of-the-art (SOTA) methods is increasing as well, which may hinder the convenient analysis and comparison of methods. In this paper, we propose a simple baseline that exceeds the SOTA methods and is computationally efficient. To further simplify the baseline, we reveal that the nonlinear activation functions, e.g. Sigmoid, ReLU, GELU, Softmax, etc. are not necessary: they could be replaced by multiplication or removed. Thus, we derive a Nonlinear Activation Free Network, namely NAFNet, from the baseline. SOTA results are achieved on various challenging benchmarks, e.g. 33.69 dB PSNR on GoPro (for image deblurring), exceeding the previous SOTA 0.38 dB with only 8.4% of its computational costs; 40.30 dB PSNR on SIDD (for image denoising), exceeding the previous SOTA 0.28 dB with less than half of its computational costs. The code and the pre-trained models are released at github.com/megvii-research/NAFNet.

References

[1]

Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

[2]

Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)

[3]

Chen, H., et al.: Pre-trained image processing transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12299–12310 (2021)

[4]

Chen, L., Lu, X., Zhang, J., Chu, X., Chen, C.: HINet: half instance normalization network for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 182–192 (2021)

[5]

Cheng, S., Wang, Y., Huang, H., Liu, D., Fan, H., Liu, S.: NBNet: noise basis learning for image denoising with subspace projection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4896–4906 (2021)

[6]

Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4641–4650 (2021)

[7]

Chu, X., Chen, L., Chen, C., Lu, X.: Improving image restoration by revisiting global information aggregation. arXiv preprint arXiv:2112.04491 (2021)

[8]

Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-XL: attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 (2019)

[9]

Dauphin, Y.N., Fan, A., Auli, M., Grangier, D.: Language modeling with gated convolutional networks. In: International Conference on Machine Learning, pp. 933–941. PMLR (2017)

[10]

De S and Smith S Batch normalization biases residual blocks towards the identity function in deep networks Adv. Neural. Inf. Process. Syst. 2020 33 19964-19975

[11]

Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

[12]

Han, Q., et al.: Demystifying local vision transformer: Sparse connectivity, weight sharing, and dynamic weight. arXiv preprint arXiv:2106.04263 (2021)

[13]

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

[14]

Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)

[15]

Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

[16]

Hua, W., Dai, Z., Liu, H., Le, Q.V.: Transformer quality in linear time. arXiv preprint arXiv:2202.10447 (2022)

[17]

Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. PMLR (2015)

[18]

Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

[19]

Liang, J., et al.: VRT: a video restoration transformer. arXiv preprint arXiv:2201.12288 (2022)

[20]

Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: SwinIR: image restoration using swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1833–1844 (2021)

[21]

Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)

[22]

Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. arXiv preprint arXiv:2201.03545 (2022)

[23]

Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)

[24]

Mao, X., Liu, Y., Shen, W., Li, Q., Wang, Y.: Deep residual Fourier transformation for single image deblurring. arXiv preprint arXiv:2111.11745 (2021)

[25]

Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883–3891 (2017)

[26]

Nah, S., Son, S., Lee, S., Timofte, R., Lee, K.M.: NTIRE 2021 challenge on image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 149–165 (2021)

[27]

Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: ICML (2010)

[28]

Ronneberger O, Fischer P, and Brox T Navab N, Hornegger J, Wells WM, and Frangi AF U-Net: convolutional networks for biomedical image segmentation Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 2015 Cham Springer 234-241

[29]

Shazeer, N.: GLU variants improve transformer. arXiv preprint arXiv:2002.05202 (2020)

[30]

Tu, Z., et al.: Maxim: Multi-axis MLP for image processing. arXiv preprint arXiv:2201.02973 (2022)

[31]

Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)

[32]

Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)

[33]

Wang Y, Huang H, Xu Q, Liu J, Liu Y, and Wang J Vedaldi A, Bischof H, Brox T, and Frahm J-M Practical deep raw image denoising on mobile devices Computer Vision – ECCV 2020 2020 Cham Springer 1-16

[34]

Wang, Z., Cun, X., Bao, J., Liu, J.: Uformer: a general U-shaped transformer for image restoration. arXiv preprint arXiv:2106.03106 (2021)

[35]

Waqas Zamir, S., et al.: Multi-stage progressive image restoration. arXiv preprint arXiv:2102.02808 (2021)

[36]

Yan, J., Wan, R., Zhang, X., Zhang, W., Wei, Y., Sun, J.: Towards stabilizing batch statistics in backward propagation of batch normalization. arXiv preprint arXiv:2001.06838 (2020)

[37]

Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: efficient transformer for high-resolution image restoration. arXiv preprint arXiv:2111.09881 (2021)

[38]

Zamir SW et al. Vedaldi A, Bischof H, Brox T, Frahm J-M, et al. Learning enriched features for real image restoration and enhancement Computer Vision – ECCV 2020 2020 Cham Springer 492-511

Cited By

Yue CPeng ZMa JDu SWei PZhang DSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Image restoration through generalized Ornstein-Uhlenbeck bridgeProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694465(58068-58089)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694465
Yu WLi JZhang SJi XSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Learning scale-aware spatio-temporal implicit representation for event-based motion deblurringProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694443(57527-57543)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694443
Agnihotri SJung SKeuper MSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)CosPGDProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692089(416-451)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3692089
Show More Cited By

Index Terms

Simple Baselines for Image Restoration
1. Computing methodologies
2. Mathematics of computing
  1. Mathematical analysis

Index terms have been assigned to the content through auto-classification.

Recommendations

Blurred image restoration: A fast method of finding the motion length and angle

Motion blur in photographic images is a result of camera movement or shake. Methods such as Blind Deconvolution are used when information about the direction and size of blur is not known. Restoration methods, such as Lucy and Richardson or Wiener ...
Image Restoration Using Gaussian Scale Mixtures in Complex Curvelet Transform Domain
ICMTMA '10: Proceedings of the 2010 International Conference on Measuring Technology and Mechatronics Automation - Volume 02

In this paper, a complex Curvelet transform is presented at first. The key innovation can be generalized as follows:2D and 1D complex wavelet transform instead à trous algorithm sub-band decomposition and ID wavelet transform respectively, and increase ...
Multispectral Joint Image Restoration via Optimizing a Scale Map
Color, infrared and flash images captured in different fields can be employed to effectively eliminate noise and other visual artifacts. We propose a two-image restoration framework considering input images from different fields, for example, one noisy ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII

Oct 2022

799 pages

ISBN:978-3-031-20070-0

DOI:10.1007/978-3-031-20071-7

Editors:
Shai Avidan
Tel Aviv University, Tel Aviv, Israel
,
Gabriel Brostow
University College London, London, UK
,
Moustapha Cissé
Google AI, Accra, Ghana
,
Giovanni Maria Farinella
University of Catania, Catania, Italy
,
Tal Hassner
Facebook (United States), Menlo Park, CA, USA

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 23 October 2022

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

65
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yue CPeng ZMa JDu SWei PZhang DSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Image restoration through generalized Ornstein-Uhlenbeck bridgeProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694465(58068-58089)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694465
Yu WLi JZhang SJi XSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Learning scale-aware spatio-temporal implicit representation for event-based motion deblurringProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694443(57527-57543)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694443
Agnihotri SJung SKeuper MSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)CosPGDProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692089(416-451)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3692089
Cui YLiu MRen WKnoll ALarson K(2024)Hybrid frequency modulation network for image restorationProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/80(722-730)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/80
Yan HWang SLarson K(2024)DFMDA-NetProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/169(1525-1533)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/169
Sun YCao BZhu PHu QLarson K(2024)Dynamic brightness adaptation for robust multi-modal image fusionProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/146(1317-1325)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/146
Song TJin GLi PJiang KChen XJin JLarson K(2024)Learning a spiking neural network for efficient image derainingProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/139(1254-1262)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/139
Li HShi HGao XLarson K(2024)A coarse-to-fine fusion network for event-based image deblurringProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/108(974-982)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/108
Wang JMa SBayer KZhang YWang PZhou BNayar SKrishnan G(2024)Perspective-Aligned AR Mirror with Under-Display CameraACM Transactions on Graphics10.1145/368799543:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687995
Wang LZhang LGao FKang YZhang J(2024)NFPLight: Deep SVBRDF Estimation via the Combination of Near and Far Field Point LightingACM Transactions on Graphics10.1145/368797843:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687978
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents