Abstract
The key objective of super-resolution (SR) imaging is to reconstruct a higher-resolution image based on a set of images, acquired from the same scene and denoted as ‘low-resolution’ images, to overcome the limitation and/or ill-posed conditions of the image acquisition process for facilitating better content visualization and scene recognition. In this paper, we provide a comprehensive review of SR image and video reconstruction methods developed in the literature and highlight the future research challenges. The SR image approaches reconstruct a single higher-resolution image from a set of given lower-resolution images, and the SR video approaches reconstruct an image sequence with a higher-resolution from a group of adjacent lower-resolution image frames. Furthermore, several SR applications are discussed to contribute some insightful comments on future SR research directions. Specifically, the SR computations for multi-view images and the SR video computation in the temporal domain are discussed.
Similar content being viewed by others
References
Chaudhuri S.: Super-Resolution Imaging. Kluwer Academic Publishers, Boston (2001)
Katsaggelos A.K., Molina R., Mateos J.: Super Resolution of Images and Video. Morgan & Claypool, San Rafael (2007)
Bannore V.: Iterative-Interpolation Super-Resolution Image Reconstruction: A Computationally Efficient Technique. Springer, Berlin (2009)
Milanfar P.: Super-Resolution Imaging. CRC Press, Boca Raton (2010)
Kang M.G., Chaudhuri S.: Super-resolution image reconstruction. IEEE Signal Process. Mag. 20, 19–20 (2003)
Bose N.K., Chan R.H., Ng M.K.: Special issue on high resolution image reconstruction. Int. J. Imaging Syst. Technol. 14(2–3), 35–145 (2004)
Ng M.K., Chan T., Kang M.G., Milanfar P.: Special issue on super-resolution imaging: analysis, algorithms, and applications. EURASIP J. Appl. Signal Process. 2006, 1–2 (2006)
Ng M.K., Lam E., Tong C.S.: Special issue on superresolution imaging: theory, algorithms and applications. Multidimens. Syst. Signal Process. 18, 55–188 (2007)
Hardie R.C., Schultz R.R., Barner K.E.: Special issue on super-resolution enhancement of digital video. EURASIP J. Appl. Signal Process. 2007, 1–3 (2007)
Katsaggelos A.K., Molina R.: Special issue on super resolution. Comput. J. 52, 1–167 (2008)
Hunt B.R.: Superresolution of images: algorithms, principles, performance. Int. J. Imaging Syst. Technol. 6(4), 297–304 (1995)
Borman, S., Stevenson R.L.: Spatial resolution enhancement of low-resolution image sequences: a comprehensive review with directions for future research. University of Notre Dame, Technical Report (1998)
Borman, S., Stevenson, R.L.: Super-resolution from image sequences—a review. In: Proceedings of the IEEE Midwest Symposium on Circuits and Systems, pp. 374–378. Notre Dame, IN (1998)
Park S.C., Park M.K., Kang M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. 20, 21–36 (2003)
Ng M.K., Bose N.K.: Mathematical analysis of super-resolution methodology. IEEE Signal Process. Mag. 20, 62–74 (2003)
Capel D., Zisserman A.: Computer vision applied to super resolution. IEEE Signal Process. Mag. 20, 75–86 (2003)
Choi E., Choi J., Kang M.G.: Super-resolution approach to overcome physical limitations of imaging sensors: an overview. Int. J. Imaging Syst. Technol. 14, 36–46 (2004)
Farsiu S., Robinson D., Elad M., Milanfar P.: Advances and challenges in super-resolution. Int. J. Imaging Syst. Technol. 14, 47–57 (2004)
Ouwerkerk J.D.: Image super-resolution survey. Image Vis. Comput. 24, 1039–1052 (2006)
Cristobal, G., Gila, E., Sroubek, F., Flusser, J., Miravet, C., Rodriguez, F.B.: Superresolution imaging: a survey of current techniques. In: Proceedings of the SPIE Conference on Advanced Signal Processing Algorithms, Architectures, and Implementations, pp. 70740C.1–70740C.18 (2008)
Tsai R.Y., Huang T.S.: Multiframe image restoration and registration. In: Huang, T.S. (ed.) Advances in Computer Vision and Image Processing, JAI Press Inc., London (1984)
Max B., Wolf E.: Principles of Optics. Cambridge University Press, Cambridge (1997)
Dropkin, H., Ly, C.: Superresolution for scanning antenna. In: Proceedings of the IEEE International Radar Conference, pp. 306–30. (1997)
Caner, G., Tekalp, A.M., Heinzelman, W.: Performance evaluation of super-resolution reconstruction from video. In: Proceedings of the IS&T/SPIE’s Annual Symposium on Electronic Imaging, pp. 177–185. Santa Clara, CA (2003)
Tanaka, M., Okutomi, M.: Theoretical analysis on reconstruction-based super-resolution for an arbitrary PSF. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 947–954. San Diego, CA (2005)
Rajagopalan A.N., Kiran V.P.: Motion-free superresolution and the role of relative blur. Opt. Soc. Am. J. A Opt. Image Sci. 20, 2022–2032 (2003)
Pham, T.Q., van Vliet, L.J., Schutte, K.: Influence of signal-to-noise ratio and point spread function on limits of superresolution. In: Proceedings of the SPIE Conference on Image Processing: Algorithms and Systems IV, pp. 169–180. San Jose, CA (2005)
Wang Z.-Z., Qi F.-H.: Analysis of multiframe super-resolution reconstruction for image anti-aliasing and deblurring. Image Vis. Comput. 23, 393–404 (2005)
Eekeren A.W.M., Schutte K., Oudegeest O.R., Vliet L.J.: Performance evaluation of super-resolution reconstruction methods on real-world data. EURASIP J. Appl. Signal Process. 2007(43953), 1–11 (2007)
Baker S., Kanade T.: Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. 24, 1167–1183 (2002)
Lin Z., Shum H.-Y.: Fundamental limits of reconstruction-based superresolution algorithms under local translation. IEEE Trans. Pattern Anal. Mach. Intell. 26, 83–97 (2004)
Lin, Z.-C., He, J.-F., Tang, X.-O., Tang, C.-K.: Limits of learning-based superresolution algorithms. In: Proceedings of the IEEE International Conference on Computer Vision. Rio de Janeiro, Brazil (2007)
Robinson D., Milanfar P.: Statistical performance analysis of super-resolution. IEEE Trans. Image Process. 15, 1413–1428 (2006)
Chiang M.-C., Boult T.E.: Efficient super-resolution via image warping. Image Vis. Comput. 28, 761–771 (2000)
Patti A., Altunbasak Y.: Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants. IEEE Trans. Image Process. 10, 179–186 (2001)
Lertrattanapanich S., Bose N.K.: High resolution image formation from low resolution frames using Delaunay triangulation. IEEE Trans. Image Process. 11, 1427–1441 (2002)
Wang Z.-Z., Qi F.-H.: On ambiguities in super-resolution modeling. IEEE Signal Process. Lett. 11, 678–681 (2004)
Nguyen N., Milanfar P.: A wavelet-based interpolation-restoration method for superresolution. Circuits Syst. and Signal Process. 19, 321–338 (2000)
Ei-Khamy S.E., Hadhoud M.M., Dessouky M.I., Salam B.M., Ei-Samie F.E.: Regularized super-resolution reconstruction of images using wavelet fusion. Opt. Eng. 44, 097001.1–097001.10 (2005)
Ei-Khamy S.E., Hadhoud M.M., Dessouky M.I., Salam B.M., Ei-Samie F.E.: Wavelet fusion: a tool to break the limits on LMMSE image super-resolution. Int. J. Wavel. Multiresolut. Inf. Process. 4, 105–118 (2006)
Ji, H., Fermuller, C.: Wavelet-based super-resolution reconstruction: theory and algorithm. In: Proceedings of the European Conference on Computer Vision, pp. 295–307. Graz, Austria (2006)
Ji H., Fermuller C.: Robust wavelet-based super-resolution reconstruction: theory and algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 31, 649–660 (2009)
Chappalli M.B., Bose N.K.: Simultaneous noise filtering and super-resolution with second-generation wavelets. IEEE Signal Process. Lett. 12, 772–775 (2005)
Ur H., Gross D.: Improved resolution from subpixel shifted pictures. CVGIP Graph. Models Image Process. 54, 181–186 (1992)
Bose N.K., Ahuja N.A.: Superresolution and noise filtering using moving least squares. IEEE Trans. Image Process. 15, 2239–2248 (2006)
Irani M., Peleg S.: Improving resolution by image registration. CVGIP Graph. Models Image Process. 53, 231–239 (1991)
Patti A.J., Tekalp A.M.: Super resolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Trans. Image Process. 6, 1446–1451 (1997)
Tom, B.C., Katsaggelos, A.K.: Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images. In: Proceedings of the IEEE International Conference on Image Processing, pp. 539–542. Washington, DC (1995)
Hardie R.C., Barnard K.J., Armstrong E.E.: Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Trans. Image Process. 6, 1621–1633 (1997)
Tian J., Ma K.-K.: Stochastic super-resolution image reconstruction. J. Vis. Commun. Image Represent. 21, 232–244 (2010)
Tian, J., Ma, K.-K.: Edge-adaptive super-resolution image reconstruction using a Markov chain Monte Carlo approach. In: Proceedings of the International Conference on Information, Communications and Signal Processing. Singapore (2007)
Schultz R.R., Stevenson R.L.: Extraction of high-resolution frames from video sequences. IEEE Trans. Image Process. 5, 996–1011 (1996)
Kong, D., Han, M., Xu, W., Tao, H., Gong Y.: A conditional random field model for video super-resolution. In: Proceedings of the IEEE International Conference on Pattern Recognition, pp. 619–622. Hong Kong (2006)
Suresh K.V., Rajagopalan A.N.: Robust and computationally efficient superresolution algorithm. Opt. Soc. Am. J. A Opt. Image Sci. 24, 984–992 (2007)
Belekos S., Galatsanos N.P., Katsaggelos A.K.: Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Trans. Image Process. 19, 1451–1464 (2010)
Shen H., Zhang L., Huang B., Li P.: A MAP approach for joint motion estimation, segmentation, and super resolution. IEEE Trans. Image Process. 16, 479–490 (2007)
Humblot F., Mohammad-Djafari A.: Super-resolution using hidden Markov model and Bayesian detection estimation framework. EURASIP J. Appl. Signal Process. 2006(36971), 1–16 (2006)
Mohammad-Djafari A.: Super-resolution: A short review, a new method based on hidden Markov modeling of hr image and future challenges. Comput. J. 52, 126–141 (2008)
Capel, D., Zisserman, A.: Super-resolution enhancement of text image sequences. In: Proceedings of the IEEE International Conference on Pattern Recognition, pp. 600–605. Barcelona, Spain (2000)
Chakrabarti A., Rajagopalan A.N., Chellappa R.: Super-resolution of face images using kernel PCA-based prior. IEEE Trans. Multimed. 9, 888–892 (2007)
Kim, K.I., Franz, M., Scholkopf, B.: Kernel hebbian algorithm for single-frame super-resolution. In: Proceedings of the ECCV Workshop on Statistical Learning in Computer Vision, pp. 135–149. Prague, Czech Republic (2004)
Tipping M.E., Bishop C.M.: Bayesian image super-resolution. In: Becker, S., Thrun, S., Obermeyer, K. (eds) Advances in Neural Information Processing Systems, MIT Press, Cambridge (2002)
Pickup L.C., Capel D., Roberts S.J., Zisserman A.: Bayesian image super-resolution, continued. In: Scholkopf, B., Platt, J., Hoffman, T. (eds) Advances in Neural Information Processing Systems, pp. 1089–1096. MIT Press, Cambridge (2006)
He H., Kondi L.P.: Resolution enhancement of video sequences with simultaneous estimation of the regularization parameter. J. Electron. Imaging 13, 586–596 (2004)
Zibetti M., Bazan F., Mayer J.: Determining the regularization parameters for super-resolution problems. Signal Process. 88, 2890–2901 (2008)
Zibettia M.V.W., Baznb F.S.V., Mayer J.: Estimation of the parameters in regularized simultaneous super-resolution. Pattern Recognit. Lett. 32, 69–78 (2011)
Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of the SIGGRAPH, pp. 327–340. Los Angeles (2001)
Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 275–282. Washington, D.C. (2004)
Freeman W.T., Pasztor E.C., Carmichael O.T.: Learning low-level vision. Int. J. Comput. Vis. 40, 25–47 (2000)
Pickup L.C., Roberts S.J., Zisserman A.: A sampled texture prior for image super-resolution. In: Thrun, S., Saul, L., Scholkopf, B. (eds) Advances in Neural Information Processing Systems, MIT Press, Cambridge (2003)
Datsenko D., Elad M.: Example-based single document image super-resolution: a global MAP approach with outlier rejection. Multidimens. Syst. Signal Process. 18, 103–121 (2007)
Wang J., Zhu S., Gong Y.: Resolution enhancement based on learning the sparse association of image patches. Pattern Recognit. Lett. 31, 1–10 (2010)
Kim K.I., Kwon Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1127–1133 (2010)
Yang J., Wright J., Huang T., Ma Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19, 2861–2873 (2010)
Rhee S., Kang M.G.: Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Opt. Eng. 38, 1348–1356 (1999)
Woods N.A., Galatsanos N.P., Katsaggelos A.K.: Stochastic methods for joint registration, restoration, and interpolation of multiple undersampled images. IEEE Trans. Image Process. 15, 201–213 (2006)
Dempster P., Laird N.M., Rubin D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. B 39(1), 1–38 (1977)
Bertero M., Boccacci P.: Introduction to Inverse Problems in Imaging. IOP Publishing Ltd, Philadelphia (1998)
Li S.Z.: Markov Random Field Modeling in Computer Vision. Springer, New York (1995)
Rue H.: Gaussian Markov Random Fields: Theory and Applications. Chapman & Hall, Boca Raton (2005)
Bose, N.K., Lertrattanapanich, S., Koo, J.: Advances in superresolution using L-curve. In: Proceedings of the IEEE International Symposium on Circuits and Systems, pp. 433–436. Sydney, Australia (2001)
Wu C.F.J.: On the convergence properties of the EM algorithm. Ann. Stat. 11, 95–103 (1983)
Metropolis N., Rosenbluth M.N., Rosenbluth A.W., Teller A.H., Teller E.: Equations of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1092 (1953)
Schultz R.R., Li M., Stevenson R.L.: Subpixel motion estimation for super-resolution image sequence enhancement. J. Vis. Commun. Image Represent. 9, 38–50 (1998)
Protter M., Elad M.: Super resolution with probabilistic motion estimation. IEEE Trans. Image Process. 18, 1899–1904 (2009)
Barreto D., Alvarez L.D., Abad J.: Motion estimation techniques in super-resolution image reconstruction: a performance evaluation. In: Tsvetkov, M., Golev, V., Murtagh, F., Molina, R. (eds) Virtual Observatory. Plate Content Digitalization, Archive Mining and Image Sequence Processing, pp. 254–268. Sofia, Bulgary (2005)
Callico G., Lopez S., Sosa O., Lopez J.F., Sarmiento R.: Analysis of fast block matching motion estimation algorithms for video super-resolution systems. IEEE Trans. Consum. Electron. 54, 1430–1438 (2008)
Baker, S., Kanade, T.: Super resolution optical flow. The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, Technical Report, Oct (1999)
Zhao, W., Sawhney, H.: Is super-resolution with optical flow feasible? In: Proceedings of the European Conference on Computer Vision, pp. 599–613. Copenhagen, Denamrk (2002)
Lin, F., Fookes, C., Chandran, V., Sridharan, S.: Investigation into optical flow super-resolution for surveillance applications. In: Proceedings of the APRS Workshop on Digital Image Computing, pp. 73–78 (2005)
Le H.V., Seetharaman G.: A super-resolution imaging method based on dense subpixel-accurate motion fields. J. VLSI Signal Process. 42, 79–89 (2006)
Fransens R., Strecha C., Gool L.V.: Optical flow based super-resolution: a probabilistic approach. Comput. Vis. Image Underst. 106, 106–115 (2007)
Suresh K.V., Kumar G.M., Rajagopalan A.N.: Superresolution of license plates in real traffic videos. IEEE Trans. Intell. Trans. Syst. 8, 321–331 (2007)
Narayanan B., Hardie R.C., Barner K.E., Shao M.: A computationally efficient super-resolution algorithm for video processing using partition filters. IEEE Trans. Circuits Syst. Video Technol. 17, 621–634 (2007)
Ng M.K., Shen H.-F., Zhang L.-P., Lam E.: A total variation regularization based super-resolution reconstruction algorithm for digital video. EURASIP J. Appl. Signal Process. 2007(74585), 1–16 (2007)
Patanavijit V., Jitapunkul S.: A Lorentzian stochastic estimation for a robust iterative multiframe super-resolution reconstruction with Lorentzian-Tikhonov regularization. EURASIP J. Appl. Signal Process. 2007(34821), 1–21 (2007)
Borman, S., Stevenson, R.L.: Simultaneous multi-frame MAP super-resolution video enhancement using spatio-temporal priors. In: Proceedings of the IEEE International Conference on Image Processing, pp. 469–473. Kobe, Japan (1999)
Zibetti M., Mayer J.: A robust and computationally efficient simultaneous super-resolution scheme for image sequences. IEEE Trans. Circuits Syst. Video Technol. 17, 1288–1300 (2007)
Alvarez, L.D., Molina, R., Katsaggelos, A.K.: Multi-channel reconstruction of video sequences from low-resolution and compressed observations. In: The 8th Iberoamerican Congress on Pattern Recognition, pp. 46–53. Havana, Cuba (2003)
Elad M., Feuer A.: Superresolution restoration of an image sequence: adaptive filtering approach. IEEE Trans. Image Process. 8, 387–395 (1999)
Elad M., Feuer A.: Super-resolution reconstruction of image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 21, 817–834 (1999)
Farsiu S., Elad M., Milanfar P.: Video-to-video dynamic superresolution for grayscale and color sequences. EURASIP J. Appl. Signal Process. 2006(61859), 1–15 (2006)
Costa G.H., Bermudez J.C.M.: Statistical analysis of the LMS algorithm applied to super-resolution image reconstruction. IEEE Trans. Signal Process. 55, 2084–2095 (2007)
Tian J., Ma K.-K.: A state-space super-resolution approach for video reconstruction. Signal Image Video Process. 3, 217–240 (2009)
Bishop, C.M., Blake, A., Marthi, B.: Super-resolution enhancement of video. In: Proceedings of the International Conference on Artificial Intelligence and Statistics. Key West, FL (2003)
Dedeoglu, G., Kanade, T., August, J.: High-zoom video hallucination by exploiting spatio-temporal regularities. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 151–158. Washington, DC (2004)
Kong, D., Han, M., Xu, W., Tao, H., Gong, Y.: Video superresolution with scene-specific priors. In: Proceedings of the British Machine Vision Association. Edinburgh (2006)
Kimura, K., Nagai, T., Nagayoshi, H., Sako, H.: Simultaneous estimation of super-resolved image and 3D information using multiple stereo-pair images. In: Proceedings of the IEEE International Conference on Image Processing, pp. 417–420. (2007)
Bhavsar, A.V., Rajagopalan, A.N.: Resolution enhancement for binocular stereo. In: Proceedings of the International Conference on Pattern Recognition, pp. 1–4. (2008)
Bhavsar A.V., Rajagopalan A.N.: Resolution enhancement in multi-image stereo. IEEE Trans. Pattern Anal. Mach. Intell. 99, 1721–1728 (2010)
Robertson M.A., Stevenson R.L.: Temporal resolution enhancement in compressed video sequences. EURASIP J. Appl. Signal Process. 2001, 230–238 (2001)
Shechtman E., Caspi Y., Irani M.: Space-time super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 27, 531–545 (2005)
Watanabe, K., Iwai, Y., Nagahara, H., Yachnida, M., Suzuki, T.: Video synthesis with high spatio-temporal resolution using motion compensation and image fusion in wavelet domain. In: Proceedings of the Asian Conference on Computer Vision, pp. 480–489. Hyderabad, India (2006)
Maor N., Feuer A., Goodwin G.C.: Compression at the source for digital camcorders. EURASIP J. Image Video Process. 2007(41516), 1–11 (2007)
Ishikawa, H., Geiger, D.: Rethinking the prior model for stereo. In: Proceedings of the European Conference on Computer Vision, pp. 526–537. (2006)
Szeliski, R., Veksler, O., Agarwala, A., Rother, C.: A comparative study of energy minimization methods for Markov random fields. In: Proceedings of the European Conference on Computer Vision, pp. 16–29. (2006)
Krishnamurthy R., Woods J., Moulin P.: Frame interpolation and bidirectional prediction of video using compactly encoded optical flow fields and label fields. IEEE Trans. Circuits Syst. Video Technol. 9, 713–726 (1999)
Zhang Y., Zhao D., Ji X., Wang R., Gao W.: A spatio-temporal auto regressive model for frame rate upconversion. IEEE Trans. Circuits Syst. Video Technol. 19, 1289–1301 (2009)
Zhang Y., Zhao D., Ma S., Wang R., Gao W.: A motion-aligned auto-regressive model for frame rate up conversion. IEEE Trans. Image Process. 19, 1248–1258 (2010)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tian, J., Ma, KK. A survey on super-resolution imaging. SIViP 5, 329–342 (2011). https://doi.org/10.1007/s11760-010-0204-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-010-0204-6