Abstract
Objective image quality assessment (IQA) plays an important role in various visual communication systems, which can automatically and efficiently predict the perceived quality of images. The human eye is the ultimate evaluator for visual experience, thus the modeling of human visual system (HVS) is a core issue for objective IQA and visual experience optimization. The traditional model based on black box fitting has low interpretability and it is difficult to guide the experience optimization effectively, while the model based on physiological simulation is hard to integrate into practical visual communication services due to its high computational complexity. For bridging the gap between signal distortion and visual experience, in this paper, we propose a novel perceptual no-reference (NR) IQA algorithm based on structural computational modeling of HVS. According to the mechanism of the human brain, we divide the visual signal processing into a low-level visual layer, a middle-level visual layer and a high-level visual layer, which conduct pixel information processing, primitive information processing and global image information processing, respectively. The natural scene statistics (NSS) based features, deep features and free-energy based features are extracted from these three layers. The support vector regression (SVR) is employed to aggregate features to the final quality prediction. Extensive experimental comparisons on three widely used benchmark IQA databases (LIVE, CSIQ and TID2013) demonstrate that our proposed metric is highly competitive with or outperforms the state-of-the-art NR IQA measures.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
G. T. Zhai, X. K. Min. Perceptual image quality assessment: A survey. Science China Information Sciences, vol. 63, no. 11, Article number 211301, 2020. DOI: https://doi.org/10.1007/s11432-019-2757-1.
W. S. Lin, C. C. J. Kuo. Perceptual visual quality metrics: A survey. Journal of Visual Communication and Image Representation, vol. 22, no. 4, pp. 297–312, 2011. DOI: https://doi.org/10.1016/j.jvcir.2011.01.005.
Z. Wang, A. C. Bovik. Mean squared error: Love it or leave it? A new look at signal fidelity measures IEEE Signal Processing Magazine, vol. 26, no. 1, pp. 98–117, 2009. DOI: https://doi.org/10.1109/MSP.2008.930649.
Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. DOI: https://doi.org/10.1109/TIP.2003.819861.
H. R. Sheikh, A. C Bovik. Image information and visual quality. IEEE Transactions on Image Processing, vol. 15, no. 2, pp. 430–444, 2006. DOI: https://doi.org/10.1109/TIP.2005.859378.
D. M. Chandler, S. S. Hemami. VSNR: A wavelet-based visual signal-to-noise ratio for natural images. IEEE Transactions on Image Processing, vol. 16, no. 9, pp. 2284–2298, 2007. DOI: https://doi.org/10.1109/TIP.2007.901820.
K. Gu, L. D. Li, H. Lu, X. K. Min, W. S. Lin. A fast reliable image quality predictor by fusing micro- and macrostructures. IEEE Transactions on Industrial Electronics, vol. 64, no. 5, pp. 3903–3912, 2017. DOI: https://doi.org/10.1109/TIE.2017.2652339.
Z. Wang, G. X. Wu, H. R. Sheikh, E. P. Simoncelli, E. H. Yang, A. C. Bovik. Quality-aware images. IEEE Transactions on Image Processing, vol. 15, no. 6, pp. 1680–1689, 2006. DOI: https://doi.org/10.1109/TIP.2005.864165.
M. Liu, K. Gu, G. T. Zhai, P. Le Callet, W. J. Zhang. Perceptual reduced-reference visual quality aseessment for contrast alteration. IEEE Transactions on Broadcasting, vol. 63, no. 1, pp. 71–81, 2017. DOI: https://doi.org/10.1109/TBC.2016.2597545.
R. Soundararajan, A. C. Bovik. RRED indices: Reduced reference entropic differencing for image quality assessment. IEEE Transactions on Image Processing, vol. 21, no. 2, pp. 517–526, 2012. DOI: https://doi.org/10.1109/TIP.2011.2166082.
X. K. Min, K. Gu, G. T. Zhai, M. H. Hu, X. K. Yang. Saliency-induced reduced-reference quality index for natural scene nnd screen content images. Signal Processing, vol. 145, pp. 127–136, 2018. DOI: https://doi.org/10.1016/j.sigpro.2017.10.025.
Y. T. Liu, K. Gu, S. Q. Wang, D. B. Zhao, W. Gao. Blind quality assessment of camera mages based on low-level and high-level statistical features. IEEE Transactions on Multimedia, vol. 21, no. 1, pp. 135–146, 2019. DOI: https://doi.org/10.1109/TMM.2018.2849602.
A. K. Moorthy, A. C. Bovik. Blind image quality assessment: From natural scene statistics to perceptual quality. IEEE Transactions on Image Processing, vol. 20, no. 12, pp. 3350–3364, 2011. DOI: https://doi.org/10.1109/TIP.2011.2147325.
Mittal, R Soundararajan, A. C. Bovik. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013. DOI: https://doi.org/10.1109/LSP.2012.2227726.
X. K. Min, K. Gu, G. T. Zhai, J. Liu, X. K. Yang, C. W. Chen. Blind quality assessment based on pseudo-reference image. IEEE Transactions on Multimedia, vol 20, no. 8, pp. 2049–2062, 2018. DOI: https://doi.org/10.1109/TMM.2017.2788206.
Y. T. Liu, K. Gu, Y. B. Zhang, X. Li, G. T. Zhai, D. B. Zhao, W. Gao. Unsupervised bind image quality evaluation via statistical measurements of structure, naturalness, and perception. IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 4, pp. 929–943, 2020. DOI: https://doi.org/10.1109/TCSVT.2019.2900472.
P. Ye, J. Kumar, L. Kang, D. Doermann. Unsupervised feature learning framework for no-reference image quality assessment. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Providence, USA, pp. 1098–1105, 2012. DOI: https://doi.org/10.1109/CVPR.2012.6247789.
J. T. Xu, P. Ye, Q. H. Li, H. Q. Du, Y. Liu, D. Doermann. Blind image quality assessment based on high order statistics aggregation. IEEE Transactions on Image Processing, vol. 25, no. 9, pp. 4444–4457, 2016. DOI: https://doi.org/10.1109/TIP.2016.2585880.
J. Kim, S. Lee. Fully deep blind image quality predictor. IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 1, pp. 206–220, 2017. DOI: https://doi.org/10.1109/JSTSP.2016.2639328.
G. T. Zhai, X. L. Wu, X. K. Yang, W. S. Lin, W. J. Zhang. A plychovisual quality metric in free-energy principle. IEEE Transactions on Image Processing, vol. 21, no. 1, pp. 41–52, 2012. DOI: https://doi.org/10.1109/TIP.2011.2161092.
K. Gu, G. T. Zhai, X. K. Yang, W. J. Zhang. Using free energy principle for blind image quality assessment. IEEE Transactions on Multimedia, vol. 17, no. 1, pp. 50–63, 2015. DOI: https://doi.org/10.1109/TMM.2014.2373812.
Q. H. Li, W. S. Lin, J. T. Xu, Y. M Fang. Blind image quality assessment using statistical structural and luminance features. IEEE Transactions on Multimedia, vol. 18, no. 12, pp. 2457–2469, 2016. DOI: https://doi.org/10.1109/TMM.2016.2601028.
Q. H. Li, W. S. Lin, Y. M. Fang. BSD: Blind image quality assessment based on structural degradation. Neurocomputing, vol. 236, pp. 93–103, 2017. DOI: https://doi.org/10.1016/j.neucom.2016.09.105.
A. Saha, Q. M. J. Wu. Utilizing image scales towards totally training free blind image quality assessment. IEEE Transactions on Image Processing, vol. 24, no. 6, pp. 1879–1892, 2015. DOI: https://doi.org/10.1109/TIP.2015.2411436.
N. Dijkstra, P. Zeidman, S. Ondobaka, M. A. J. Van Gerven, K. Friston. Distinct top-down and bottom-up brain connectivity during visual perception and imagery. Scientific Reports, vol. 7, no. 1, Article number 5677, 2017. DOI: https://doi.org/10.1038/s41598-017-05888-8.
S. M. Kosslyn, G. Ganis, W. L. Thompson. Neural foundations of imagery. Nature Reviews Neuroscience, vol. 2, no. 9, pp. 635–642, 2001. DOI: https://doi.org/10.1038/35090055.
F. Katsuki, C. Constantinidis. Bottom-up and top-down attention: Different processes and overlapping neural systems. The Neuroscientist, vol. 20, no. 5, pp. 509–521, 2014. DOI: https://doi.org/10.1177/1073858413514136.
H. J. Park, K. Friston. Structural and functional bram networks: From connections to cognition. Science, vol. 342, no. 6158, Article number 1238411, 2013. DOI: https://doi.org/10.1126/science.1238411.
K. E. Stephan. On the role of general system theory for functional neuroimaging. Journal of Anatomy, vol. 205, no. 6, pp. 443–470, 2004. DOI: https://doi.org/10.1111/j.0021-8782.2004.00359.x.
H. R. Sheikh, Z. Wang, L. Cormack, A. C. Bovik. Live image quality assessment database release 2, [Online], Available: http://live.ece.utexas.edu/research/quality/, 2020.
E. C Larson, D. M. Chandler. Most apparent distortion: Full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging, vol. 19, no. 1, Article number 011006, 2010. DOI: https://doi.org/10.1117/1.3267105.
N. Ponomarenko, L. N. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti, C. C. J. Kuo. Image database TID2013: Peculiarities, results and perspectives. Signal Processing: Image Communication, vol. 30, pp. 57–77, 2015. DOI: https://doi.org/10.1016/j.image.2014.10.009.
A. Mittal, A. K. Moorthy, A. C. Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing, vol. 21, no. 12, pp. 4695–4708, 2012. DOI: https://doi.org/10.1109/TIP.2012.2214050.
D. L. Ruderman. The statistics of natural images. Network Computation in Neural Systems, vol. 5, no. 4, pp. 517–548, 1994. DOI: https://doi.org/10.1088/0954-898X_5_4_006.
K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 770–778, 2016. DOI: https://doi.org/10.1109/CVPR.2016.90.
M. El Mallahi, A. Zouhri, H. Qjidaa. Radial meixner moment invariants for 2D and 3D image recognition. Pattern Recognition and Image Analysis, vol. 28, no. 2, pp. 207–216, 2018. DOI: https://doi.org/10.1134/S1054661818020128.
T. Y. Lin, P. Dollár, R. Girshick, K. M. He, B. Hariharan, S. Belongie. Feature pyramid networks for object detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 936–944, 2017. DOI: https://doi.org/10.1109/CVPR.2017.106.
X. Y. Gong, H. Su, D. Xu, Z. T. Zhang, F. Shen, H. B. Yang. An overview of contour detection approaches. International Journal of Automation and Computing, vol. 15, no. 6, pp. 656–672, 2018. DOI: https://doi.org/10.1007/s11633-018-1117-z.
C. Ma, J. B. Huang, X. K. Yang, M. H. Yang. Hierarchical convolutional features for visual tracking. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sant ago, Chile, pp. 3074–3082, 2015. DOI: https://doi.org/10.1109/ICCV.2015.352.
C. Ma, X. K. Yang, C. Y. Zhang, M. H. Yang. Long-term correlation tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Boston, USA, pp. 5388–5396, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7299177.
X. K. Min, G. T. Zhai, K. Gu, Y. T. Liu, X. K. Yang. Blind image quality estimation via distortion aggravation. IEEE Transactions on Broadcasting, vol. 64, no. 2, pp. 508–517, 2018. DOI: https://doi.org/10.1109/TBC.2018.2816783.
K. Y. Ding, K. D. Ma, S. Q. Wang, E. P. Simoncelli. Image quality assessment: Unifying structure and texture similarity. [Online], Available: //https://arxiv/abs/2004.07728, 2020.
K. Friston. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, vol. 11, no. 2, pp. 127–138, 2010. DOI: https://doi.org/10.1038/nrn2787.
K. Friston, J. Kilner, L. Harrison. A free energy principle for the brain. Journal of Physiology-Paris, vol. 100, no. 1–3, pp. 70–87, 2006. DOI: https://doi.org/10.1016/j.jphysparis.2006.10.001.
R. P. Feynman. Statistical Mechanics: A Set of Lectures (Advanced Books Classics), Reading, USA: Westview Press, 1998.
B. A. Olshausen, D. J. Field. How close are we to understanding v1? Neural Computation, vol. 17, no. 8, pp. 1665–1699, 2005. DOI: https://doi.org/10.1162/0899766054026639.
Y. T. Liu, G. T. Zhai, X. M. Liu, D. B. Zhao. Perceptual image quality assessment combining free-energy principle and sparse representation. In Proceedings of IEEE International Symposium on Circuits and Systems, IEEE, Montreal, Canada, 2016, pp. 1586–5589. DOI: https://doi.org/10.1109/ISCAS.2016.7538867.
W. H. Zhu, G. T. Zhai, X. K. Min, M. H. Hu, J. Liu, G. D. Guo, X. K. Yang. Multi-channel decomposition in tandem with free-energy principle for reduced-reference image quality assessment. IEEE Transactions on Multimedia, vol. 21, no. 9, pp. 2334–2346, 2019. DOI: https://doi.org/10.1109/TMM.2019.2902484.
B. SchöLkopf, A. J. Smola, R. C. Williamson, P. L. Bartlett. New support vector algorithms. Neural Computation, vol. 12, no. 5, pp. 1207–1245, 2000. DOI: https://doi.org/10.1162/089976600300015565.
C. C. Chang, C. J. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 3, Article number 27, 2011. DOI: https://doi.org/10.1145/1961189.1961199.
H. H. Lin, V. Hosu, D. Saupe. KADID-10K: A large-scale artificially distorted IQA database. In Proceedings of the 11th International Conference on Quality of Multimedia Experience, IEEE, Berlin, Germany, 2019, DOI: https://doi.org/10.1109/QoMEX.2019.8743252.
J. A. Tropp, A. C. Gilbert. Signal recovery from random measurements via orthogonal matching pursuit. IEEE Transactions on Information Theory, vol. 53, no. 12, pp. 4655–4666, 2007. DOI: https://doi.org/10.1109/TIT.2007.909108.
J. Demsar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, vol. 7, pp. 1–30, 2006. DOI: https://doi.org/10.5555/1248547.1248548.
M. A. Saad, A. C. Bovik, C. Charrier. Blind image quality assessment: A natural scene statistics approach in the DCT domain. IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3339–3352, 2012. DOI: https://doi.org/10.1109/TIP.2012.2191563.
W. F. Xue, L. Zhang, X. Q. Mou. Learning without human scores for blind image quality assessment. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Portland, USA, pp. 995–1002, 2013. DOI: https://doi.org/10.1109/CVPR.2013.133.
L. Zhang, L. Zhang, A. C. Bovik. A feature-enriched completely blind image quality evaluator. IEEE Transactions on Image Processing, vol. 24, no. 8, pp. 2579–2591, 2015. DOI: https://doi.org/10.1109/TIP.2015.2426416.
Q. B. Wu, Z. Wang, H. L. Li. A highly efficient method for blind image quality assessment. In Proceedings of IEEE International Conference on Image Processing, IEEE, Quebec City, Canada, pp. 339–343, 2015. DOI: https://doi.org/10.1109/ICIP.2015.7350816.
X. K. Min, G. T. Zhai, K. Gu, X. K. Yang, X. P. Guan. Objective quality evaluation of dehazed images. IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 11, pp. 2879–2892, 2019. DOI: https://doi.org/10.1109/TITS.2018.2868771.
X. K. Min, K. D. Ma, K. Gu, G. T. Zhai, Z. Wang, W. S. Lin. Unified blind quality assessment of compressed natural, graphic, and screen content images. IEEE Transactions on Image Processing, vol. 26, no. 11, pp. 5462–5474, 2017. DOI: https://doi.org/10.1109/TIP.2017.2735192.
VQEG. Final report from the video quality experts group on the validation of objective models of video quality assessment. [Online], Available: https://www.its.bldrdoc.gov/vqeg/projects/frtv-phase-i/frtv-phasi-i.aspx, 2000.
Acknowledgements
This work was supported by National Natural Science Foundation of China (Nos. 61831015 and 61901260), Key Research and Development Program of China (No. 2019YFB1405902).
Author information
Authors and Affiliations
Corresponding author
Additional information
Colored figures are available in the online version at https://link.springer.com/journal/11633
Wen-Han Zhu received the B. Eng. degree in electronic information engineering from Huazhong University of Science and Technology, China in 2015. He is currently a Ph. D. degree candidate in information and communication engineering with Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, China.
His research interests include image quality assessment and image processing.
Wei Sun received the B. Eng. degree in automation from East China University of Science and Technology, China in 2016. He is currently a Ph. D. degree candidate in control science and engineering at Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, China.
His research interests include image quality assessment, perceptual signal processing and mobile video processing.
Xiong-Kuo Min received the B. Eng. degree in electronic and information engineering from Wuhan University, China in 2013, and the Ph. D. degree in information and communication engineering from Shanghai Jiao Tong University, China in 2018. From January 2016 to January 2017, he was a visiting student at Department of Electrical and Computer Engineering, University of Waterloo, Canada. He is currently a post-doctoral fellow with Shanghai Jiao Tong University. He received the Best Student Paper Award at IEEE ICME 2016.
His research interests include visual quality assessment, visual attention modeling and perceptual signal processing.
Guang-Tao Zhai received the B. Eng. degree in information science and engineering and M. Eng. degree in information science and engineering from Shandong University, China in 2001 and 2004, respectively, and the Ph. D. degree in communication and information system from Shanghai Jiao Tong University, China in 2009, where he is currently a research professor with Institute of Image Communication and Information Processing. From 2008 to 2009, he was a visiting student with Department of Electrical and Computer Engineering, McMaster University, Canada, where he was a post-doctoral fellow from 2010 to 2012. From 2012 to 2013, he was a Humboldt Research Fellow with Institute of Multimedia Communication and Signal Processing, Friedrich Alexander University of Erlangen-Nuremberg, Germany. He received the Award of National Excellent Ph.D. Thesis from the Ministry of Education of China in 2012.
His research interests include multimedia signal processing and perceptual signal processing.
Xiao-Kang Yang received the B. Sc. degree in physics from Xiamen University, China in 1994, the M. Sc. degree in physics from Chinese Academy of Sciences, China in 1997, and the Ph. D. degree in pattern recognition and intelligent system from Shanghai Jiao Tong University, China in 2000. He is currently a Distinguished Professor with the School of Electronic Information and Electrical Engineering, and the Deputy Director of the Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, China. From 2000 to 2002, he was a Research Fellow with the Centre for Signal Processing, Nanyang Technological University, Singapore. From 2002 to 2004, he was a Research Scientist with the Institute for Infocomm Research, Singapore. From 2007 to 2008, he visited Institute for Computer Science, University of Freiburg, Germany, as an Alexander von Humboldt Research Fellow. He has published over 200 refereed papers, and has filed 60 patents. He is an Associate Editor of IEEE Transactions on Multimedia and a Senior Associate Editor of IEEE Signal Processing Letters. He was a Series Editor of Springer CCIS, and an Editorial Board Member of Digital Signal Processing. He is a member of Asia-Pacific Signal and Information Processing Association, the VSPC Technical Committee of the IEEE Circuits and Systems Society, and the MMSP Technical Committee of the IEEE Signal Processing Society. He is also Chair of the Multimedia Big Data Interest Group of MMTC Technical Committee, IEEE Communication Society.
His research interests include image processing and communication, computer vision, and machine learning.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.3.
About this article
Cite this article
Zhu, WH., Sun, W., Min, XK. et al. Structured Computational Modeling of Human Visual System for No-reference Image Quality Assessment. Int. J. Autom. Comput. 18, 204–218 (2021). https://doi.org/10.1007/s11633-020-1270-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11633-020-1270-z