Naturalness Preserved Image Aesthetic Enhancement with Perceptual Encoder Constraint
Authors: 
Leida Li, 
Yuzhe Yang, Hancheng ZhuAuthors Info & Claims
Pages 364 - 372
Abstract
Typical supervised image enhancement pipeline is to minimize the distance between the enhanced image and the reference one. Pixel-wise and perceptual-wise loss functions could help to improve the general image quality, however are not very efficient in improving the image aesthetic quality. In this paper, we propose a novel Residual connected Dilated U-Net (RDU-Net) for improving the image aesthetic quality. By using different dilation rates, the RDU-Net can extract multiple receptive-field features and merge the maximum information from local to global, which are highly desired in image enhancement. Also, we propose an encoder constraint perceptual loss, which could teach the enhancement network to dig out the latent aesthetic factors and make the enhanced image more natural and aesthetically appealing. The proposed approach can alleviate the over-enhancement phenomenons. The experimental results show that the proposed perceptual loss function could give a steady back propagation and the proposed method outperforms the state-of-the-arts.
References
[1]
Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Fredo Durand. 2011. Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs. In The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition.
[2]
Qifeng Chen, Susan Jia Xu, and Vladlen Koltun. 2017. Fast Image Processing with Fully-Convolutional Networks. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 2516--2525.
[3]
Yunjin Chen and Thomas Pock. 2017. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE transactions on pattern analysis and machine intelligence 39, 6 (2017), 1256--1272.
[4]
Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z Wang. 2006. Studying aesthetics in photographic images using a computational approach. In European conference on computer vision. Springer, 288--301.
[5]
Etienne de Stoutz, Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, and Luc Van Gool. 2018. Fast Perceptual Image Enhancement. In Proceedings of the European Conference on Computer Vision. 260--275.
[6]
Cheng Deng, Erkun Yang, Tongliang Liu, Wei Liu, Jie Li, and Dacheng Tao. 2019. Unsupervised Semantic-Preserving Adversarial Hashing for Image Search. IEEE Transactions on Image Processing (2019).
[7]
Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2018. Aesthetic driven image enhancement by adversarial learning. In 2018 ACM Multimedia Conference on Multimedia Conference. ACM, 870--878.
[8]
Ke Gu, Guangtao Zhai, Xiaokang Yang, Wenjun Zhang, and Chang Wen Chen. 2015. Automatic Contrast Enhancement Technology with Saliency Preservation. IEEE Transactions on Circuits and Systems for Video Technology 25, 9 (2015), 1480--1494.
[9]
Yuchen Guo, Guiguang Ding, and Jungong Han. 2018. Robust quantization for general similarity search. IEEE Transactions on Image Processing 27, 2 (2018), 949--963.
[10]
Kaiming He, Jian Sun, and Xiaoou Tang. 2011. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 12 (2011), 2341--2353.
[11]
Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool. 2018. WESPE: weakly supervised photo enhancer for digital cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 691--700.
[12]
Xin Jin, Jingying Chi, Siwei Peng, Yulu Tian, Chaochen Ye, and Xiaodong Li. 2016. Deep image aesthetics classification using inception modules and fine-tuning connected layer. In 2016 8th International Conference on Wireless Communications & Signal Processing (WCSP). IEEE, 1--6.
[13]
Daniel J Jobson, Zia-ur Rahman, and Glenn A Woodell. 1997. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Transactions on Image processing 6, 7 (1997), 965--976.
[14]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision. Springer, 694--711.
[15]
Yueying Kao, Ran He, and Kaiqi Huang. 2017. Deep aesthetic quality assessment with semantic information. IEEE Transactions on Image Processing 26, 3 (2017), 1482--1495.
[16]
Yan Ke, Xiaoou Tang, and Feng Jing. 2006. The design of highlevel features for photo quality assessment. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), Vol. 1. IEEE, 419--426.
[17]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1646--1654.
[18]
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jirí Matas. 2018. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8183--8192.
[19]
Leida Li, Weisi Lin, Xuesong Wang, Gaobo Yang, Khosro Bahrami, and Alex C Kot. 2016. No-reference image blur assessment based on discrete orthogonal moments. IEEE transactions on cybernetics 46, 1 (2016), 39--50.
[20]
Leida Li, Wenhan Xia, Weisi Lin, Yuming Fang, and Shiqi Wang. 2017. No-reference and robust image sharpness evaluation based on multiscale spatial and spectral features. IEEE Transactions on Multimedia 19, 5 (2017), 1030--1040.
[21]
Alice Lucas, Santiago Lopez-Tapiad, Rafael Molinae, and Aggelos K Katsaggelos. 2019. Generative Adversarial Networks and Perceptual Losses for Video Super-Resolution. IEEE Transactions on Image Processing (2019).
[22]
Naila Murray, Luca Marchesotti, and Florent Perronnin. 2012. AVA: A large-scale database for aesthetic visual analysis. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2408--2415.
[23]
Etta D Pisano, Shuquan Zong, Bradley M Hemminger, Marla DeLuca, R Eugene Johnston, Keith Muller, M Patricia Braeuning, and Stephen M Pizer. 1998. Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms. Journal of Digital imaging 11, 4 (1998), 193.
[24]
Xuexin Qu, Xin Wang, Zihan Wang, Lei Wang, and Lingchen Zhang. 2018. Perceptual-DualGAN: Perceptual Losses for Image to Image Translation with Generative Adversarial Nets. In 2018 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--8.
[25]
Zia-ur Rahman, Daniel J Jobson, and Glenn A Woodell. 1996. Multi-scale retinex for color image enhancement. In Proceedings of 3rd IEEE International Conference on Image Processing, Vol. 3. IEEE, 1003--1006.
[26]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. Unet: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.
[27]
Hossein Talebi and Peyman Milanfar. 2018. Learned perceptual image enhancement. In 2018 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--13.
[28]
Hossein Talebi and Peyman Milanfar. 2018. NIMA: Neural Image Assessment. IEEE Transactions on Image Processing 27 (2018), 3998--4011.
[29]
Yeqing Wang, Yi Li, and Fatih Porikli. 2016. Finetuning Convolutional Neural Networks for visual aesthetics. In 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, 3554--3559.
[30]
Zhicheng Yan, Hao Zhang, Baoyuan Wang, Sylvain Paris, and Yizhou Yu. 2016. Automatic photo adjustment using deep neural networks. ACM Transactions on Graphics (TOG) 35, 2 (2016), 11.
[31]
Fisher Yu and Vladlen Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. CoRR abs/1511.07122 (2016).
[32]
Chao Zhang, Ce Zhu, Xun Xu, Yipeng Liu, Jimin Xiao, and Tammam Tillo. 2018. Visual aesthetic understanding: sample-specific aesthetic classification and deep activation map visualization. Signal Processing: Image Communication 67 (2018), 12--21.
[33]
Richard Zhang, Phillip Isola, and Alexei A Efros. 2016. Colorful Image Colorization. In ECCV.
[34]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 586--595.
[35]
Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. 2017. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging 3, 1 (2017), 47--57.
[36]
Sicheng Zhao, Guiguang Ding, Yue Gao, and Jungong Han. 2017. Approximating discrete probability distribution of image emotions by multi-modal features fusion. Transfer 1000, 1 (2017), 4669-- 4675.
[37]
Sicheng Zhao, Guiguang Ding, Qingming Huang, Tat-Seng Chua, Björn W Schuller, and Kurt Keutzer. 2018. Affective Image Content Analysis: A Comprehensive Survey. In IJCAI. 5534-- 5541.
[38]
Sicheng Zhao, Hongxun Yao, Yue Gao, Guiguang Ding, and Tat-Seng Chua. 2016. Predicting personalized image emotion perceptions in social networks. IEEE Transactions on Affective Computing (2016).
[39]
Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, and Guiguang Ding. 2017. Continuous probability distribution prediction of image emotions via multitask shared sparse regression. IEEE Transactions on Multimedia 19, 3 (2017), 632--645.
Index Terms
- Naturalness Preserved Image Aesthetic Enhancement with Perceptual Encoder Constraint
Recommendations
Image enhancement with naturalness preservation
AbstractThere exist many techniques in the literature to enhance poorly illuminated images. Most of these techniques are based on retinex theory and enhance images considering only the reflectance component of the image and ignore illumination component. ...
A locally tuned nonlinear technique for color image enhancement
An innovative technique for the enhancement of digital color images captured under extremely non-uniform lighting conditions is proposed in this paper. The key contributions of this technique are adaptive intensity enhancement, contrast enhancement and ...
Perceptual Image Dehazing Based on Generative Adversarial Learning
Advances in Multimedia Information Processing – PCM 2018AbstractConvolutional Neural Networks (CNN) based single image dehazing methods have recently gained much attention. However, as they heavily rely on synthetic haze images, existing CNN-based dehazing methods have limitations in achieving visually ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2019
427 pages
ISBN:9781450367653
DOI:10.1145/3323873
- General Chairs:
- Abdulmotaleb El Saddik,
- Alberto Del Bimbo,
- Zhongfei Zhang,
- Program Chairs:
- Alexander Hauptmann,
- K. Selcuk Candan,
- Marco Bertini,
- Lexing Xie,
- Xiao-Yong Wei
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 June 2019
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China
- Natural Science Foundation of Jiangsu Province
- Qing Lan Project of Jiangsu Province
- Six Talent Peaks High-Level Talents in Jiangsu Province
Conference
ICMR '19
Sponsor:
Acceptance Rates
Overall Acceptance Rate 254 of 830 submissions, 31%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 278Total Downloads
- Downloads (Last 12 months)9
- Downloads (Last 6 weeks)1
Reflects downloads up to 14 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in