Abstract
In this paper, based on our previous work, we present a multi-scale super-resolution (SR) hardware (HW) architecture using a convolutional neural network (CNN), where the up-scaling factors of 2, 3 and 4 are supported. In our dedicated multi-scale CNN-based SR HW, low-resolution (LR) input frames are processed line-by-line, and the number of convolutional filter parameters is significantly reduced by incorporating depth-wise separable convolutions with residual connections. As for 3× and 4× up-scaling, the number of channels for point-wise convolution layer before a pixel-shuffle layer is set to 9 and 16, respectively. Additionally, we propose an integrated timing generator that supports 3× and 4× up-scaling. For efficient HW implementation, we use a simple and effective quantization method with a minimal peak signal-to-noise ratio (PSNR) degradation. Also, we propose a compression method to efficiently store intermediate feature map data to reduce the number of line memories used in HW. Our CNN-based SR HW implementation on the FPGA can generate 4K ultra high-definition (UHD) frames of higher PSNR at 60 fps, which have higher visual quality compared to conventional CNN-based SR methods that were trained and tested in software. The resources in our CNN-based SR HW can be shared for multi-scale upscaling factors of 2, 3 and 4 so that can be implemented to generate 8K UHD frames from 2K FHD input frames.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Kim, Y., Choi, J., Kim, M.: A real-time convolutional neural network for super-resolution on FPGA with applications to 4 K UHD 60 fps video services. IEEE Trans. Circuits Syst. Video Technol. 29(8), 2521–2534 (2019). https://doi.org/10.1109/TCSVT.2018.2864321
Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016, ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25
Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874–1883. IEEE, Las Vegas, June 2016
Xilinx. https://www.xilinx.com/products/boards-and-kits/kcu105.html. Accessed 19 Sept 2019
TED’s TB-FMCH-HDMI4K Hardware User Manual. https://solutions.inrevium.com/products/pdf/TB_FMCH_HDMI4K_HWUserManual_2.04.pdf. Accessed 19 Sept 2019
Acknowledgement
This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00419, Intelligent High Realistic Visual Processing for Smart Broadcasting Media).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Kim, Y., Choi, JS., Lee, J., Kim, M. (2020). A CNN-Based Multi-scale Super-Resolution Architecture on FPGA for 4K/8K UHD Applications. In: Ro, Y., et al. MultiMedia Modeling. MMM 2020. Lecture Notes in Computer Science(), vol 11962. Springer, Cham. https://doi.org/10.1007/978-3-030-37734-2_63
Download citation
DOI: https://doi.org/10.1007/978-3-030-37734-2_63
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37733-5
Online ISBN: 978-3-030-37734-2
eBook Packages: Computer ScienceComputer Science (R0)