A Video Frame Resolution and Frame Rate Amplification Method with Optical Flow Method and ESPCN Model

Super-resolution reconstruction for video sequences includes two expansions: the video frame resolution and frame rate, which enhances the resolution of the video frame while increasing the frame rate (frame expansion). Three problems of superresolution to solve include: a) how to use adjacent frames to achieve high quality reconstruction; b) how to generate motion compensation frames to supplement the video; and c) how to improve the efficiency of reconstruction calculation and control the running time. Although many methods have been successfully applied to video super-resolution, these methods still face great challenges in balancing motion compensation accuracy, computational complexity, reconstruction quality and running time. In this paper, an optical flow approach combined with an efficient sub-pixel convolutional neural network (ESPCN) model is proposed for frame resolution and frame rate amplification. By adopting the technical strategy of hyper-splitting before frame insertion (enhancement before frame expansion), the motion compensation frame is generated through the image optical flow to improve the frame rate. The super resolution of video frame is realized by constructing the model combining the motion estimation (ME) between adjacent frames with ESPCN (named as ME+ESPCN). The strategy of first superresolution reconstruction and then inserting frame (enhancement before frame expansion) is adopted to generate motion compensation frame through image optical flow to improve the frame rate. The simulation results show that compared with Sparse Dictionary Learning (SDL) and Super-Resolution Convolutional Neural Network (SRCNN), the proposed method based on ME+ESPCN model accelerates the reconstruction operation significantly, takes about 18ms on average, has higher real-time performance, and the average PSNR value for evaluating the frame restoration quality is about 0.12dB higher than ESPCN (no motion estimation). In addition, compared with the SRCNN method, the PSNR of the amplification technology strategy designed in this paper is improved by about 0.32dB on average.