CN104200498A

CN104200498A - Real-time video super-resolution processing method integrated with cortex-A7

Info

Publication number: CN104200498A
Application number: CN201410406695.3A
Authority: CN
Inventors: 苏秉华; 唐佳林; 庄广利
Original assignee: Beijing Institute of Technology BIT
Current assignee: Beijing Institute of Technology BIT
Priority date: 2014-08-18
Filing date: 2014-08-18
Publication date: 2014-12-10
Anticipated expiration: 2034-08-18
Also published as: CN104200498B

Abstract

The invention discloses a real-time video super-resolution processing method fused with Cortex-A7, comprising: A, performing video sampling, obtaining low-resolution video frames and inputting them into the SOC system on chip; B, sequentially performing low-resolution video frames Perform complexity processing, feature vector extraction, and sample set training to obtain feature vectors that need to be matched. The sample set is constructed using high-resolution high-frequency components; The super-resolution algorithm based on clustering dictionary self-learning and feature sparse representation combined with SOC system-on-chip codec technology performs super-resolution processing on low-resolution video frames, thereby outputting high-resolution video frame streams. The invention has the advantages of real-time, low distortion rate, fast processing speed, low processing cost and high quality, and can be widely used in the field of video image processing.

Description

Real-time video super-resolution processing method integrated with Cortex-A7

技术领域technical field

本发明涉及视频图像处理领域，尤其是融合Cortex-A7的实时视频超分辨率处理方法。The invention relates to the field of video image processing, in particular to a real-time video super-resolution processing method integrated with Cortex-A7.

背景技术Background technique

目前，对大多数成像设备而言，其获取的图像分辨率还很低，而更换设备需要大量的物力和人力投入。同时在视频图像采集中，由于受成像设备精度或设备与目标的距离、目标的运动及噪声等多种因素的影响，其得到的通常是具有噪声、模糊和分辨率比较低的视频图像，却很难获得一幅理想分辨率图像。有限的图像分辨率会影响到系统的性能，如低分辨率图像会降低系统的识别性能。这往往给目标识别、身份辨认或刑事侦查等工作带来困难，无法满足实际的需求。因此，业内迫切需要研究一种新的超分辨率技术，可以将同一场景下的若干帧低分辨率图像通过信号处理的方法恢复为一帧高分辨率图像，以降低设备的成本。At present, for most imaging devices, the resolution of images acquired is still very low, and replacing devices requires a lot of material and manpower input. At the same time, in the video image acquisition, due to the influence of various factors such as the accuracy of the imaging equipment or the distance between the equipment and the target, the movement of the target, and noise, the obtained video images are usually noisy, blurred, and relatively low-resolution. It is difficult to obtain an ideal resolution image. Limited image resolution will affect the performance of the system, such as low-resolution images will reduce the recognition performance of the system. This often brings difficulties to work such as target recognition, identification or criminal investigation, and cannot meet actual needs. Therefore, there is an urgent need to study a new super-resolution technology in the industry, which can restore several frames of low-resolution images in the same scene to one frame of high-resolution images through signal processing, so as to reduce the cost of equipment.

目前主流的超分辨率算法包括基于插值的算法和基于重建的方法。其中，基于插值的算法，具有较低的算法复杂度，但其没有利用图像的先验信息，导致恢复的图像过平滑。而基于重建的方法受限于低分辨率图像的数量以及错误的配准，适应性较差。利用高斯马尔可夫随机场模型作为图像的先验信息可以改善这种情况。然而在低分辨率图像的数量有限时，基于高斯马尔可夫随机场模型的超分辨率算法容易丢失重要的细节信息，失真率较高。此外，现有的目前的超分辨率算法仍局限在处理单图像上，处理实时视频流的技术尚未成熟。如果仍沿用现有的超分辨率算法来处理实时视频流，会导致其处理速度较慢、处理成本较高和处理质量较低。The current mainstream super-resolution algorithms include interpolation-based algorithms and reconstruction-based methods. Among them, the algorithm based on interpolation has low algorithm complexity, but it does not use the prior information of the image, resulting in the restored image being too smooth. However, reconstruction-based methods are limited by the number of low-resolution images and wrong registrations, making them less adaptable. This situation can be improved by using a Gaussian Markov random field model as the prior information of the image. However, when the number of low-resolution images is limited, the super-resolution algorithm based on the Gauss Markov random field model tends to lose important details and has a high distortion rate. In addition, existing current super-resolution algorithms are still limited to processing single images, and the technology for processing real-time video streams is not yet mature. If the existing super-resolution algorithm is still used to process the real-time video stream, it will result in slower processing speed, higher processing cost and lower processing quality.

发明内容Contents of the invention

为了解决上述技术问题，本发明的目的是：提供一种实时、失真率较低、处理速度较快、处理成本较低和质量较高的，融合Cortex-A7的实时视频超分辨率处理方法。In order to solve the above-mentioned technical problems, the object of the present invention is: provide a real-time, low distortion rate, fast processing speed, low processing cost and high quality, a real-time video super-resolution processing method fused with Cortex-A7.

本发明解决其技术问题所采用的技术方案是：The technical solution adopted by the present invention to solve its technical problems is:

融合Cortex-A7的实时视频超分辨率处理方法，包括：Real-time video super-resolution processing method integrated with Cortex-A7, including:

A、进行视频采样，获取低分辨率视频帧并输入到SOC片上系统中；A. Carry out video sampling, obtain low-resolution video frames and input them into the SOC system on chip;

B、对低分辨率视频帧依次进行复杂度处理、特征向量提取和样本集训练，从而得到需要进行匹配的特征向量，所述样本集采用高分辨率的高频分量构建而成；B. Perform complexity processing, feature vector extraction, and sample set training on the low-resolution video frame in sequence, so as to obtain feature vectors that need to be matched, and the sample set is constructed using high-resolution high-frequency components;

C、根据需要进行匹配的特征向量，采用改进的基于聚类字典自学习及特征稀疏表示的超分辨率算法并结合SOC片上系统的编解码技术，对低分辨率视频帧进行超分辨率处理，从而输出高分辨率视频帧流。C. According to the feature vectors that need to be matched, the improved super-resolution algorithm based on clustering dictionary self-learning and feature sparse representation is combined with the encoding and decoding technology of the SOC system on chip to perform super-resolution processing on low-resolution video frames. Thus outputting a stream of high-resolution video frames.

进一步，所述步骤C，其包括：Further, said step C, which includes:

C1、构建具备主特征改进的基于聚类字典自学习及特征稀疏表示的超分辨率算法；C1. Construct a super-resolution algorithm based on clustering dictionary self-learning and feature sparse representation with improved main features;

C2、根据需要进行匹配的特征向量和构建的超分辨率算法对低分辨率视频帧进行超分辨率处理，从而输出高分辨率视频帧流。C2. Perform super-resolution processing on the low-resolution video frames according to the required matching feature vectors and the constructed super-resolution algorithm, so as to output a stream of high-resolution video frames.

进一步，所述步骤C1，其包括：Further, the step C1 includes:

C11、建立一个过完备数据库；C11. Establish an over-complete database;

C12、计算输入的低分辨率视频图像块的稀疏表示系数；C12. Calculate the sparse representation coefficient of the input low-resolution video image block;

C13、计算低分辨率字典下的稀疏编码系数和高分辨率字典下的稀疏编码系数；C13. Calculating the sparse coding coefficients under the low-resolution dictionary and the sparse coding coefficients under the high-resolution dictionary;

C14、根据低分辨率视频图像块的稀疏表示系数、稀疏编码系数、过完备数据库中的高分辨率图像库和低分辨率字典重建出高分辨率视频图像块；C14. Reconstructing high-resolution video image blocks according to the sparse representation coefficients, sparse coding coefficients, high-resolution image libraries and low-resolution dictionaries in the over-complete database of low-resolution video image blocks;

C15、采用聚类算法和主成分分析法提取视频图像块集合，然后采用K-SVD算法对高、低分辨率图像块集合进行联合训练；C15, using a clustering algorithm and a principal component analysis method to extract a video image block set, and then using the K-SVD algorithm to jointly train the high and low resolution image block sets;

C16、根据联合训练的结果采用正交匹配追踪法得到具备主特征改进的基于聚类字典自学习及特征稀疏表示的超分辨率算法。C16. According to the results of joint training, the orthogonal matching pursuit method is used to obtain a super-resolution algorithm based on clustering dictionary self-learning and feature sparse representation with improved main features.

进一步，所述步骤C13，其具体为：Further, the step C13 is specifically:

计算低分辨率字典下的稀疏编码系数和高分辨率字典下的稀疏编码系数，所述低分辨率字典下的稀疏编码系数和高分辨率字典下的稀疏编码系数的计算公式分别为：Calculate the sparse coding coefficients under the low-resolution dictionary and the sparse coding coefficients under the high-resolution dictionary, the sparse coding coefficients under the low-resolution dictionary and the sparse coding coefficients under the high-resolution dictionary The calculation formulas are respectively:

${δ δ}_{L L}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11},,$

${δ δ}_{H h}^{ω ω} = = arg arg min min {| | | | {N N}_{H h}^{ω ω} - - {K K}_{H h} {δ δ}_{H h}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{H h}^{ω ω}} | | | |}_{11},,$

其中，K_L为低分辨率字典函数，表示低分辨率图像中的视频流， ρ为矩阵范数的给定参数，表示经过推导后理想的低分辨率字典函数约束项，ε是经过特征处理后的复杂正则化参数，L为代替稀疏编码的范数；Among them, K _L is a low-resolution dictionary function, Represents a video stream in a low-resolution image, ρ is a given parameter of the matrix norm, Indicates the ideal low-resolution dictionary function constraint item after derivation, ε is the complex regularization parameter after feature processing, and L is the norm instead of sparse coding;

K_H为高分辨率字典函数，表示高分辨率图像中的视频流，表示经过推导后理想的高分辨率字典函数约束项，H为代替稀疏编码的范数。K _H is a high-resolution dictionary function, Represents a video stream in a high-resolution image, Denotes the ideal high-resolution dictionary function constraint item after derivation, and H is the norm instead of sparse coding.

进一步，所述步骤C15，其具体为：Further, the step C15 is specifically:

采用聚类算法和主成分分析法提取视频图像块集合，然后采用K-SVD算法对高、低分辨率图像块集合进行联合训练，从而得到联合训练的结果数据，所述联合训练的结果数据{K_H,K_L,δ,ω}为：A clustering algorithm and a principal component analysis method are used to extract a video image block set, and then the K-SVD algorithm is used to jointly train the high-resolution and low-resolution image block sets, thereby obtaining the result data of the joint training, and the result data of the joint training { K _H , K _L ,δ,ω} are:

其中，N是高、低分辨率图像块联合起来进行训练后得到的列向量， $N = {[N_{H} \sec (N_{H}) / e^{iω w_{1}}, N_{L} \sec (N_{L}) / e^{iω w_{2}}]}^{T},$ $K = {[K_{H} \sec (K_{H}) / e^{iω w_{1}}, K_{L} \sec (K_{L}) / e^{iω w_{2}}]}^{T}$ 为高低分辨率联合字典，w₁和w₂分别为通过训练后的高低分辨率视频流的列向量的维数，δ＝[δ₁,δ₂,δ₃,δ₄,......,δ₅]为编码系数矩阵， $ω = | | \begin{matrix} ω_{1} & ω_{2} \\ ω_{3} & ω_{4} \end{matrix} | |$ 为解码系数矩阵。Among them, N is the column vector obtained after the high-resolution and low-resolution image blocks are jointly trained, $N = {[N_{h} \sec (N_{h}) / e^{iω w_{1}}, N_{L} \sec (N_{L}) / e^{iω w_{2}}]}^{T},$ $K = {[K_{h} \sec (K_{h}) / e^{iω w_{1}}, K_{L} \sec (K_{L}) / e^{iω w_{2}}]}^{T}$ is the high and low resolution joint dictionary, w ₁ and w ₂ are the dimensions of the column vectors of the high and low resolution video streams after training respectively, δ=[δ ₁ ,δ ₂ ,δ ₃ ,δ ₄ ,..... .,δ ₅ ] is the coding coefficient matrix, $ω = | | \begin{matrix} ω_{1} & ω_{2} \\ ω_{3} & ω_{4} \end{matrix} | |$ is the decoding coefficient matrix.

进一步，所述步骤C2，其包括：Further, the step C2 includes:

C21、将需要进行匹配的特征向量在字典数据库中进行匹配，并判断匹配是否成功，若是，则执行步骤C23，反之，则执行步骤C22；C21. Match the feature vectors that need to be matched in the dictionary database, and judge whether the matching is successful, if so, execute step C23, otherwise, execute step C22;

C22、采用改进的K值迭代算法对低分辨率视频帧进行聚类字典自学习，然后执行步骤C23；C22, using the improved K value iterative algorithm to carry out clustering dictionary self-study on the low-resolution video frame, and then perform step C23;

C23、根据约束项特征数系数，从过完备数据库中找出高速训练字典库，并融合SOC片上系统的视频编解码技术对低分辨率视频帧进行优化集成处理，从而输出高品质高分辨率视频流。C23. According to the feature number coefficient of the constraint item, find out the high-speed training dictionary library from the complete database, and integrate the video coding and decoding technology of the SOC system on chip to optimize and integrate the low-resolution video frame, thereby outputting high-quality and high-resolution video flow.

进一步，所述步骤C22，其包括：Further, the step C22 includes:

C221、对已有的视频帧进行训练样本处理，从而得到初级聚类函数公式，所述初级聚类函数的表达式为：C221. Perform training sample processing on existing video frames, thereby obtaining the primary clustering function formula, the expression of the primary clustering function is:

其中，I为字典类型，C为初级聚类系数，为迭代次数，n为常数系数，为训练样本的参考数据，为聚类差异系数；Among them, I is the dictionary type, C is the primary clustering coefficient, is the number of iterations, n is a constant coefficient, is the reference data for training samples, is the cluster difference coefficient;

C222、从每帧视频图像左上角的第一个像素点开始，每隔一个像素取一个视频图像块，并对所取的视频图像块采用LASSO算法求解初级聚类函数的最优解所述初级聚类函数的最优解的表达式为：C222, starting from the first pixel in the upper left corner of each frame of video image, take a video image block every other pixel, and use the LASSO algorithm to solve the optimal solution of the primary clustering function for the taken video image block The optimal solution of the primary clustering function The expression is:

${δ δ}_{L L 11}^{ω ω} * * &dtri; &dtri; = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} {Π Π}_{L L}^{&Element; &Element;} \underset{00 \leq \leq x x \leq \leq 11}{max max} \overset{~ ~}{X x} {e e}^{- - {x x}^{22}};;$

其中，▽为LASSO算子，为超分辨率最优集成系数，的表达式为：Among them, ▽ is the LASSO operator, is the optimal integration coefficient for super-resolution, The expression is:

$\overset{~ ~}{X x} = = arg arg {min min}_{α α} {Σ Σ}_{i i} {| | | | {R R}^{i i} (({X x}_{H h} - - {X x}_{L L})) - - \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} | | | |}_{66}^{88};;$

C223、对视频图像块进行分类，得到K个聚类，然后从每个聚类中学习出一个子字典，从而得到最优集成系数下的K个子字典。C223. Classify the video image blocks to obtain K clusters, and then learn a sub-dictionary from each cluster to obtain the optimal integration coefficient The following K sub-dictionaries.

进一步，所述步骤C23，其包括：Further, the step C23 includes:

C231、对初级聚类函数的最优解进行特征稀疏编码均值约束处理，从而得到加入约束项特征数系数后的目标函数所述目标函数的表达式为：C231, the optimal solution to the primary clustering function Perform feature sparse coding mean value constraint processing, so as to obtain the objective function after adding the feature number coefficient of the constraint item The objective function The expression is:

$\begin{matrix} {δ δ}_{L L 22}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} \\ + + θ θ {| | | | (({δ δ}_{H h}^{ω ω} - - {δ δ}_{L L}^{ω ω})) {Σ Σ}_{n no = = 11}^{\infty \infty} (({δ δ}_{n no} cos cos \frac{nπω nπω}{{δ δ}_{H h}^{ω ω} - - {δ δ}_{L L}^{ω ω}} + + sin sin \frac{nπω nπω}{{δ δ}_{H h}^{ω ω} - - {δ δ}_{L L}^{ω ω}})) | | | |}_{ϵ ϵ}^{22} \end{matrix},,$

其中，θ为特征常量，且 $θ = \arg \log_{5} | | \sqrt[ω]{δ_{L}^{ω}} + N_{L}^{ω} | |,$ where θ is a characteristic constant, and $θ = \arg \log_{5} | | \sqrt[ω]{δ_{L}^{ω}} + N_{L}^{ω} | |,$

$(δ_{H}^{ω} - δ_{L}^{ω}) Σ_{n = 1}^{\infty} (δ_{n} \cos \frac{nπω}{δ_{H}^{ω} - δ_{L}^{ω}} + \sin \frac{nπω}{δ_{H}^{ω} - δ_{L}^{ω}})$ 为表示和特征向量距离的特征量； $(δ_{h}^{ω} - δ_{L}^{ω}) Σ_{no = 1}^{\infty} (δ_{no} \cos \frac{nπω}{δ_{h}^{ω} - δ_{L}^{ω}} + \sin \frac{nπω}{δ_{h}^{ω} - δ_{L}^{ω}})$ to represent and eigenvalue of eigenvector distance;

C232、计算聚类字典学习与稀疏编码的优化均值所述优化均值的计算公式为；C232. Calculate the optimized mean of clustering dictionary learning and sparse coding The optimized mean The calculation formula is;

${Δ Δ}^{ω ω}_{11}^{T T} = = {Σ Σ}_{(({K K}_{H h},, {K K}_{L L},, δ δ,, ω ω)) &Element; &Element; N N} {δ δ}_{L L}^{ω ω} * * &dtri; &dtri; {{{K K}_{H h},, {K K}_{L L},, δ δ,, ω ω}},,$

C233、对加入约束项特征数系数后的目标函数和优化均值进行合并优化处理，从而得到优化集成目标函数所述优化集成目标函数的表达式为：C233, for the objective function after adding the characteristic number coefficient of the constraint item and optimized mean Combine and optimize to obtain the optimized integrated objective function The optimization ensemble objective function The expression is:

$\begin{matrix} {δ δ}_{L L 44}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} \\ + + \sqrt[T T]{{Σ Σ}_{(({K K}_{H h},, {K K}_{L L},, δ δ,, ω ω)) &Element; &Element; N N} {δ δ}_{L L}^{ω ω} * * &dtri; &dtri; {{{K K}_{H h},, {K K}_{L L},, δ δ,, ω ω}}} \end{matrix};;$

C234、根据优化集成目标函数对低分辨率视频帧进行优化集成处理，从而生成高分辨率视频流并进行输出。C234. Perform optimization integration processing on the low-resolution video frame according to the optimization integration objective function, so as to generate and output a high-resolution video stream.

进一步，所述步骤C232，其包括：Further, the step C232 includes:

S1、采用零均值随机变量对加入约束项特征数系数后的目标函数进行变形，从而得到变形后的目标函数所述变形后的目标函数为：S1. Using a zero-mean random variable pair to the objective function after adding the characteristic number coefficient of the constraint item Transform to obtain the deformed objective function The deformed objective function for:

${δ δ}_{L L 33}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} + + {Δ Δ}^{ω ω},,$

其中，Δ^ω用于表示 $θ {| | (δ_{H}^{ω} - δ_{L}^{ω}) Σ_{n = 1}^{\infty} (δ_{n} \cos \frac{nπω}{δ_{H}^{ω} - δ_{L}^{ω}} + \sin \frac{nπω}{δ_{K}^{ω} - δ_{L}^{ω}}) | |}_{ϵ}^{2};$ Among them, ^Δω is used to represent $θ {| | (δ_{h}^{ω} - δ_{L}^{ω}) Σ_{no = 1}^{\infty} (δ_{no} \cos \frac{nπω}{δ_{h}^{ω} - δ_{L}^{ω}} + \sin \frac{nπω}{δ_{K}^{ω} - δ_{L}^{ω}}) | |}_{ϵ}^{2};$

S2、根据变形后的目标函数计算聚类字典学习与稀疏编码的优化均值 S2. Calculate the optimized mean value of clustering dictionary learning and sparse coding according to the deformed objective function

本发明的有益效果是：通过价廉物美的SOC片上系统来实现实时视频超分辨率处理，处理成本较低；采用改进的基于聚类字典自学习及特征稀疏表示的超分辨率算法进行实时视频超分辨率处理并结合SOC片上系统的编解码技术，解决了处理实时视频流的效果不佳的问题，具有实时、失真率较低、处理速度较快、处理成本较低和质量较高的优点。The beneficial effects of the present invention are: real-time video super-resolution processing is realized through a cheap and high-quality SOC chip system, and the processing cost is low; the improved super-resolution algorithm based on clustering dictionary self-learning and feature sparse representation is used to perform real-time video super-resolution processing. Super-resolution processing combined with SOC system-on-chip codec technology solves the problem of poor processing of real-time video streams, and has the advantages of real-time, low distortion rate, fast processing speed, low processing cost and high quality .

附图说明Description of drawings

下面结合附图和实施例对本发明作进一步说明。The present invention will be further described below in conjunction with drawings and embodiments.

图1为本发明融合Cortex-A7的实时视频超分辨率处理方法的整体流程图；Fig. 1 is the overall flowchart of the real-time video super-resolution processing method of fusion Cortex-A7 of the present invention;

图2为本发明步骤C的流程图；Fig. 2 is the flowchart of step C of the present invention;

图3为本发明步骤C1的流程图；Fig. 3 is the flowchart of step C1 of the present invention;

图4为本发明步骤C2的流程图；Fig. 4 is the flowchart of step C2 of the present invention;

图5为本发明步骤C22的流程图；Fig. 5 is the flowchart of step C22 of the present invention;

图6为本发明步骤C23的流程图；Fig. 6 is the flowchart of step C23 of the present invention;

图7为本发明步骤C232的流程图Fig. 7 is the flowchart of step C232 of the present invention

图8为本发明实施例一中的硬件模块结构图；FIG. 8 is a structural diagram of a hardware module in Embodiment 1 of the present invention;

图9为本发明实施例二的算法流程图。FIG. 9 is an algorithm flow chart of Embodiment 2 of the present invention.

具体实施方式Detailed ways

参照图1，融合Cortex-A7的实时视频超分辨率处理方法，包括：Referring to Figure 1, the real-time video super-resolution processing method integrated with Cortex-A7 includes:

其中，进行复杂度处理和特征向量提取，用于获取低分辨率图像的纹理和几何结构特征。进行复杂度处理的对象为单图像的多个特征或多帧视频图像的特征。Among them, the complexity processing and feature vector extraction are used to obtain the texture and geometric structure features of the low-resolution image. The object of complexity processing is multiple features of a single image or features of multiple frames of video images.

参照图2，进一步作为优选的实施方式，所述步骤C，其包括：With reference to Fig. 2, further as a preferred embodiment, described step C, it comprises:

参照图3，进一步作为优选的实施方式，所述步骤C1，其包括：Referring to Fig. 3, further as a preferred embodiment, the step C1 includes:

其中，过完备数据库，用于存储训练样本集、迭代的高分辨率图像和训练字典等。Among them, the over-complete database is used to store training sample sets, iterative high-resolution images and training dictionaries, etc.

K-SVD算法，是一种经典的字典训练算法，依据误差最小原则，对误差项进行SVD分解，然后选择使误差最小的分解项作为更新的字典原子和对应的原子系数，并经过不断的迭代最终得到优化的解。The K-SVD algorithm is a classic dictionary training algorithm. According to the principle of minimum error, the error item is decomposed by SVD, and then the decomposition item that minimizes the error is selected as the updated dictionary atom and the corresponding atomic coefficient, and after continuous iteration Finally, an optimal solution is obtained.

进一步作为优选的实施方式，所述步骤C13，其具体为：Further as a preferred embodiment, the step C13 is specifically:

其中，arg函数用于求复数的幅角。ρ为矩阵范数的给定参数，根据视频流的实际情况而设定。ε是经过特征处理后的复杂正则化参数，主要用来平衡K_L和之间的存在比例。Among them, the arg function is used to find the argument of a complex number. ρ is a given parameter of the matrix norm, which is set according to the actual situation of the video stream. ε is a complex regularization parameter after feature processing, which is mainly used to balance K _L and ratio between them.

其中，L和H均被用来代替稀疏编码的范数，作为推导公式的辅助函数。Among them, both L and H are used to replace the norm of sparse coding as auxiliary functions for deriving the formula.

进一步作为优选的实施方式，所述步骤C15，其具体为：Further as a preferred embodiment, the step C15 is specifically:

其中，N是高、低分辨率图像块联合起来进行训练后得到的列向量，Among them, N is the column vector obtained after the high-resolution and low-resolution image blocks are jointly trained,

$N N = = {[[{N N}_{H h} sec sec (({N N}_{H h})) / / {e e}^{iω iω {w w}_{11}},, {N N}_{L L} sec sec (({N N}_{L L})) / / {e e}^{iω iω {w w}_{22}}]]}^{T T},,$

$K = {[K_{H} \sec (K_{H}) / e^{iω w_{1}}, K_{L} \sec (K_{L}) / e^{iω w_{2}}]}^{T}$ 为高低分辨率联合字典，w₁和w₂分别为通过训练后的高低分辨率视频流的列向量的维数，δ＝[δ₁,δ₂,δ₃,δ₄,......,δ₅]为编码系数矩阵， $ω = | | \begin{matrix} ω_{1} & ω_{2} \\ ω_{3} & ω_{4} \end{matrix} | |$ 为解码系数矩阵。 $K = {[K_{h} \sec (K_{h}) / e^{iω w_{1}}, K_{L} \sec (K_{L}) / e^{iω w_{2}}]}^{T}$ is the high and low resolution joint dictionary, w ₁ and w ₂ are the dimensions of the column vectors of the high and low resolution video streams after training respectively, δ=[δ ₁ ,δ ₂ ,δ ₃ ,δ ₄ ,..... .,δ ₅ ] is the coding coefficient matrix, $ω = | | \begin{matrix} ω_{1} & ω_{2} \\ ω_{3} & ω_{4} \end{matrix} | |$ is the decoding coefficient matrix.

其中，sec函数为正割三角函数。Wherein, the sec function is a secant trigonometric function.

参照图4，进一步作为优选的实施方式，所述步骤C2，其包括：Referring to Fig. 4, further as a preferred embodiment, the step C2 includes:

参照图5，进一步作为优选的实施方式，所述步骤C22，其包括：Referring to Fig. 5, further as a preferred embodiment, the step C22 includes:

参照图6，进一步作为优选的实施方式，所述步骤C23，其包括：Referring to Fig. 6, further as a preferred embodiment, the step C23 includes:

参照图7，进一步作为优选的实施方式，所述步骤C232，其包括：Referring to Fig. 7, further as a preferred embodiment, the step C232 includes:

下面结合说明书附图和具体实施例对本发明作进一步详细说明。The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

实施例一Embodiment one

本发明为节约硬件成本及人力成本，通过价廉物美的SOC片上系统来实现实时视频超分辨率的功能，以满足企业及相关机构对高分辨率成像的要求。In order to save hardware cost and labor cost, the present invention implements the function of real-time video super-resolution through a cheap and high-quality SOC system on chip, so as to meet the requirements of enterprises and related institutions for high-resolution imaging.

本发明的SOC片上系统由硬件模块及软件模块两部分组成。其中，硬件部分主要组成部分包括：a.基于ARMv7-A架构的高能效处理芯片；b.DDR3处理器及其外部设备(摄像头、鼠标、键盘、显示器、电源)；c.SOC芯片对外接口(包括高速SDRAM数据接口、视频数据输入接口、视频数据输出接口、中断接口、DDR3数据接口)。整个硬件模块结构如图8所示。The SOC system on chip of the present invention is composed of two parts, a hardware module and a software module. Among them, the main components of the hardware part include: a. High-efficiency processing chip based on ARMv7-A architecture; b. DDR3 processor and its external devices (camera, mouse, keyboard, display, power supply); c. SOC chip external interface ( Including high-speed SDRAM data interface, video data input interface, video data output interface, interrupt interface, DDR3 data interface). The entire hardware module structure is shown in Figure 8.

图8中，1为基于SOC的片上系统主控处理器，2为系统的数据缓存芯片同步动态随机存储器，3为双倍速率同步动态随机存储器，4为是SOC主控处理器与SDRAM之间的数据缓存接口，5为是SOC主控处理器与双倍速率同步动态随机存储器之间的数据缓存接口，6为鼠标数据数据输入接口，7为键盘数据控制输入接口，8是为整个系统的控制指令输入输出接口，9是硬盘数据与片上系统的数据交换接口，10为视频图像数据流输入接口，11为SOC片上系统自带的显示系统供显示屏的显示接口，12为连接终端显示屏的电源线以及连接SOC主控开发板的电源线，13为SOC系统、终端显示屏及其它外设的供电电源，14为整个系统的终端显示屏，15为整个系统提供视频采集数据的设备(如普通摄像头、摄像机、工业摄像头和工业摄像机)，16为整个SOC开发系统提供储存视频数据的硬盘，17为数据服务交换处理器，18为整个SOC系统及整个系统工作模式的中央处理器提供控制指令输入的设备(如键盘)，19为整个SOC系统及整个系统工作模式的中央处理器提供控制指令输入的外部设备(如鼠标)。其中，为整个系统提供视频采集数据的设备，其采集的格式为AVI或者MPEG，视频输入接口接收来自SOC芯片的数据流，而数据缓存接口连接高速同步动态随机存储器，控制指令设备输入接口接收来自外部设备鼠标键盘以及其它外问硬件设备的控制指令，SOC片上系统自带的显示系统供显示屏的显示接口为终端显示屏提供显示数据及实时控制时序，整个系统处于稳定工作模式。In Fig. 8, 1 is the main control processor of the system on a chip based on SOC, 2 is the synchronous DRAM of the data cache chip of the system, 3 is the double speed synchronous DRAM, 4 is between the SOC main control processor and SDRAM 5 is the data cache interface between the SOC main control processor and the double-speed synchronous dynamic random access memory, 6 is the mouse data data input interface, 7 is the keyboard data control input interface, 8 is the whole system Control command input and output interface, 9 is the data exchange interface between the hard disk data and the system on chip, 10 is the video image data stream input interface, 11 is the display interface for the display system provided by the SOC system on chip, and 12 is the display interface for connecting the terminal display 13 is the power supply for the SOC system, terminal display and other peripherals, 14 is the terminal display of the entire system, and 15 is the device that provides video acquisition data for the entire system ( Such as ordinary cameras, video cameras, industrial cameras and industrial cameras), 16 provides hard disks for storing video data for the entire SOC development system, 17 provides data service switching processors, and 18 provides control for the entire SOC system and the central processing unit of the entire system working mode The instruction input device (such as a keyboard), 19 provides an external device (such as a mouse) for controlling the instruction input for the central processing unit of the whole SOC system and the whole system working mode. Among them, the equipment that provides video acquisition data for the whole system, its acquisition format is AVI or MPEG, the video input interface receives the data stream from the SOC chip, and the data buffer interface is connected to the high-speed synchronous dynamic random access memory, and the control instruction equipment input interface receives the data stream from the SOC chip. The external device mouse keyboard and other external hardware devices control commands, the display system of the SOC system provides the display interface of the display screen to provide display data and real-time control timing for the terminal display screen, and the whole system is in a stable working mode.

软件模块包括Ubuntu嵌入式系统、OpenCV函数库、超分辨率算法模块和编译器这四大部分。The software module includes four parts: Ubuntu embedded system, OpenCV function library, super-resolution algorithm module and compiler.

正常工作时，软件系统在SOC硬件系统的支撑下，能够独立运行SOC片上系统。其中，Ubuntu嵌入式系统，是一个完整的开发系统，能够为整个系统提供一个稳定且高性能的开发环境。OpenCV函数库，支撑硬件系统及软件系统的一个媒介，它能够通过调用已经并且经过人工训练好的算法，对开发系统进行实时且完美的功能实现。超分辨率算法模块，融合了Cortex-A7的实时视频超分辨率的核心算法技术，在硬件环境及软件环境均搭建好的情况下，通过改进的算法技术，就能够对本发明进行完整的实现。编译器，用于对代码程序进行编写及编译，相当于一个媒介，能够对已经完成并且设计好的项目进行效果实现。When working normally, the software system can run the SOC system on chip independently under the support of the SOC hardware system. Among them, the Ubuntu embedded system is a complete development system that can provide a stable and high-performance development environment for the entire system. The OpenCV function library is a medium that supports hardware systems and software systems. It can realize real-time and perfect functions of the development system by calling algorithms that have been manually trained. The super-resolution algorithm module integrates the core algorithm technology of real-time video super-resolution of Cortex-A7. When the hardware environment and software environment are all built, the present invention can be completely realized through the improved algorithm technology. The compiler is used to write and compile the code program, which is equivalent to a medium that can realize the effect of the completed and designed project.

本发明软件模块实现的过程包括：(1)Ubuntu嵌入式系统调用OpenCV函数库的过程；(2)超分辨率算法模块调用OpenCV函数库的过程；(3)编译器对项目(已经完成的算法会编写成项目的形式)进行编译，形式可操作程序；(4)可操作程序通过Ubuntu系统自带的工具完整实现应用程序的过程；(5)Ubuntu嵌入式系统安装OpenCV函数库的过程或者回归的过程；(6)训练后的算法编译进OpenCV库备用的过程；(7)编译器对算法或者项目进行反复调试或者效果比对的实验过程；(8)为Ubuntu嵌入式系统更新或者调用编译器的过程。The process that the software module of the present invention realizes comprises: (1) the process that the Ubuntu embedded system calls the OpenCV function library; (2) the process that the super-resolution algorithm module calls the OpenCV function library; (3) the compiler to the project (completed algorithm will be written into the form of the project) to compile, and the form can be operated as a program; (4) the process of implementing the operable program through the tools that come with the Ubuntu system; (5) the process of installing the OpenCV function library or returning to the Ubuntu embedded system (6) The process of compiling the trained algorithm into the OpenCV library for backup; (7) The compiler repeatedly debugs the algorithm or the project or compares the experimental process; (8) Updates or calls the compiler for the Ubuntu embedded system device process.

实施例二Embodiment two

本发明融合Cortex-A7的实时视频超分辨率处理方法，把传统超分辨率处理技术、传统视频编码解码技术及传统的图像编码技术紧密结合起来，研究并且提出一种基于软硬件平台的新的改进算法。在融合Cortex-A7的实时视频超分辨率的改进算法编码算法中，保证其能够在硬件平台SOC片上系统顺畅运行且效果最佳，关键在于视频编解码技术及其处理视频图像算法技术的客观质量。因为只有在相同的解码图像峰值信噪比(Peak signal to noise ratio,PSNR)前提下比特率降低了，其编解码算法才具有说服力。而影响解码视频图像客观质量重要因素之一是超分辨率处理之后的视频图像的理论质量。本发明改进后的算法在解码端对相关参数和视频采样图像序列进行数次重复的超分辨率算法实验，经过各种复杂环境的实验条件下得到最终解码视频图像。将本发明融合Cortex-A7的实时视频超分辨率的改进算法融合到视频编码标准MPEG-2中，测试结果表明，在相同的PSNR比较下，其比特率有一定程度的降低，改进后的算法具备非常强的实用性。另外，由于改进后的算法中使用了全新的通过SOC片上系统的视频编解码以及结合经过改进的聚类字典自学习及特征稀疏表示的超分辨率算法技术，该改进后的超分辨率算法能够融合到H.264/AVC标准框架中的编码算法中。该改进后的算法在编码端对原始视频进行下采样并编码，而则在解码端经过改进的聚类字典学习及特征稀疏表示的超分辨率算法技术处理B帧(双向预测内插编码帧)和P帧(前向预测编码帧)。The present invention integrates the real-time video super-resolution processing method of Cortex-A7, closely combines traditional super-resolution processing technology, traditional video coding and decoding technology and traditional image coding technology, researches and proposes a new method based on software and hardware platforms improve algorithm. In the improved algorithm encoding algorithm of real-time video super-resolution integrated with Cortex-A7, to ensure that it can run smoothly and achieve the best effect on the hardware platform SOC system on chip, the key lies in the objective quality of video encoding and decoding technology and its processing video image algorithm technology . Because the codec algorithm is convincing only if the bit rate is reduced under the premise of the same decoded image peak signal-to-noise ratio (PSNR). One of the important factors affecting the objective quality of the decoded video image is the theoretical quality of the video image after super-resolution processing. The improved algorithm of the present invention performs repeated super-resolution algorithm experiments on relevant parameters and video sampling image sequences at the decoding end, and obtains final decoded video images under various experimental conditions in complex environments. The improved algorithm of the real-time video super-resolution of the fusion of Cortex-A7 of the present invention is integrated into the video coding standard MPEG-2, and the test results show that, under the same PSNR comparison, its bit rate has a certain degree of reduction, and the improved algorithm It has very strong practicality. In addition, since the improved algorithm uses a new video codec through the SOC system on a chip and a super-resolution algorithm technology combined with the improved clustering dictionary self-learning and feature sparse representation, the improved super-resolution algorithm can Integrate into the encoding algorithm in the H.264/AVC standard framework. The improved algorithm downsamples and encodes the original video at the encoding end, and processes B frames (bidirectional predictive interpolation encoded frames) through improved clustering dictionary learning and super-resolution algorithm technology of feature sparse representation at the decoding end and P frames (forward predictive coded frames).

实施例三Embodiment three

OpenCV函数库有自带的多种插值函数，同时也自带有基于变分算法的集成函数，但是实验效果并不理想。近期主流的词典学习和稀疏表示的超分辨率技术在OpenCV函数库是没有的。因此，迫于市场经济的需求及技术发展的需要，本发明提出一个改进的新的算法，嵌入到OpenCV函数库，以达到可移植的、实时的、SOC片上系统的实时视频超分辨率处理的需求。The OpenCV function library has a variety of interpolation functions and an integration function based on a variational algorithm, but the experimental results are not ideal. The recent mainstream dictionary learning and sparse representation super-resolution techniques are not available in the OpenCV library. Therefore, under the demands of market economy and the needs of technological development, the present invention proposes an improved new algorithm, which is embedded in the OpenCV function library, to achieve portable, real-time, real-time video super-resolution processing of SOC system-on-chip need.

本实施例对本发明全新的通过SOC片上系统的视频编解码以及结合经过改进的聚类字典学习及特征稀疏表示的超分辨率算法进行详细说明。This embodiment describes in detail the new video encoding and decoding through the SOC system of the present invention and the super-resolution algorithm combined with the improved clustering dictionary learning and feature sparse representation.

经过改进的聚类字典学习及特征稀疏表示的超分辨率算法是基于稀疏表示的超分辨率重建技术和基于字典学习的超分辨率的基础上进行完善后提出的一种新的实用性强的算法。该算法首先建立一个合适的过完备数据库，然后计算出输入的低分辨率视频图像块的稀疏表示系数，接着计算出高分辨率和低分辨率字典，训练的过程将这些系数、高分辨率图像库和低分辨率字典重建出高分辨率视频图像块，利用聚类算法和主成分分析(PCA)提取视频图像块集合中的部分，并采用K-SVD方法对高、低分辨率图像块集合进行联合训练，在超分辨率图像处理过程采取正交匹配追踪OMP的方法得到自学习的改进的聚类字典学习及特征稀疏表示的超分辨率算法，构造出具备主特征改进的聚类字典学习及特征稀疏表示的超分辨率算法。该方法完全确保了高、低分辨率视频图像块表示系数的一致性，同时也降低了重建的复杂度，有利于提高聚类字典学习及特征稀疏表示的自适应性同时缩短了训练的时间。本发明的算法具有良好的性能，超分辨率处理后的结果具有更高的PSNR和MSSIM(平均结构相似度)。The improved clustering dictionary learning and feature sparse representation super-resolution algorithm is a new practical algorithm based on the improvement of sparse representation-based super-resolution reconstruction technology and dictionary learning-based super-resolution. algorithm. The algorithm first establishes a suitable over-complete database, then calculates the sparse representation coefficients of the input low-resolution video image blocks, and then calculates the high-resolution and low-resolution dictionaries. The training process combines these coefficients, high-resolution images Library and low-resolution dictionary to reconstruct high-resolution video image blocks, use clustering algorithm and principal component analysis (PCA) to extract the part of the video image block set, and use K-SVD method to analyze the high-resolution and low-resolution image block sets Carry out joint training, adopt the method of orthogonal matching pursuit OMP in the process of super-resolution image processing to obtain self-learning improved clustering dictionary learning and super-resolution algorithm with sparse representation of features, and construct a clustering dictionary learning with improved main features And the super-resolution algorithm of feature sparse representation. This method fully ensures the consistency of high-resolution and low-resolution video image block representation coefficients, and also reduces the complexity of reconstruction, which is conducive to improving the adaptability of clustering dictionary learning and feature sparse representation, and shortens the training time. The algorithm of the invention has good performance, and the result after super-resolution processing has higher PSNR and MSSIM (mean structure similarity).

本发明中，低分辨字典下经过推导后计算出的稀疏编码系数表示为：In the present invention, the sparse coding coefficient calculated after derivation under the low-resolution dictionary is expressed as:

${δ δ}_{L L}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} - - - - - - ((11))$

经过计算求得低分辨率视频流数据在字典K_L下的稀疏编码系数接着就可以表示为在视频图像超分辨率的情况下，由于高分辨率视频流和低分辨率视频流具备相同的稀疏表示系数，故高分辨率视频图像聚类字典可以引用低分辨率的推导公式，即：The sparse coding coefficients of the low-resolution video stream data under the dictionary K _L are obtained through calculation Then it can be expressed as In the case of video image super-resolution, since the high-resolution video stream and the low-resolution video stream have the same sparse representation coefficients, the high-resolution video image clustering dictionary can refer to the derivation formula of the low-resolution, namely:

${δ δ}_{H h}^{ω ω} = = arg arg min min {| | | | {N N}_{H h}^{ω ω} - - {K K}_{H h} {δ δ}_{H h}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{H h}^{ω ω}} | | | |}_{11} - - - - - - ((22))$

为了让推导公式中的和具备高度的一致性，以便于为以后的工作做更精确的计算，本发明接下来将在公式(1)和(2)的基础上对推导公式进行联合训练，训练结果如下：In order to make the sum in the derivation formula have a high degree of consistency, so as to do more accurate calculations for future work, the present invention will carry out joint training on the derivation formula on the basis of formulas (1) and (2), training The result is as follows:

在本发明中，以上系数公式推导成功后，接下来正式进入经过改进的聚类字典学习及特征稀疏表示的超分辨率算法的核心算法过程。In the present invention, after the above coefficient formulas are successfully deduced, the core algorithm process of the improved clustering dictionary learning and super-resolution algorithm of feature sparse representation is formally entered.

改进的聚类字典学习通过建立数字模型，对已有的视频进行处理，进行成千上万次的样本训练，得到如下式(4)所示的初级聚类系数公式：The improved clustering dictionary learning establishes a digital model, processes existing videos, and performs thousands of sample trainings to obtain the primary clustering coefficient formula shown in the following formula (4):

式(4)中，首要的训练工作是学习出低分辨率聚类子字典IH和IL，接着采集一定数量的视频帧，对这些视频帧进行聚类学习处理，得到的样本集如下式(5)和式(6)所示：In formula (4), the primary training task is to learn the low-resolution clustering sub-dictionary IH and IL, and then collect a certain number of video frames, and perform clustering learning on these video frames. The obtained sample set is as follows (5 ) and formula (6):

${&aleph; &aleph;}_{H h} = = {{{&aleph; &aleph;}_{H h}^{11},, {&aleph; &aleph;}_{H h}^{22},, {&aleph; &aleph;}_{H h}^{33},, {&aleph; &aleph;}_{H h}^{44},, {&aleph; &aleph;}_{H h}^{55},, . . . . . . . . . . . . {&aleph; &aleph;}_{H h}^{i i}}} - - - - - - ((55))$

${&aleph; &aleph;}_{L L} = = {{{&aleph; &aleph;}_{L L}^{11},, {&aleph; &aleph;}_{L L}^{22},, {&aleph; &aleph;}_{L L}^{33},, {&aleph; &aleph;}_{L L}^{44},, {&aleph; &aleph;}_{L L}^{55},, . . . . . . . . . . . . {&aleph; &aleph;}_{L L}^{i i}}} - - - - - - ((66))$

式中，为高分辨率的样本集，为低分辨率的样本集；对于I从每帧视频图像的左上角的第一个像素点开始，每隔一个像素取一个视频图像块。In the formula, For a high-resolution sample set, is a low-resolution sample set; for I, starting from the first pixel in the upper left corner of each frame of video image, a video image block is taken every other pixel.

接着，运用LASSO算法求解最优问题，得到下式(7)：Then, the LASSO algorithm is used to solve the optimal problem, and the following formula (7) is obtained:

${δ δ}_{L L 11}^{ω ω} * * &dtri; &dtri; = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} {Π Π}_{L L}^{&Element; &Element;} \underset{00 \leq \leq x x \leq \leq 11}{max max} \overset{~ ~}{X x} {e e}^{- - {x x}^{22}} - - - - - - ((77))$

其中，为超分辨率最优集成系数。该系数的值如下所示：in, is the optimal integration coefficient for super-resolution. The coefficient The values for are as follows:

$\overset{~ ~}{X x} = = arg arg {min min}_{α α} {Σ Σ}_{i i} {| | | | {R R}^{i i} (({X x}_{H h} - - {X x}_{L L})) - - \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} | | | |}_{66}^{88} - - - - - - ((88))$

为了得到最优化系数情况下的K个子字典，本发明将视频图像块进行分类，得到K个聚类，然后从每个聚类中学习出一个子字典。显然，K个聚类能够表示K个不同的结构模式，从而能够获得感知意义上的聚类分辨函数。In order to get the optimal coefficient In the case of K sub-dictionaries, the present invention classifies video image blocks to obtain K clusters, and then learns a sub-dictionary from each cluster. Obviously, K clusters can represent K different structural patterns, so that a perceptual cluster discrimination function can be obtained.

本发明采用了改进的K值聚类字典学习解决噪声问题，引入了K值后自然要引入特征稀疏表示的超分辨率算法。通过特征稀疏表编码均值约束来提高算法的质量。其中，式(1)始终保证了无限接近于但是无法保证能够无条件的无限接近于通过实验条件及实际应用，可以证明两者之前存在着特征稀疏编码噪声 The present invention adopts the improved K-value clustering dictionary learning to solve the noise problem, and after introducing the K-value, it is natural to introduce a super-resolution algorithm of feature sparse representation. The quality of the algorithm is improved by encoding the mean constraint with a feature sparsity table. Among them, formula (1) always guarantees infinitely close to but there is no guarantee able to unconditionally and infinitely approach Through experimental conditions and practical applications, it can be proved that there is characteristic sparse coding noise before the two

因此，需要对特征稀疏编码噪声进行降低，也就是需把特征稀疏编码噪声作为一个新的约束项。加入约束项后的目标函数系统变为：Therefore, it is necessary to reduce the feature sparse coding noise, that is, to take the feature sparse coding noise as a new constraint item. After adding constraints, the objective function system becomes:

$\begin{matrix} {δ δ}_{L L 22}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} \\ + + θ θ {| | | | (({δ δ}_{H h}^{ω ω} - - {δ δ}_{L L}^{ω ω})) {Σ Σ}_{n no = = 11}^{\infty \infty} (({δ δ}_{n no} cos cos \frac{nπω nπω}{{δ δ}_{H h}^{ω ω} - - {δ δ}_{L L}^{ω ω}} + + sin sin \frac{nπω nπω}{{δ δ}_{H h}^{ω ω} - - {δ δ}_{L L}^{ω ω}})) | | | |}_{ϵ ϵ}^{22} \end{matrix} - - - - - - ((1010))$

接下来采用零均值随机变量对式(10)进行变形，变形的结果为：Next, the zero-mean random variable is used to deform the formula (10), and the result of the deformation is:

${δ δ}_{L L 33}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} + + {Δ Δ}^{ω ω} - - - - - - ((1111))$

本发明采用一种新的加权迭代的K值方法来表示距离特征，权重越小时的距离特征越明显；权值越大时的距离特征越模糊。基于这些距离特征，本发明提出一条新的公式(12)来计算获得聚类字典学习与稀疏编码优化均值：The present invention adopts a new weighted iterative K value method to represent the distance feature, the smaller the weight is, the more obvious the distance feature is; the larger the weight is, the more blurred the distance feature is. Based on these distance features, the present invention proposes a new formula (12) to calculate and obtain clustering dictionary learning and sparse coding optimization mean:

${Δ Δ}^{ω ω}_{11}^{T T} = = {Σ Σ}_{(({K K}_{H h},, {K K}_{L L},, δ δ,, ω ω)) &Element; &Element; N N} {δ δ}_{L L}^{ω ω} * * &dtri; &dtri; {{{K K}_{H h},, {K K}_{L L},, δ δ,, ω ω}} - - - - - - ((1212))$

经过精密计算之后，本发明能够获得高精度的特征稀疏表示的超分辨率复杂系数。After precise calculation, the present invention can obtain super-resolution complex coefficients of high-precision feature sparse representation.

至此，经过改进的聚类字典学习及特征稀疏表示的超分辨率算法已经完成了大部分工作，但考虑到同构性问题，为了使算法稳定性可靠性进一步提高，本发明将式(10)(12)合并优化处理后得：So far, the improved clustering dictionary learning and the super-resolution algorithm of feature sparse representation have completed most of the work, but considering the problem of isomorphism, in order to further improve the stability and reliability of the algorithm, the present invention uses formula (10) (12) After merging and optimizing:

$\begin{matrix} {δ δ}_{L L 44}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} \\ + + \sqrt[T T]{{Σ Σ}_{(({K K}_{H h},, {K K}_{L L},, δ δ,, ω ω)) &Element; &Element; N N} {δ δ}_{L L}^{ω ω} * * &dtri; &dtri; {{{K K}_{H h},, {K K}_{L L},, δ δ,, ω ω}}} \end{matrix} - - - - - - ((1414)) . .$

本发明的整个系统及算法的流程图如图9所示。The flowchart of the whole system and algorithm of the present invention is shown in FIG. 9 .

实施例三Embodiment three

将本发明的算法与各传统算法的PSNR(dB)和SSIM值进行一个对比，如下表1所示。每个样本(Test1～Test5)的第一行为PSNR，第二行为SSIM。The algorithm of the present invention is compared with the PSNR (dB) and SSIM values of various traditional algorithms, as shown in Table 1 below. The first row of each sample (Test1~Test5) is PSNR, and the second row is SSIM.

表1Table 1

从表1可以看出，本发明的算法可以非常高质量地对视频系列图像进行超分辨率视频处理，从而获得的质量高的视频处理超分辨率图像。It can be seen from Table 1 that the algorithm of the present invention can perform super-resolution video processing on video series images with very high quality, thereby obtaining high-quality video-processed super-resolution images.

本发明融合了SOC片上系统，先通过复杂度处理和特征向量提取获取低分辨率图像的纹理和几何结构特征，然后采用训练集进行样本训练，最后再根据训练的结果采用很好地利用了图像的先验知识，能有效防止恢复的图像过平滑；包括了进行复杂度处理的过程，并采用高分辨率的高频分量构成的样本集和特征稀疏表示的分辨率超分辨率算法进行处理，降低了低分辨率图像的数量以及错误的配准所带来的影响，适应性较好，且不易丢失重要的细节信息，失真率较低。实际的实验测试结果表明，采用本发明的方法处理后获得的视频流图像质量较高，处理结果较理想。The present invention integrates the SOC system on a chip, first obtains the texture and geometric structure features of the low-resolution image through complexity processing and feature vector extraction, then uses the training set for sample training, and finally adopts a well-utilized image according to the training result The prior knowledge can effectively prevent the restored image from being too smooth; it includes the process of complex processing, and uses a sample set composed of high-resolution high-frequency components and a resolution super-resolution algorithm with sparse representation of features for processing. The number of low-resolution images and the impact of wrong registration are reduced, the adaptability is better, and important detail information is not easy to be lost, and the distortion rate is low. The actual experimental test results show that the image quality of the video stream obtained after processing by the method of the present invention is relatively high, and the processing result is ideal.

以上是对本发明的较佳实施进行了具体说明，但本发明创造并不限于所述实施例，熟悉本领域的技术人员在不违背本发明精神的前提下还可做作出种种的等同变形或替换，这些等同的变形或替换均包含在本申请权利要求所限定的范围内。The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the described embodiments, and those skilled in the art can also make various equivalent deformations or replacements without violating the spirit of the present invention. , these equivalent modifications or replacements are all within the scope defined by the claims of the present application.

Claims

1. The real-time video super-resolution processing method that merges Cortex-A7 is characterized in that: comprising:

A. Carry out video sampling, obtain low-resolution video frames and input them into the SOC system on chip;

B. Perform complexity processing, feature vector extraction, and sample set training on the low-resolution video frame in sequence, so as to obtain feature vectors that need to be matched, and the sample set is constructed using high-resolution high-frequency components;

C. According to the feature vectors that need to be matched, the improved super-resolution algorithm based on clustering dictionary self-learning and feature sparse representation is combined with the encoding and decoding technology of the SOC system on chip to perform super-resolution processing on low-resolution video frames. Thus outputting a stream of high-resolution video frames.

2. the real-time video super-resolution processing method of fusion Cortex-A7 according to claim 1, is characterized in that: described step C, it comprises:

C1. Construct a super-resolution algorithm based on clustering dictionary self-learning and feature sparse representation with improved main features;

C2. Perform super-resolution processing on the low-resolution video frames according to the required matching feature vectors and the constructed super-resolution algorithm, so as to output a stream of high-resolution video frames.

3. the real-time video super-resolution processing method of fusion Cortex-A7 according to claim 2, is characterized in that: described step C1, it comprises:

C11. Establish an over-complete database;

C12. Calculate the sparse representation coefficient of the input low-resolution video image block;

C13. Calculating the sparse coding coefficients under the low-resolution dictionary and the sparse coding coefficients under the high-resolution dictionary;

C14. Reconstructing high-resolution video image blocks according to the sparse representation coefficients, sparse coding coefficients, high-resolution image libraries and low-resolution dictionaries in the over-complete database of low-resolution video image blocks;

C15, using a clustering algorithm and a principal component analysis method to extract a video image block set, and then using the K-SVD algorithm to jointly train the high and low resolution image block sets;

C16. According to the results of joint training, the orthogonal matching pursuit method is used to obtain a super-resolution algorithm based on clustering dictionary self-learning and feature sparse representation with improved main features.

4. the real-time video super-resolution processing method of fusion Cortex-A7 according to claim 3, is characterized in that: described step C13, it is specifically:

Calculate the sparse coding coefficients under the low-resolution dictionary and the sparse coding coefficients under the high-resolution dictionary, the sparse coding coefficients under the low-resolution dictionary and the sparse coding coefficients under the high-resolution dictionary The calculation formulas are respectively:

{δ δ}_{L L}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11},,

{δ δ}_{H h}^{ω ω} = = arg arg min min {| | | | {N N}_{H h}^{ω ω} - - {K K}_{H h} {δ δ}_{H h}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{H h}^{ω ω}} | | | |}_{11},,

Among them, K _L is a low-resolution dictionary function, Represents a video stream in a low-resolution image, ρ is a given parameter of the matrix norm, Indicates the ideal low-resolution dictionary function constraint item after derivation, ε is the complex regularization parameter after feature processing, and L is the norm instead of sparse coding;

K _H is a high-resolution dictionary function, Represents a video stream in a high-resolution image, Denotes the ideal high-resolution dictionary function constraint item after derivation, and H is the norm instead of sparse coding.

5. the real-time video super-resolution processing method of fusion Cortex-A7 according to claim 4, is characterized in that: described step C15, it is specifically:

A clustering algorithm and a principal component analysis method are used to extract a video image block set, and then the K-SVD algorithm is used to jointly train the high-resolution and low-resolution image block sets, thereby obtaining the result data of the joint training, and the result data of the joint training { K _H , K _L ,δ,ω} are:

Among them, N is the column vector obtained after the high-resolution and low-resolution image blocks are jointly trained,

N = {[N_{h} \sec (N_{h}) / e^{iω w_{1}}, N_{L} \sec (N_{L}) / e^{iω w_{2}}]}^{T},

K = {[K_{h} \sec (K_{h}) / e^{iω w_{1}}, K_{L} \sec (K_{L}) / e^{iω w_{2}}]}^{T}

is the high and low resolution joint dictionary, w ₁ and w ₂ are the dimensions of the column vectors of the high and low resolution video streams after training respectively, δ=[δ ₁ ,δ ₂ ,δ ₃ ,δ ₄ ,..... .,δ ₅ ] is the coding coefficient matrix,

ω = | | \begin{matrix} ω_{1} & ω_{2} \\ ω_{3} & ω_{4} \end{matrix} | |

is the decoding coefficient matrix.

6. the real-time video super-resolution processing method of fusion Cortex-A7 according to claim 5, is characterized in that: described step C2, it comprises:

C21. Match the feature vectors that need to be matched in the dictionary database, and judge whether the matching is successful, if so, execute step C23, otherwise, execute step C22;

C22, using the improved K value iterative algorithm to carry out clustering dictionary self-study on the low-resolution video frame, and then perform step C23;

C23. According to the feature number coefficient of the constraint item, find out the high-speed training dictionary library from the complete database, and integrate the video coding and decoding technology of the SOC system on chip to optimize and integrate the low-resolution video frame, thereby outputting high-quality and high-resolution video flow.

7. the real-time video super-resolution processing method of fusion Cortex-A7 according to claim 6, is characterized in that: described step C22, it comprises:

C221. Perform training sample processing on existing video frames, thereby obtaining the primary clustering function formula, the expression of the primary clustering function is:

Among them, I is the dictionary type, C is the primary clustering coefficient, is the number of iterations, n is a constant coefficient, is the reference data for training samples, is the cluster difference coefficient;

C222, starting from the first pixel in the upper left corner of each frame of video image, take a video image block every other pixel, and use the LASSO algorithm to solve the optimal solution of the primary clustering function for the taken video image block The optimal solution of the primary clustering function The expression is:

{δ δ}_{L L 11}^{ω ω} * * &dtri; &dtri; = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} {Π Π}_{L L}^{&Element; &Element;} \underset{00 \leq \leq x x \leq \leq 11}{max max} \overset{~ ~}{X x} {e e}^{- - {x x}^{22}};;

Among them, ▽ is the LASSO operator, is the optimal integration coefficient for super-resolution, The expression is:

\overset{~ ~}{X x} = = arg arg {min min}_{α α} {Σ Σ}_{i i} {| | | | {R R}^{i i} (({X x}_{H h} - - {X x}_{L L})) - - \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} | | | |}_{66}^{88};;

C223. Classify the video image blocks to obtain K clusters, and then learn a sub-dictionary from each cluster to obtain the optimal integration coefficient The following K sub-dictionaries.

8. the real-time video super-resolution processing method of fusion Cortex-A7 according to claim 7, is characterized in that: described step C23, it comprises:

C231, the optimal solution to the primary clustering function Perform feature sparse coding mean value constraint processing, so as to obtain the objective function after adding the feature number coefficient of the constraint item The objective function The expression is:

\begin{matrix} {δ δ}_{L L 22}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} \\ + + θ θ {| | | | (({δ δ}_{H h}^{ω ω} - - {δ δ}_{L L}^{ω ω})) {Σ Σ}_{n no = = 11}^{\infty \infty} (({δ δ}_{n no} cos cos \frac{nπω nπω}{{δ δ}_{H h}^{ω ω} - - {δ δ}_{L L}^{ω ω}} + + sin sin \frac{nπω nπω}{{δ δ}_{H h}^{ω ω} - - {δ δ}_{L L}^{ω ω}})) | | | |}_{ϵ ϵ}^{22} \end{matrix},,

where θ is a characteristic constant, and

θ = \arg \log_{5} | | \sqrt[ω]{δ_{L}^{ω}} + N_{L}^{ω} | |,

(δ_{h}^{ω} - δ_{L}^{ω}) Σ_{no = 1}^{\infty} (δ_{no} \cos \frac{nπω}{δ_{h}^{ω} - δ_{L}^{ω}} + \sin \frac{nπω}{δ_{h}^{ω} - δ_{L}^{ω}})

to represent and eigenvalue of eigenvector distance;

C232. Calculate the optimized mean of clustering dictionary learning and sparse coding The optimized mean The calculation formula is;

{Δ Δ}^{ω ω}_{11}^{T T} = = {Σ Σ}_{(({K K}_{H h},, {K K}_{L L},, δ δ,, ω ω)) &Element; &Element; N N} {δ δ}_{L L}^{ω ω} * * &dtri; &dtri; {{{K K}_{H h},, {K K}_{L L},, δ δ,, ω ω}},,

C233, for the objective function after adding the characteristic number coefficient of the constraint item and optimized mean Combine and optimize to obtain the optimized integrated objective function The optimization ensemble objective function The expression is:

\begin{matrix} {δ δ}_{L L 44}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} \\ + + \sqrt[T T]{{Σ Σ}_{(({K K}_{H h},, {K K}_{L L},, δ δ,, ω ω)) &Element; &Element; N N} {δ δ}_{L L}^{ω ω} * * &dtri; &dtri; {{{K K}_{H h},, {K K}_{L L},, δ δ,, ω ω}}} \end{matrix};;

C234. Perform optimization integration processing on the low-resolution video frame according to the optimization integration objective function, so as to generate and output a high-resolution video stream.

9. the real-time video super-resolution processing method of fusion Cortex-A7 according to claim 8, is characterized in that: described step C232, it comprises:

S1. Using a zero-mean random variable pair to the objective function after adding the characteristic number coefficient of the constraint item Transform to obtain the deformed objective function The deformed objective function for:

{δ δ}_{L L 33}^{ω ω} = = arg arg min min {| | | | {N N}_{L L}^{ω ω} - - {K K}_{L L} {δ δ}_{L L}^{ω ω} | | | |}_{22}^{ρ ρ} + + \frac{11}{1515} ϵ ϵ {| | | | \sqrt[ω ω]{{δ δ}_{L L}^{ω ω}} | | | |}_{11} + + {Δ Δ}^{ω ω},,

Among them, ^Δω is used to represent

θ {| | (δ_{h}^{ω} - δ_{L}^{ω}) Σ_{no = 1}^{\infty} (δ_{no} \cos \frac{nπω}{δ_{h}^{ω} - δ_{L}^{ω}} + \sin \frac{nπω}{δ_{K}^{ω} - δ_{L}^{ω}}) | |}_{ϵ}^{2};

S2. Calculate the optimized mean value of clustering dictionary learning and sparse coding according to the deformed objective function