CN108537733A - Super resolution ratio reconstruction method based on multipath depth convolutional neural networks - Google Patents
Super resolution ratio reconstruction method based on multipath depth convolutional neural networks Download PDFInfo
- Publication number
- CN108537733A CN108537733A CN201810325131.5A CN201810325131A CN108537733A CN 108537733 A CN108537733 A CN 108537733A CN 201810325131 A CN201810325131 A CN 201810325131A CN 108537733 A CN108537733 A CN 108537733A
- Authority
- CN
- China
- Prior art keywords
- image
- neural networks
- network
- convolutional neural
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 39
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012549 training Methods 0.000 claims abstract description 51
- 238000012360 testing method Methods 0.000 claims abstract description 20
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 238000013507 mapping Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 9
- 101100365548 Caenorhabditis elegans set-14 gene Proteins 0.000 claims description 6
- 230000003321 amplification Effects 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 6
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 6
- 230000002708 enhancing effect Effects 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 4
- 230000006870 function Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 238000007796 conventional method Methods 0.000 claims description 2
- 238000003475 lamination Methods 0.000 claims 1
- 230000000007 visual effect Effects 0.000 abstract description 8
- 230000001537 neural effect Effects 0.000 abstract description 2
- 238000005516 engineering process Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 235000013399 edible fruits Nutrition 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000003306 harvesting Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a kind of super resolution ratio reconstruction methods based on multipath depth convolutional neural networks, include the following steps：Total training set, total training set image preprocessing, test set is obtained to prepare and realize image reconstruction with the convolutional layer of convolutional neural networks；Multipath convolutional neural networks structure proposed by the present invention, a plurality of branch is increased on original single path neural net base, the convolution kernel of the characteristics of image different number of different scale can be handled, method more original on reconstruction quality and visual effect has promotion while not increasing population parameter amount.
Description
Technical field：
The present invention relates to a kind of super resolution ratio reconstruction methods based on multipath depth convolutional neural networks, belong at image
Manage technical field.
Background technology：
With the development of science and technology with the progress of human society, information interchange and processing are more and more important, compared to word, sound
The information such as sound, smell, image information has the characteristics that intuitive, image and contains much information, the study found that complete acquired in the mankind
In portion's information, the ratio of visual information is up to 60%, so image, which becomes, obtains the important next of information in people's Working Life
Source has great importance to the research and processing of image.
Resolution ratio is the measurement to screen picture precision, it is the quantity of display display pixel.The point of screen picture,
Line, face are made of pixel one by one, therefore the pixel for constituting image is more, and screen picture picture is finer clear, similarly
Transmissive information content is also abundanter in image-region, and visual effect is also better, and such case is referred to as high-resolution
(HighResolution, abbreviation HR), it is on the contrary then be known as low resolution (Low Resolution, abbreviation LR).Resolution ratio reflects
The measurement of informative degree and image detail expressive ability height that image is included.
It is a kind of approach for improving image resolution ratio to the hardware device in sampling system improve, and there are mainly two types of sides
Method：Improve the density of sensing unit；Increase the quantity of sensing unit.Both methods increases the cost of hardware device, and
And the former makes sensing chip size become smaller and bring more interference noises.As a kind of software technology, Image Super-resolution
Rate (Super-Resolution, abbreviation SR) algorithm for reconstructing can realize image or video resolution under hardware device constraint
Enhancing improves the visual effect of image or video.
The purpose of image super-resolution rebuilding technology is to improve the resolution ratio of input picture, i.e., by enhancing in input picture
The clarity of appearance restores the not existing detailed information of input picture itself, the HR images of final output high quality.Oversubscription
Resolution reconstruction technique is suitable for being difficult to obtain the occasion of high quality graphic, such as in video monitoring and remote sensing, the former obtains past
Toward the low quality face for being only tens pixels, and for the latter's HR numbers
The cost of word imaging device is often excessively high, is limited by hardware cost and people are to high image resolution ratio
Unremitting pursuit, image SR reconstruction techniques obtain extensive research concern.
Image SR algorithm for reconstructing is a kind of skill recovering HR images by a secondary LR images or a LR image sequences reconstruct
Art.The HR images that this technology obtains can recover the detail of the high frequency of observed image loss, have better vision effect
Fruit is conducive to the various subsequent processings to image, and the target of image SR technologies is to solve this typical ill-conditioning problem.
Super-resolution rebuilding algorithm is generally divided into three classes:Super-resolution reconstruction based on interpolation, based on rebuilding and based on study
Algorithm is built, the method based on interpolation realizes that simple computation is efficient, but reconstructed results are excessively smooth, and it is thin can not to generate high frequency
Section；Method based on reconstruction can not handle the complicated image structure in natural image；Method based on study relies on sample image
The mapping between low-resolution image and high-definition picture is established in library, when inputting new low-resolution image, according to this
Mapping relations reconstruct high-definition picture, so the key of such method is the learning algorithm itself for sample image library,
The present invention is mainly using the super resolution ratio reconstruction method based on study.
Being disclosed in the information of the background technology part, it is only intended to increase understanding of the overall background of the invention, without answering
It has been the prior art well known to persons skilled in the art when being considered as recognizing or imply that the information is constituted in any form.
Invention content：
The purpose of the present invention is in view of the above-mentioned problems, exploring the depth convolution god suitable for natural image super-resolution problem
It through network structure, accounts in terms of network depth and width two, is improved under the premise of not increasing network population parameter amount
The quality and visual effect of image reconstruction.
To achieve the above object, specific implementation step of the present invention is as follows：
A kind of super resolution ratio reconstruction method based on multipath depth convolutional neural networks, includes the following steps：
(1) total training set is obtained：
Training set is in general super-resolution network training collection 91images and Berkeley partitioned data set
200images, original training set are 291 nature pictures, and 0.9,0.8,0.7,0.6 is carried out respectively to each Zhang Xunlian pictures
The diminution of multiple, and carry out 90 °, 180 °, 270 ° of rotations and mirror image switch, the nature picture warp in original training set
Crossing data enhances to obtain 20 pictures, and total training set size is 291*20=5820 nature pictures；
(4) total training set image preprocessing：
Picture in total training set is carried out 2 times, 3 times and 4 times resolution ratio with BICUBIC methods respectively to reduce, is cut into
The picture block of 41*41 sizes, patch is dimensioned to 64, and training set is saved as HDF5 formats as convolutional neural networks
Input；
(5) test set prepares：
Setting 4 groups of test sets is respectively：Set5, Set14, B100, Urban100, wherein Set5, Set14, B100 are three
A common image super-resolution data set, separately includes 5,14,100 various sizes of natural pictures, and urban100 is
100 different City scenarios pictures；Test set carries out identical pretreatment according to training set in step (2), defeated after BICUBIC
Enter network and carry out image super-resolution rebuilding, patch is dimensioned to 2；
(4) convolutional layer of convolutional neural networks is used to realize image reconstruction：
It is divided into three parts：Feature extraction, Feature Mapping and image reconstruction, feature extraction and image recovered part are respectively with one
A 3*3 convolution kernels realize that Feature Mapping part is realized with 27 3*3 convolution kernels；First by the coloured silk of RGB in training set and test set
Color image is converted into YCbCr color spaces, only calculates the channels Y, that is, luminance channel, is then cut into the patch that size is 41*41 and makees
For input；A ReLU activation primitive is closely followed after each convolutional layer.
Image super-resolution refers to by a width low-resolution image (Low Resolution, abbreviation LR) or low resolution figure
As sequence recovers high-definition picture (High Resolution, abbreviation HR), which is the warp in computer vision field
Allusion quotation problem can increase effective information in original low resolution picture, improve the visual effect of picture.The present invention is for single
The super-resolution problem of image, it is intended to be explored using depth convolutional neural networks and a kind of realize that speed is fast, recovery quality is high
Image super-resolution method.
The technical solution that the present invention further limits is：
Further：Convolutional neural networks in step (2) and (4) are specially：Include 29 convolutional layers, each layer of volume altogether
Product core size is 3*3, and it is to generate 64 kinds of features by convolution that characteristic pattern quantity, which is 64, to keep output characteristic pattern size constant,
Padding is set as 1；Characteristic extraction part includes one layer；Feature Mapping part includes 29 layers；Image recovered part includes one
Layer, output characteristic pattern quantity are 1, i.e., the high-resolution residual error image that e-learning arrives, longest path receptive field calculates：
RFn-1=(RFn- 1) * Stride+Kerneral_size,
Wherein, RFnFor (n-1)th layer of receptive field of n-th layer pair, Stride is convolution step-length, and Kerneral_size is convolution
Core size successively calculates forward receptive field using the calculation of top to down from last layer；Neural network it is last
One layer of receptive field to the second last layer be 3, according to before formula to derive receptive field be 41, that is, the high-definition picture generated
In each pixel have relationship with 41*41 pixel in input picture.
Further：Super-resolution method in step (3) is specially：In original chain type convolutional neural networks structure base
Multichannel gauge structure is introduced on plinth, since second convolutional layer, each two rolls up base's additional convolutional layer in parallel, and in parallel
Convolution layer parameter and original convolutional layer parameter sharing, carry out Fusion Features in the end of each parallel-connection structure, it is multiple similar
Parallel-connection structure nesting constitute multichannel gauge structure, the reconstruction performance of network can be promoted in the case where not increasing network parameter.
Further：The first part of convolutional neural networks structure is extraction and the character representation of image block, utilizes convolution
The feature of the property extraction residual image of network.
Further：Detailed process is indicated with following formula：
F1(X)=max (0, W1*X+B1),
Wherein, W1And B1Convolution filter and biasing, W are indicated respectively1Size be c*f1*f1*n1, c is input picture
Port number, f1For the size of convolution filter, n1For the quantity of convolution filter；W1By n1A convolution filter is applied to image,
And each convolution kernel size is c*f1*f1；Output is by n1A character network figure composition；B1It is a n1The vector of dimension, each of which
Element is related with a convolution filter respectively.
Further：Feature is mapped and merged using multichannel gauge structure in the second part of network, formula is：
F2(X)=max (0, W2*F1(X)+B2),
F3(X)=max (0, W3*F2(X)+B3),
F21(X)=max (0, W2*F1(X)+B2),
Wherein, F2(X), F3(X) it is second and the output of third convolutional layer, F21(X) it is the output of convolutional layer in parallel, and
Join W and B and the second layer parameter sharing of convolutional layer, then with the mode being added by F21(X) and F3 *(X) it merges.Specific such as attached drawing
Shown in 2,9 similar structures form final multipath convolutional neural networks, and the final output of network configuration second part is
F1*9(X)。
Further：The Part III of network try again convolution carry out image reconstruction, similar to being averaged for conventional method
Processing, formula are as follows：
F20(X)=W20*F19(X)+B20,
Wherein, F20(X) it is the output of the 20th convolutional layer, the residual image that dimension 1, i.e. e-learning arrive, according to complete
Office's residual error learning structure, network inputs X, the high-definition picture that network finally reconstructs are Y, then：
Y=X+F20(X),
Assuming that training set isThat is N is to the high-low resolution images pair of S kind amplification factors, for each
To sample, the high-resolution residual error image to be learnt is represented byThe network Nonlinear Mapping to be learnt is F
(X), if F (X) parameter is θ={ WK,bk, so the loss function of network is represented by：
Using above-mentioned convolutional neural networks, learnt using multichannel gauge structure and global residual error, to reconstruct final height
Image in different resolution.
Further：In step (4), ReLU activation primitives, that is, Unit (ReLU, max (0, x)).
Beneficial effects of the present invention：
(1) super-resolution rebuilding result.
Multipath convolutional neural networks structure proposed by the present invention increases on original single path neural net base
A plurality of branch can handle the convolution kernel of the characteristics of image different number of different scale, not increase the same of population parameter amount
When method more original on reconstruction quality and visual effect have promotion.
(2) data enhance.
The present invention rotates each image in original 291images data sets, overturning and different multiples
It reduces, has effectively expanded data set so that deep neural network can be trained up.
(3) the various sizes of amplification of image.
The present invention uses multi-scale method in training network, and original image is carried out 2 times, 3 times and 4 times different resolutions
The diminution of size, so in test can also 2 times be carried out to image, the amplification of 3 times and 4 times resolution ratio.
(4) training deep neural network skill
Global residual error structure is added at training depth convolutional Neural network in the present invention in a network, to improve network receipts
Speed is held back, and trained iteration is set every 300,000 times to 10 times of diminutions of learning rate progress, to ensure that network is received
It holds back in optimum position.
Super resolution ratio reconstruction method proposed by the present invention based on multipath convolutional neural networks, it is advantageous that：
1. multichannel gauge structure：
Inception modules in Googlenet are in parallel by different convolutional layers, improved while increasing network-wide
Adaptability of the network to different scale feature, can convert same input Mapping implementation difference, and all by their result
Be connected in single one output, this module effectively improves the performance of the neural network for classification task, the present invention by
It inspires in inception modules, in conjunction with the characteristics of this problem of image super-resolution, has built and be used for image super-resolution task
Multipath convolutional neural networks structure.
The present invention maps characteristics of image using different number of convolutional layer parallel method, in the end of parallel-connection structure
Fusion Features, volume base parameter in parallel and original volume base parameter sharing are carried out, is made of this multiple class formation nesting more
Path structure so that the reconstruction quality and visual effect of image are all promoted.
2. residual error learns：
Depth residual error network is that the depth convolutional network proposed in 2015 just harvests figure once being born in ImageNet
As the champion of classification, detection, positioning three.The thought of residual error network is that layer is expressed as study residual error function according to input, residual
Poor network is easier to optimize, and can improve the accuracy rate of network by increasing comparable depth.Its core is that solve
The side effect (degenerate problem) for increasing depth zone, can improve network performance by merely increasing network depth in this way.
Residual error network is specifically shown in bibliography K.He, X.Zhang, S.Ren, and J.Sun.Deep residual learning for
image recognition.In CVPR 2016。
Convolutional neural networks algorithm is applied to Image Super-resolution field by SRCNN networks first, and achieves more traditional side
Method preferably rebuilds effect, but SRCNN networks directly learn mapping of the low-resolution image to high-definition picture, and network is received
It holds back slowly, once deepening the network number of plies, training would become hard to be restrained.SRCNN networks are specifically shown in bibliography Dong, C., Loy,
C.C.,He,K.,Tang,X.:Learning a deep convolutional network for image super-
resolution.In:ECCV.(2014)184–199.The it is proposed of residual error learning structure so that network can learn input with it is defeated
Residual error between going out, accelerates the convergence of deep layer convolutional neural networks, and allows network deeper, and effect is better.The present invention
Global residual error learning structure is introduced, the residual error between e-learning low-resolution image and full resolution pricture is counted in network layer
Up to 20 layers, when convolution kernel number 29, also can be compared with rapid convergence.
3. multiple dimensioned training method：
SRCNN networks can only carry out image the amplification of single scale, and re -training net is needed when different scale being needed to amplify
Network, present invention introduces multiple dimensioned training methods, and training set is made to the diminution of different resolution multiple so that the same network structure
2 times, 3 times and 4 times of amplification can be realized to image resolution ratio.
4. algorithm is realized:
Among numerous deep learning frames, the present invention selects caffe to realize proposed network structure.Caffe is one
A clear, efficient deep learning frame, kernel language is C++, it supports order line, Python and Matlab interfaces, it was both
It can be run on CPU, also can accelerate training and test with GPU.Because caffe is to carry out network with the mode of configuration file to take
It builds, so realizing that various convolutional neural networks structures are convenient and efficient in caffe, this is also the reason of present invention is using caffe.
For programming software, the present invention uses VS2013 and Matlab.The place of training set and test set is realized with Matlab
Science and engineering is made, and in data test phase, the present invention divides single picture with the Matconvnet frames realization on Matlab platforms
Resolution is amplified, and finally carries out statistics and analysis to experimental data.
Description of the drawings：
Attached drawing 1 is flow diagram of the present invention；
Attached drawing 2 is multipath neural network structure figure.
Specific implementation mode：
The specific implementation mode of the present invention is described in detail below, it is to be understood that protection scope of the present invention is not
It is restricted by specific implementation.
Unless otherwise explicitly stated, otherwise in entire disclosure and claims, term " comprising " or its change
It changes such as "comprising" or " including " etc. and will be understood to comprise stated element or component, and do not exclude other members
Part or other component parts.
(1) prepare training set.It will be in general super-resolution network training collection 91images and Berkeley partitioned data set
200images is incorporated as training set, and original training set is 291 nature pictures, in order to make full use of trained picture, experiment
In data enhancing has been done to training set, carry out the diminution of 0.9,0.8,0.7,0.6 multiple respectively to each Zhang Xunlian pictures, and
Carry out 90 °, 180 °, 270 ° of rotations and mirror image switch, therefore a nature picture in original training set is by data enhancing
20 pictures are obtained, total training set size is 291*20=5820 nature pictures；
(2) training set image preprocessing：Picture in the enhanced training set of data is carried out respectively with BICUBIC methods
2 times, 3 times and 4 times resolution ratio reduce, and cut into the picture block of 41*41 sizes, patch is dimensioned to 64, and training set is protected
Save as input of the HDF5 formats as convolutional neural networks；
(3) setup test collection.4 groups of test sets：Set5, Set14, B100, Urban100, wherein urban100 are 100
Different City scenarios pictures；Test set and training set carry out identical pretreatment, and input network progress image is super after BICUBIC
Resolution reconstruction, patch are dimensioned to 2；
(4) training network.In caffe deep learning platforms, realized in attached drawing 2 by net.prototxt configuration files
Multipath convolutional neural networks structure, by solver.prototxt configuration files be arranged network training relevant parameter, it is excellent
Change function is adam, and basic learning rate is set as 10e-4, and learning strategy step, stepsize 300000, gamma are
0.1, i.e., often train 300000 learning rates to fall to original 0.1 times, total iterations are 900000 times, and use gpu
Accelerate training, training duration is about 30 hours.Computer is configured to Intel Core i7-6700K, NVIDIA GeForce GTX
980Ti GDDR5 6GB, 32GB RAM,
Operating system win10；
(5) test network.The trained neural network models of caffe are extracted with Matlab, on Matconvnet platforms
Network model is imported respectively to Set5, Set14, B100, this 4 test sets of Urban100 are tested, and image reconstruction knot is preserved
Fruit and the PSNR values and its mean value for calculating every reconstruction image.PSNR calculation formula are：
Wherein, f is true picture, f*For super-resolution rebuilding image, M is the number of pixels of f, and the unit of PSNR is dB.
1. test result of the present invention of table and each algorithm the PSNR values that be averaged compare, and unit dB, runic is peak.
Graphical results illustrate that, using PSNR as image reconstruction quality evaluation standard, inventive algorithm takes on 4 test sets
Obtained best image reconstruction effect.
The description of the aforementioned specific exemplary embodiment to the present invention is in order to illustrate and illustration purpose.These descriptions
It is not wishing to limit the invention to disclosed precise forms, and it will be apparent that according to the above instruction, can much be changed
And variation.The purpose of selecting and describing the exemplary embodiment is that explaining the specific principle of the present invention and its actually answering
With so that those skilled in the art can realize and utilize the present invention a variety of different exemplary implementation schemes and
Various chooses and changes.The scope of the present invention is intended to be limited by claims and its equivalents.
Claims (8)
1. a kind of super resolution ratio reconstruction method based on multipath depth convolutional neural networks, includes the following steps：
(1) total training set is obtained：
Training set is the 200images in general super-resolution network training collection 91images and Berkeley partitioned data set, former
The training set of beginning is 291 nature pictures, carries out the diminution of 0.9,0.8,0.7,0.6 multiple respectively to each Zhang Xunlian pictures,
And 90 ° are carried out, 180 °, 270 ° of rotations and mirror image switch, a nature picture in original training set is by data enhancing
20 pictures are obtained, total training set size is 291*20=5820 nature pictures；
(2) total training set image preprocessing：
Picture in total training set is carried out 2 times, 3 times and 4 times resolution ratio with BICUBIC methods respectively to reduce, cuts into 41*41
The picture block of size, patch is dimensioned to 64, and training set is saved as HDF5 formats as the defeated of convolutional neural networks
Enter；
(3) test set prepares：
Setting 4 groups of test sets is respectively：Set5, Set14, B100, Urban100, wherein Set5, Set14, B100 are three normal
With image super-resolution data set, 5,14,100 various sizes of natural pictures are separately included, urban100 is 100
Different City scenarios pictures；Test set carries out identical pretreatment according to training set in step (2), and net is inputted after BICUBIC
Network carries out image super-resolution rebuilding, and patch is dimensioned to 2；
(4) convolutional layer of convolutional neural networks is used to realize image reconstruction：
It is divided into three parts：Feature extraction, Feature Mapping and image reconstruction, feature extraction and image recovered part are respectively with a 3*
3 convolution kernels realize that Feature Mapping part is realized with 27 3*3 convolution kernels；First by the cromogram of RGB in training set and test set
As being converted into YCbCr color spaces, the channels Y, that is, luminance channel is only calculated, it is the patch of 41*41 as defeated to be then cut into size
Enter；A ReLU activation primitive is closely followed after each convolutional layer.
2. the super resolution ratio reconstruction method according to claim 1 based on multipath depth convolutional neural networks, feature
It is：Convolutional neural networks in step (2) and (4) are specially：Include 29 convolutional layers altogether, each layer of convolution kernel size is
3*3, it is to generate 64 kinds of features by convolution that characteristic pattern quantity, which is 64, and to keep output characteristic pattern size constant, padding is set as
1；Characteristic extraction part includes one layer；Feature Mapping part includes 29 layers；Image recovered part includes one layer, exports characteristic pattern number
Amount is 1, i.e., the high-resolution residual error image that e-learning arrives, longest path receptive field calculates：
RFn-1=(RFn- 1) * Stride+Kerneral_size,
Wherein, RFnFor (n-1)th layer of receptive field of n-th layer pair, Stride is convolution step-length, and Kerneral_size is that convolution kernel is big
It is small, using the calculation of top to down, receptive field is successively calculated forward from last layer；Last layer of neural network
Be 3 to the receptive field of the second last layer, according to before formula to derive receptive field is 41, that is, in the high-definition picture generated
Each pixel has relationship with 41*41 pixel in input picture.
3. the super resolution ratio reconstruction method according to claim 1 based on multipath depth convolutional neural networks, feature
It is：Super-resolution method in step (3) is specially：Multichannel is introduced on original chain type convolutional neural networks architecture basics
Gauge structure, since second convolutional layer, each two rolls up base's additional convolutional layer in parallel, and convolution layer parameter in parallel
With original convolutional layer parameter sharing, Fusion Features are carried out in the end of each parallel-connection structure, multiple similar parallel-connection structures are embedding
Set constitutes multichannel gauge structure, and the reconstruction performance of network can be promoted in the case where not increasing network parameter.
4. the super resolution ratio reconstruction method according to claim 2 based on multipath depth convolutional neural networks, feature
It is：The first part of convolutional neural networks structure is extraction and the character representation of image block, is carried using the property of convolutional network
Take the feature of residual image.
5. the super resolution ratio reconstruction method according to claim 4 based on multipath depth convolutional neural networks, feature
It is：Detailed process is indicated with following formula：
F1(X)=max (0, W1*X+B1),
Wherein, W1And B1Convolution filter and biasing, W are indicated respectively1Size be c*f1*f1*n1, c is the channel of input picture
Number, f1For the size of convolution filter, n1For the quantity of convolution filter；W1By n1A convolution filter is applied to image, and every
A convolution kernel size is c*f1*f1；Output is by n1A character network figure composition；B1It is a n1The vector of dimension, each of which element
It is related with a convolution filter respectively.
6. the super resolution ratio reconstruction method according to claim 2 based on multipath depth convolutional neural networks, feature
It is：Feature is mapped and is merged using multichannel gauge structure in the second part of network,
Its formula is：
F2(X)=max (0, W2*F1(X)+B2),
F3(X)=max (0, W3*F2(X)+B3),
F21(X)=max (0, W2*F1(X)+B2),
F3 *(X)=F3(X)+F21(X),
Wherein, F2(X), F3(X) it is second and the output of third convolutional layer, F21(X) it is the output of convolutional layer in parallel, parallel connection volume
The W and B of lamination and second layer parameter sharing, then with the mode being added by F21(X) and F3 *(X) it merges.In specific such as attached drawing 2
Shown, 9 similar structures form final multipath convolutional neural networks, and the final output of network configuration second part is
7. the super resolution ratio reconstruction method according to claim 2 based on multipath depth convolutional neural networks, feature
It is：The Part III of network try again convolution carry out image reconstruction, be similar to conventional method average treatment, formula is such as
Under：
F20(X)=W20*F19(X)+B20,
Wherein, F20(X) for the output of the 20th convolutional layer, the residual image that dimension 1, i.e. e-learning arrive, according to global residual
Poor learning structure, network inputs X, the high-definition picture that network finally reconstructs are Y, then：
Y=X+F20(X),
Assuming that training set isThat is N is to the high-low resolution images pair of S kind amplification factors, for every a pair of of sample
This, the high-resolution residual error image to be learnt is represented byThe network Nonlinear Mapping to be learnt is F (X),
If F (X) parameter is θ={ WK,bk, so the loss function of network is represented by：
Using above-mentioned convolutional neural networks, learnt using multichannel gauge structure and global residual error, to reconstruct final high-resolution
Rate image.
8. the super resolution ratio reconstruction method according to claim 1 based on multipath depth convolutional neural networks, feature
It is：In step (4), ReLU activation primitives, that is, Unit (ReLU, max (0, x)).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810325131.5A CN108537733B (en) | 2018-04-11 | 2018-04-11 | Super-resolution reconstruction method based on multi-path deep convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810325131.5A CN108537733B (en) | 2018-04-11 | 2018-04-11 | Super-resolution reconstruction method based on multi-path deep convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108537733A true CN108537733A (en) | 2018-09-14 |
| CN108537733B CN108537733B (en) | 2022-03-11 |
Family
ID=63480902
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810325131.5A Active CN108537733B (en) | 2018-04-11 | 2018-04-11 | Super-resolution reconstruction method based on multi-path deep convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108537733B (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109345476A (en) * | 2018-09-19 | 2019-02-15 | 南昌工程学院 | High spectrum image super resolution ratio reconstruction method and device based on depth residual error network |
| CN109389556A (en) * | 2018-09-21 | 2019-02-26 | 五邑大学 | The multiple dimensioned empty convolutional neural networks ultra-resolution ratio reconstructing method of one kind and device |
| CN109727195A (en) * | 2018-12-25 | 2019-05-07 | 成都元点智库科技有限公司 | A kind of image super-resolution reconstructing method |
| CN109741260A (en) * | 2018-12-29 | 2019-05-10 | 天津大学 | A kind of efficient super-resolution method based on depth back projection network |
| CN109785236A (en) * | 2019-01-21 | 2019-05-21 | 中国科学院宁波材料技术与工程研究所 | A kind of image super-resolution method based on super-pixel and convolutional neural networks |
| CN109903226A (en) * | 2019-01-30 | 2019-06-18 | 天津城建大学 | Image super-resolution rebuilding method based on symmetrical residual error convolutional neural networks |
| CN110288524A (en) * | 2019-05-09 | 2019-09-27 | 广东启迪图卫科技股份有限公司 | Deep learning super-resolution method based on enhanced up-sampling and discrimination syncretizing mechanism |
| CN110826467A (en) * | 2019-11-22 | 2020-02-21 | 中南大学湘雅三医院 | Electron microscope image reconstruction system and method |
| CN111524072A (en) * | 2020-04-26 | 2020-08-11 | 重庆紫光华山智安科技有限公司 | Super-resolution reconstruction network training method and device, electronic equipment and storage medium |
| CN111754438A (en) * | 2020-06-24 | 2020-10-09 | 安徽理工大学 | Underwater image restoration model based on multi-branch gating fusion and restoration method thereof |
| CN111768340A (en) * | 2020-06-30 | 2020-10-13 | 苏州大学 | Super-resolution image reconstruction method and system based on dense multi-path network |
| CN111861877A (en) * | 2019-04-25 | 2020-10-30 | 华为技术有限公司 | Method and apparatus for video hyper-resolution |
| CN112070667A (en) * | 2020-08-14 | 2020-12-11 | 西安理工大学 | Multi-scale feature fusion video super-resolution reconstruction method |
| CN112116526A (en) * | 2019-06-19 | 2020-12-22 | 中国石油化工股份有限公司 | Torch smoke image super-resolution method based on deep convolutional neural network |
| CN112446835A (en) * | 2019-09-04 | 2021-03-05 | 华为技术有限公司 | Image recovery method, image recovery network training method, device and storage medium |
| CN112767260A (en) * | 2020-12-30 | 2021-05-07 | 上海联影智能医疗科技有限公司 | Image quality improving method and device, computer equipment and storage medium |
| CN113160044A (en) * | 2020-01-23 | 2021-07-23 | 百度在线网络技术（北京）有限公司 | Depth image super-resolution method, training device, equipment and medium |
| CN113343762A (en) * | 2021-05-07 | 2021-09-03 | 北京邮电大学 | Human body posture estimation grouping model training method, posture estimation method and device |
| CN113409195A (en) * | 2021-07-06 | 2021-09-17 | 中国标准化研究院 | Image super-resolution reconstruction method based on improved deep convolutional neural network |
| CN114041161A (en) * | 2019-06-06 | 2022-02-11 | 三星电子株式会社 | Method and device for training neural network model for enhancing image details |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103544705A (en) * | 2013-10-25 | 2014-01-29 | 华南理工大学 | Image quality testing method based on deep convolutional neural network |
| WO2016019484A1 (en) * | 2014-08-08 | 2016-02-11 | Xiaoou Tang | An apparatus and a method for providing super-resolution of a low-resolution image |
| CN107103585A (en) * | 2017-04-28 | 2017-08-29 | 广东工业大学 | A kind of image super-resolution system |
| CN107103285A (en) * | 2017-03-24 | 2017-08-29 | 深圳市未来媒体技术研究院 | Face depth prediction approach based on convolutional neural networks |
| CN107240066A (en) * | 2017-04-28 | 2017-10-10 | 天津大学 | Image super-resolution rebuilding algorithm based on shallow-layer and deep layer convolutional neural networks |
| US20180075581A1 (en) * | 2016-09-15 | 2018-03-15 | Twitter, Inc. | Super resolution using a generative adversarial network |
-
2018
- 2018-04-11 CN CN201810325131.5A patent/CN108537733B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103544705A (en) * | 2013-10-25 | 2014-01-29 | 华南理工大学 | Image quality testing method based on deep convolutional neural network |
| WO2016019484A1 (en) * | 2014-08-08 | 2016-02-11 | Xiaoou Tang | An apparatus and a method for providing super-resolution of a low-resolution image |
| US20180075581A1 (en) * | 2016-09-15 | 2018-03-15 | Twitter, Inc. | Super resolution using a generative adversarial network |
| CN107103285A (en) * | 2017-03-24 | 2017-08-29 | 深圳市未来媒体技术研究院 | Face depth prediction approach based on convolutional neural networks |
| CN107103585A (en) * | 2017-04-28 | 2017-08-29 | 广东工业大学 | A kind of image super-resolution system |
| CN107240066A (en) * | 2017-04-28 | 2017-10-10 | 天津大学 | Image super-resolution rebuilding algorithm based on shallow-layer and deep layer convolutional neural networks |
Non-Patent Citations (3)
| Title |
|---|
| CHAO DONG 等: "Learning a Deep Convolutional Network", 《ECCV 2014: COMPUTER VISION》 * |
| JIWON KIM 等: "Accurate Image Super-Resolution Using Very Deep Convolutional Networks", 《IEEE》 * |
| 龙超: "图像超分辨率重建算法综述", 《科技视界》 * |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109345476A (en) * | 2018-09-19 | 2019-02-15 | 南昌工程学院 | High spectrum image super resolution ratio reconstruction method and device based on depth residual error network |
| CN109389556A (en) * | 2018-09-21 | 2019-02-26 | 五邑大学 | The multiple dimensioned empty convolutional neural networks ultra-resolution ratio reconstructing method of one kind and device |
| CN109727195A (en) * | 2018-12-25 | 2019-05-07 | 成都元点智库科技有限公司 | A kind of image super-resolution reconstructing method |
| CN109741260A (en) * | 2018-12-29 | 2019-05-10 | 天津大学 | A kind of efficient super-resolution method based on depth back projection network |
| CN109741260B (en) * | 2018-12-29 | 2023-05-12 | 天津大学 | Efficient super-resolution method based on depth back projection network |
| CN109785236A (en) * | 2019-01-21 | 2019-05-21 | 中国科学院宁波材料技术与工程研究所 | A kind of image super-resolution method based on super-pixel and convolutional neural networks |
| CN109785236B (en) * | 2019-01-21 | 2022-11-08 | 中国科学院宁波材料技术与工程研究所 | Image super-resolution method based on super-pixel and convolutional neural network |
| CN109903226B (en) * | 2019-01-30 | 2023-08-15 | 天津城建大学 | Image super-resolution reconstruction method based on symmetric residual convolution neural network |
| CN109903226A (en) * | 2019-01-30 | 2019-06-18 | 天津城建大学 | Image super-resolution rebuilding method based on symmetrical residual error convolutional neural networks |
| CN111861877A (en) * | 2019-04-25 | 2020-10-30 | 华为技术有限公司 | Method and apparatus for video hyper-resolution |
| CN110288524A (en) * | 2019-05-09 | 2019-09-27 | 广东启迪图卫科技股份有限公司 | Deep learning super-resolution method based on enhanced up-sampling and discrimination syncretizing mechanism |
| CN114041161A (en) * | 2019-06-06 | 2022-02-11 | 三星电子株式会社 | Method and device for training neural network model for enhancing image details |
| CN112116526B (en) * | 2019-06-19 | 2024-06-11 | 中国石油化工股份有限公司 | Super-resolution method of torch smoke image based on depth convolution neural network |
| CN112116526A (en) * | 2019-06-19 | 2020-12-22 | 中国石油化工股份有限公司 | Torch smoke image super-resolution method based on deep convolutional neural network |
| CN112446835A (en) * | 2019-09-04 | 2021-03-05 | 华为技术有限公司 | Image recovery method, image recovery network training method, device and storage medium |
| CN110826467A (en) * | 2019-11-22 | 2020-02-21 | 中南大学湘雅三医院 | Electron microscope image reconstruction system and method |
| CN110826467B (en) * | 2019-11-22 | 2023-09-29 | 中南大学湘雅三医院 | Electron microscope image reconstruction system and method thereof |
| CN113160044A (en) * | 2020-01-23 | 2021-07-23 | 百度在线网络技术（北京）有限公司 | Depth image super-resolution method, training device, equipment and medium |
| CN113160044B (en) * | 2020-01-23 | 2023-12-26 | 百度在线网络技术（北京）有限公司 | Depth image super-resolution method, training method and device, equipment and medium |
| CN111524072A (en) * | 2020-04-26 | 2020-08-11 | 重庆紫光华山智安科技有限公司 | Super-resolution reconstruction network training method and device, electronic equipment and storage medium |
| CN111754438A (en) * | 2020-06-24 | 2020-10-09 | 安徽理工大学 | Underwater image restoration model based on multi-branch gating fusion and restoration method thereof |
| CN111754438B (en) * | 2020-06-24 | 2021-04-27 | 安徽理工大学 | Underwater image restoration model based on multi-branch gating fusion and restoration method thereof |
| CN111768340B (en) * | 2020-06-30 | 2023-12-01 | 苏州大学 | Super-resolution image reconstruction method and system based on dense multipath network |
| CN111768340A (en) * | 2020-06-30 | 2020-10-13 | 苏州大学 | Super-resolution image reconstruction method and system based on dense multi-path network |
| CN112070667A (en) * | 2020-08-14 | 2020-12-11 | 西安理工大学 | Multi-scale feature fusion video super-resolution reconstruction method |
| CN112767260A (en) * | 2020-12-30 | 2021-05-07 | 上海联影智能医疗科技有限公司 | Image quality improving method and device, computer equipment and storage medium |
| CN113343762B (en) * | 2021-05-07 | 2022-03-29 | 北京邮电大学 | Human body posture estimation grouping model training method, posture estimation method and device |
| CN113343762A (en) * | 2021-05-07 | 2021-09-03 | 北京邮电大学 | Human body posture estimation grouping model training method, posture estimation method and device |
| CN113409195A (en) * | 2021-07-06 | 2021-09-17 | 中国标准化研究院 | Image super-resolution reconstruction method based on improved deep convolutional neural network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108537733B (en) | 2022-03-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108537733A (en) | Super resolution ratio reconstruction method based on multipath depth convolutional neural networks | |
| CN109903223B (en) | Image super-resolution method based on dense connection network and generation type countermeasure network | |
| Fang et al. | Soft-edge assisted network for single image super-resolution | |
| CN110197468A (en) | A kind of single image Super-resolution Reconstruction algorithm based on multiple dimensioned residual error learning network | |
| CN112597985B (en) | Crowd counting method based on multi-scale feature fusion | |
| CN107578377A (en) | A kind of super-resolution image reconstruction method and system based on deep learning | |
| CN111062872A (en) | Image super-resolution reconstruction method and system based on edge detection | |
| CN109389556A (en) | The multiple dimensioned empty convolutional neural networks ultra-resolution ratio reconstructing method of one kind and device | |
| CN110232653A (en) | The quick light-duty intensive residual error network of super-resolution rebuilding | |
| CN110136060B (en) | Image super-resolution reconstruction method based on shallow dense connection network | |
| Cai et al. | TDPN: Texture and detail-preserving network for single image super-resolution | |
| CN116309070A (en) | Super-resolution reconstruction method and device for hyperspectral remote sensing image and computer equipment | |
| CN111833261A (en) | Image super-resolution restoration method for generating countermeasure network based on attention | |
| CN114898284B (en) | Crowd counting method based on feature pyramid local difference attention mechanism | |
| CN115331104A (en) | Crop planting information extraction method based on convolutional neural network | |
| CN112288630A (en) | Super-resolution image reconstruction method and system based on improved wide-depth neural network | |
| CN112767252A (en) | Image super-resolution reconstruction method based on convolutional neural network | |
| CN115424209A (en) | Crowd counting method based on spatial pyramid attention network | |
| Muqeet et al. | Hybrid residual attention network for single image super resolution | |
| CN110533591A (en) | Super resolution image reconstruction method based on codec structure | |
| Luo et al. | Bi-GANs-ST for perceptual image super-resolution | |
| CN110288529A (en) | A kind of single image super resolution ratio reconstruction method being locally synthesized network based on recurrence | |
| CN111539434B (en) | Infrared weak and small target detection method based on similarity | |
| CN117351414A (en) | Crowd density estimation method based on deep neural network | |
| CN117196940A (en) | Super-resolution reconstruction method suitable for real scene image based on convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |