CN105681628A - Convolution network arithmetic unit, reconfigurable convolution neural network processor and image de-noising method of reconfigurable convolution neural network processor - Google Patents

Abstract

The invention discloses a convolution network arithmetic unit, a reconfigurable convolution neural network processor and an image de-noising method of the reconfigurable convolution neural network processor. The reconfigurable convolution neural network processor comprises a bus interface, a preprocessing unit, a reconfigurable hardware controller, an SRAM, an SRAM control module, an input caching module, an output caching module, a memory, a data memory controller and the convolution network arithmetic unit. The processor is featured by few resources and rapid speed and can be applicable to common convolution neural network architecture. According to the unit, the processor and the method provided by the invention, convolution neural networks can be realized; the processing speed is rapid; transplanting is liable to be carried out; the resource consumption is little; an image or a video polluted by raindrops and dusts can be recovered; and raindrop and dust removing operations can be taken as preprocessing operations for providing help in follow-up image identification or classification.

Description

A kind of convolutional network arithmetic element and restructural convolutional neural networks processor and realize the method that image denoising processes

Technical field

The present invention relates to image processing field, particularly to a kind of convolutional network arithmetic element and restructural convolutional neural networks processor with realize the method that image denoising processes.

Background technology

The removal of image raindrop and dust is significant for image procossing application, particularly video monitoring and navigation system. It can be used for recovering by raindrop, the image of contamination by dust or video, is alternatively arranged as pre-treatment operation and offers help for follow-up image recognition or classification.

The current method removing picture noise mostly utilizes the modes such as gaussian filtering, medium filtering, bilateral filtering to complete, and these method treatment effects are bad, usually can not meet specific image and process the demand of application. It is thus desirable to the method for a better effects if removes picture noise, the method for convolutional neural networks becomes a good selection.

Current degree of depth learning network is mostly run on GPU, but GPU is expensive, and power consumption is high, is not appropriate for large-scale extensive use. And the speed of service is slow on CPU, run large-scale degree of depth learning network efficiency low, it is impossible to meet performance requirement.

Can be seen that current technology is for application convolutional neural networks, the problem being primarily present has: the problems such as processor area is big, and cost is high, and power consumption is big, poor performance. Therefore this is accomplished by that a low-power consumption, area be little, the restructural convolutional neural networks processor of high treating effect.

Summary of the invention

It is an object of the invention to provide a kind of convolutional network arithmetic element and restructural convolutional neural networks processor and realize the method that image denoising processes, hardware resource consumption is low, area is little, can recover by raindrop, the image of contamination by dust or video.

To achieve these goals, the present invention adopts the following technical scheme that

A kind of convolutional network arithmetic element, including 2 restructural separation convolution module, nonlinear activation function unit and multiply-accumulator unit;

First restructural separation convolution module is output as the input of nonlinear activation function unit, and nonlinear activation function unit is output as the input of multiply-accumulator unit, and multiply-accumulator unit is output as the input of second restructural separation convolution module;

Picture signal is input to first restructural with configuration network parameter signal and separates convolution module; First restructural separation convolution module completes 16 × 16 convolution algorithms; Nonlinear activation function unit completes the computing of activation primitive in convolutional neural networks; Multiply-accumulator unit completes the computing of the articulamentum in convolutional neural networks; Second restructural separation convolution module simultaneously completes 48 × 8 convolution algorithms;

Described multiply-accumulator unit includes some multiply-accumulator and some depositors; Wherein multiply-accumulator for calculate the output valve of last layer convolutional network and weight parameter product and; The result of last layer convolutional network is input in multiply-accumulator by depositor.

Further, described restructural separation convolution module includes 16 4 × the 4 one-dimensional convolution module of restructural and the first Parasites Fauna; Picture signal and convolutional network parameter are input to the one-dimensional convolution module of restructural by the first Parasites Fauna; Restructural separation convolution module can complete 1 16 × 16 convolution or simultaneously complete 48 × 8 convolution algorithms; 4 × 4 one-dimensional convolution module of restructural include 4 first selectors, 4 the one 2 input multipliers, the one 4 input summer, 4 the 2nd 2 input multiplier and the 2nd 4 input summers; The outfan of 4 first selectors connects the input of 4 the 1st corresponding input multipliers, the weight that another one input is neutral net of 4 the 1st input multipliers; The outfan of 4 the 1st input multipliers connects the input of the one 4 input summer; The output that input is the one 4 input summer of 4 the 2nd 2 input multipliers and the weight of neutral net; The input of the 2nd 4 input summer is the output of 4 the 2nd 2 input multipliers.

Further, described nonlinear activation function unit include QD generator and arithmetical unit group; Wherein the input of QD generator is the output of restructural separation convolution, arithmetical unit group the output that input is QD generator; QD generator is for producing the parameter needed for activation primitive; Arithmetical unit, group was for calculating the end value that activation primitive is final;

Described QD generator comprises first divider; Input signal is input to the first divider, the first divider output business Q and remainder D; Described arithmetical unit, group comprised shift register, 2 first adders and the second divider; Shift register output and be the input of 2 first adders; 2 first adders are output as the input of the second divider; Shift register, first adder and the second divider are sequentially connected with;

A kind of restructural convolutional neural networks processor, including the convolutional network arithmetic element according to any one of EBI, pretreatment unit, reconfigurable hardware controller, SRAM, SRAM control unit, input buffer module, output buffer module, memorizer, data storage controller and some claims 1 to 3; EBI connects pretreatment unit, data storage controller, reconfigurable hardware controller and input-buffer, output buffer memory; Memorizer connects data storage controller; Input-buffer connects reconfigurable hardware controller and SRAM control unit; Convolutional network arithmetic element connects input buffer module, output buffer module;

The input of described pretreatment unit is image or video signal; Complete the pre-treatment such as white balance, noise filtering operation;

Described input buffer module, output buffer module are respectively used to input and the output of buffer memory convolutional network arithmetic element;

Convolutional network computing module is configured by described reconfigurable hardware controller, controls its calculating process; What send in calculating process or when terminating that interrupt requests completes with external system is mutual;

Described SRAM control unit is for controlling the transmission of convolutional network weight parameter.

Further, including 512 convolutional network arithmetic elements, it is achieved the image denoising based on convolutional neural networks processes.

Further, described a kind of restructural convolutional neural networks processor realizes 3 layers of convolutional neural networks, for removing raindrop and the dust of attachment in image or video; Described convolutional neural networks ground floor is made up of 512 16 × 16 convolution, and the second layer is neutral net articulamentum, and third layer is made up of 512 8 × 8 convolution.

A kind of restructural convolutional neural networks processor realizes the method that image denoising processes, including:

In the process that image denoising processes, the random convolution number that reduces, the consumption of minimizing hardware resource, improves processing speed;

Or, 16 × 16 convolution algorithm unit and 8 × 8 convolution algorithm unit are respectively classified into the convolution mask of 16 and 44 × 4 by the process that image denoising processes, the convolution to each 4 × 4 adopts one-dimensional convolution.

Relative to prior art, the method have the advantages that convolutional network arithmetic element utilizes Reconfiguration Technologies, 16 × 16 convolution can be completed or simultaneously complete 48 × 8 convolution algorithms, improve hardware performance and motility. The present invention utilizes the method that the degree of depth learns, it is achieved that can removing the denoising of image raindrop and dust, treatment effect meets demand. The present invention is under the premise not affecting treatment effect, and the random template number reducing convolutional network, but also the method utilizing the one-dimensional convolution of piecemeal, hardware resource consumption greatly reduces, and processing speed is greatly improved. This processor can realize the convolutional neural networks of 3 layers, it is possible to provides feature for follow-up higher level image recognition, classification. Expensive relative to GPU, power consumption is high, and area is big. The CPU speed of service is slow, runs large-scale degree of depth learning network efficiency low. The present invention adopts Reconfiguration Technologies and the method for above-mentioned minimizing template number and the one-dimensional convolution of piecemeal, it is achieved restructural convolutional neural networks processor low in resources consumption, be prone to hardware and realize, it is possible to recover by raindrop, the image of contamination by dust or video.

Accompanying drawing explanation

Fig. 1 is the structural representation of convolutional network arithmetic element;

Fig. 2 is the structural representation of non-linear activation primitive unit;

Fig. 3 is the structural representation of the one 4 × 4th one-dimensional convolution module of restructural;

Fig. 4 is the structural representation of restructural separation convolution module;

Fig. 5 is the structural representation of restructural convolutional neural networks processor;

Detailed description of the invention

Illustrate with detailed description of the invention, the present invention to be elaborated below in conjunction with accompanying drawing.

With reference to Fig. 1, the convolutional network arithmetic element used in restructural convolutional neural networks processor in the present invention includes 2 restructural separation convolution module, nonlinear activation function unit and multiply-accumulator unit; First restructural separation convolution module is output as the input of nonlinear activation function unit, and nonlinear activation function unit is output as the input of multiply-accumulator unit, and multiply-accumulator unit is output as the input of second restructural separation convolution module;

Picture signal is input to first restructural with configuration network parameter signal and separates convolution module; First restructural separation convolution module completes 16 × 16 convolution algorithms; Nonlinear activation function unit completes the computing of activation primitive in convolutional neural networks; Multiply-accumulator unit completes the computing of the articulamentum in convolutional neural networks; Second restructural separation convolution module simultaneously completes 48 × 8 convolution algorithms;

Refer to shown in Fig. 2, nonlinear activation function unit include QD generator and arithmetical unit group; Wherein the input of QD generator is the output of restructural separation convolution, arithmetical unit group the output that input is QD generator; QD generator is for producing the parameter needed for activation primitive; Arithmetical unit, group was for calculating the result that activation primitive is final.

The activation primitive of neutral net of the present invention is hyperbolic tangent function

$\tanh \left(x\right)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$

By defining territory extension and Taylor series expansion, obtain

$\tanh (Qln2+D)=\frac{1+D-2^{-2Q}(1-D)}{1+D+2^{-2Q}(1-D)}$ Wherein | D | < ln2

QD generator comprises first divider, and input signal is input to the first divider, and the first divider, divided by definite value 0.69, exports business Q and remainder D; Described arithmetical unit, group comprised shift register, 2 first adders and the second divider; Shift register output and be the input of 2 first adders; 2 first adders are output as the input of the second divider; Shift register, first adder and the second divider are sequentially connected with;

Referring to shown in Fig. 3,4 × 4 one-dimensional convolution module of restructural include 4 first selector MUX, 4 the 1st input multipliers, the one 4 input summer, 4 the 2nd 2 input multipliers, the 2nd 4 input summer. Two inputs of first selector are picture signal and previous stage result; The outfan of 4 first selectors connects an input of 4 the 1st corresponding input multipliers, the weight that another one input is neutral net of 4 the 1st input multipliers; The outfan of 4 the 1st input multipliers connects the input of the one 4 input summer; The output that input is the one 4 input summer of 4 the 2nd 2 input multipliers and the weight of neutral net; The input of the 2nd 4 input summer is the output of 4 the 2nd 2 input multipliers.

Referring to shown in Fig. 4, restructural separation convolution module includes the first Parasites Fauna, 16 4 × 4 one-dimensional convolution module of restructural, 44 input first adders and 14 input second adder. Utilize Reconfiguration Technologies, restructural separation convolution module can complete 16 × 16 or simultaneously complete 48 × 8 convolution algorithms. Picture signal and configuration signal are input to the first Parasites Fauna. The input of the one 4 × 4th convolution 1 is 1-4 row picture signal, and the input of the one 4 × 4th convolution 5 is 5-8 row picture signal.

When convolution mask is 16 × 16, the output that input is the one 4 × 4th convolution 2 of the one 4 × 4th convolution 3, the output that input is the one 4 × 4th convolution 6 of the one 4 × 4th convolution 7, the output that input is the one 4 × 4th convolution 10 of the one 4 × 4th convolution 11, the output that input is the one 4 × 4th convolution 14 of the one 4 × 4th convolution 15. The input of the one 4 × 4th convolution 9 is 9-12 row picture signal, and the input of the one 4 × 4th convolution 13 is 13-16 row picture signal. Restructural separation convolution module is output as the result of second adder.

When convolution module is 8 × 8, the input of the one 4 × 4th convolution 3 is 1-4 row picture signal, the input of the one 4 × 4th convolution 7 is 1-4 row picture signal, and the input of the one 4 × 4th convolution 11 is 1-4 row picture signal, and the input of the one 4 × 4th convolution 15 is 1-4 row picture signal. The input of the one 4 × 4th convolution 9 is 1-4 row picture signal, and the input of the one 4 × 4th convolution 13 is 5-8 row picture signal. Restructural separation convolution module is output as the result of 4 first adders. One restructural separation convolution module can simultaneously complete 48 × 8 convolution algorithms.

Referring to shown in Fig. 5, one restructural convolutional neural networks processor of the present invention includes EBI, pretreatment unit, reconfigurable hardware controller, SRAM, SRAM control unit, input-buffer, output buffer memory, memorizer, data storage controller and some convolutional network arithmetic elements; EBI connects pretreatment unit, data storage controller, reconfigurable hardware controller and input-buffer, output buffer memory; Memorizer connects data storage controller; Input-buffer connects reconfigurable hardware controller and SRAM control unit; Convolutional network arithmetic element connects input buffer module, output buffer module.

The input of pretreatment unit is image or video signal; Complete the pre-treatment such as white balance, noise filtering operation; Input buffer module, output buffer memory are respectively used to input and the output of buffer memory convolutional network arithmetic element. Convolutional network arithmetic element is configured by reconfigurable hardware controller, controls its calculating process; What send in calculating process or when terminating that interrupt requests completes with external system is mutual; SRAM control unit is for controlling the transmission of convolutional network weight parameter.

In one convolutional neural networks realizing removal image raindrop and dust, including 512 convolutional network arithmetic elements. In order to reduce resource, improve processing speed, the present invention have employed following two method in implementing process: (1) reduces the method for convolution number at random: reduce the number of convolutional network arithmetic element under the premise not affecting treatment effect, reduce the consumption of hardware resource, improve processing speed; (2) method of the one-dimensional convolution of piecemeal: the convolution mask by 16 × 16 and 8 × 8 is respectively classified into the convolution mask of 16 and 44 × 4, the convolution to each 4 × 4 adopts the mode of one-dimensional convolution.

With reference to Fig. 5, restructural 16 × 16 convolution algorithm unit include 16 4 × 4 one-dimensional convolution module of restructural (1,2,3 ..., 16), row storing module and depositor; The input of row storing module is image or video signal, the output that input is row storing module of Parasites Fauna, the output that input is Parasites Fauna of 4 × 4 one-dimensional convolution module of restructural; Row storing module is used for preserving image; The view data of serial input deposited by depositor for preserving row, and view data is input to 4 × 4 one-dimensional convolution module of restructural.

Restructural 8 × 8 convolution algorithm unit includes 4 the one-dimensional convolution module of 4 × 4 restructurals (1,2,3,4), row storing module and depositor; The input of row storing module is the output of multiply-accumulator, the output that input is row storing module of Parasites Fauna, the output that input is Parasites Fauna of 4 × 4 one-dimensional convolution module of restructural.

Claims (7)

1. a convolutional network arithmetic element, it is characterised in that: include 2 restructural separation convolution module, nonlinear activation function unit and multiply-accumulator unit;

First restructural separation convolution module is output as the input of nonlinear activation function unit, and nonlinear activation function unit is output as the input of multiply-accumulator unit, and multiply-accumulator unit is output as the input of second restructural separation convolution module;

Picture signal is input to first restructural with configuration network parameter signal and separates convolution module; First restructural separation convolution module completes 16 × 16 convolution algorithms; Nonlinear activation function unit completes the computing of activation primitive in convolutional neural networks; Multiply-accumulator unit completes the computing of the articulamentum in convolutional neural networks; Second restructural separation convolution module simultaneously completes 48 × 8 convolution algorithms;

Described multiply-accumulator unit includes some multiply-accumulator and some depositors; Wherein multiply-accumulator for calculate the output valve of last layer convolutional network and weight parameter product and; The result of last layer convolutional network is input in multiply-accumulator by depositor.

2. a kind of convolutional network arithmetic element according to claim 1, it is characterised in that: described restructural separation convolution module includes 16 4 × the 4 one-dimensional convolution module of restructural and the first Parasites Fauna; First Parasites Fauna is input to the one-dimensional convolution module of restructural for picture signal or previous stage being exported with convolutional network parameter; Restructural separation convolution module has been used for 1 16 × 16 convolution or has simultaneously completed 48 × 8 convolution algorithms;

4 × 4 one-dimensional convolution module of restructural include 4 first selectors, 4 the one 2 input multipliers, the one 4 input summer, 4 the 2nd 2 input multiplier and the 2nd 4 input summers; The outfan of 4 first selectors connects the input of 4 the 1st corresponding input multipliers, the weight that another one input is neutral net of 4 the 1st input multipliers; The outfan of 4 the 1st input multipliers connects the input of the one 4 input summer; The output that input is the one 4 input summer of 4 the 2nd 2 input multipliers and the weight of neutral net; The input of the 2nd 4 input summer is the output of 4 the 2nd 2 input multipliers.

3. a kind of convolutional network arithmetic element according to claim 1, it is characterised in that: described nonlinear activation function unit include QD generator and arithmetical unit group; Wherein the input of QD generator is the output of restructural separation convolution, arithmetical unit group the output that input is QD generator; QD generator is for producing the parameter needed for activation primitive; Arithmetical unit, group was for calculating the end value that activation primitive is final;

Described QD generator comprises first divider; Input signal is input to the first divider, the first divider output business Q and remainder D; Described arithmetical unit, group comprised shift register, 2 first adders and the second divider; Shift register output and be the input of 2 first adders; 2 first adders are output as the input of the second divider; Shift register, first adder and the second divider are sequentially connected with.

4. a restructural convolutional neural networks processor, it is characterised in that: include the convolutional network arithmetic element according to any one of EBI, pretreatment unit, reconfigurable hardware controller, SRAM, SRAM control unit, input buffer module, output buffer module, memorizer, data storage controller and some claims 1 to 3; EBI connects pretreatment unit, data storage controller, reconfigurable hardware controller and input-buffer, output buffer memory; Memorizer connects data storage controller; Input-buffer connects reconfigurable hardware controller and SRAM control unit; Convolutional network arithmetic element connects input buffer module, output buffer module;

The input of described pretreatment unit is image or video signal; Complete the pre-treatment such as white balance, noise filtering operation;

Described input buffer module, output buffer module are respectively used to input and the output of buffer memory convolutional network arithmetic element;

Convolutional network computing module is configured by described reconfigurable hardware controller, controls its calculating process; What send in calculating process or when terminating that interrupt requests completes with external system is mutual;

Described SRAM control unit is for controlling the transmission of convolutional network weight parameter.

5. a kind of restructural convolutional neural networks processor according to claim 4, it is characterised in that: include 512 convolutional network arithmetic elements, it is achieved the image denoising based on convolutional neural networks processes.

6. a kind of restructural convolutional neural networks processor according to claim 4, it is characterised in that: described a kind of restructural convolutional neural networks processor realizes 3 layers of convolutional neural networks, for removing raindrop and the dust of attachment in image or video; Described convolutional neural networks ground floor is made up of 512 16 × 16 convolution, and the second layer is neutral net articulamentum, and third layer is made up of 512 8 × 8 convolution.

7. a kind of restructural convolutional neural networks processor according to any one of claim 4 to 6 realizes the method that image denoising processes, it is characterised in that: including:

In the process that image denoising processes, the random convolution number that reduces, the consumption of minimizing hardware resource, improves processing speed;

Or, 16 × 16 convolution algorithm unit and 8 × 8 convolution algorithm unit are respectively classified into the convolution mask of 16 and 44 × 4 by the process that image denoising processes, the convolution to each 4 × 4 adopts one-dimensional convolution.