JP6964857B2 - Image recognition device, image recognition method, computer program, and product monitoring system - Google Patents

Description

The present invention relates to an image recognition device, an image recognition method, a computer program, and a product monitoring system. Specifically, the present invention relates to an image processing technique for improving the accuracy of image recognition using a hierarchical convolutional neural network.

In recent years, the performance of image recognition by deep learning has been dramatically improved. Deep learning is a general term for machine learning using a multi-layered hierarchical neural network. As the multi-layered hierarchical neural network, for example, a convolutional neural network (hereinafter, also referred to as “CNN”) is used.

CNN has a multi-layered laminated structure in which a convolution layer and a pooling layer in a local region are repeated, and it is said that the laminated structure improves the image recognition performance.
As shown in Non-Patent Documents 1 and 2, it has already been performed to recognize a class of an object by deep learning using a convolutional neural network.

"ImageNet Classification with Deep Convolutional Neural Networks" A. krizhevsky et al. In: Proc. Adv. Neural Inf. Proc. Syst. (NIPS), 2012, PP.1097-1105 "Very Deep Convolutional Networks for Large-Scale Image Recognition" K.Symonyan et al. ArXiv: 1409.1556v6 [cs.CV] 10 Apr. 2015

In image recognition using a convolutional neural network, the pixel value (RGB value) of the original image is input to the network as it is without preprocessing, or principal component analysis (Principle Component Analysis) is performed on the pixel value. It is said.
In this way, conventionally, the pixel values (raw data) of the original image are used as they are, or only preprocessing for extracting a single feature factor from the original image is performed. Therefore, in order to improve the recognition accuracy, It is necessary to collect a large number of sample images and polymorphic sample images of the same class.

In particular, since a sample image including a rotated or inverted object is rare, there is a problem that the recognition accuracy of the rotated or inverted object cannot be improved so much even if CNN learning is repeated using the sample image in a normal orientation. ..
In view of the conventional problems, an object of the present invention is to improve the accuracy of image recognition by a hierarchical neural network.

(1) The image recognition device of the present invention is a data generation unit that generates input data by performing predetermined data processing on an original image, and a hierarchical neural network that recognizes the types of objects included in the generated input data. An image processing unit including an image processing unit, and when the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network. If the original image is the target image for recognition, the process of outputting the recognition result of the network is performed, and the data processing performed by the data generation unit is one of rotation and inversion with respect to the original image. It is a process of imparting at least one immutability.

According to the image recognition device of the present invention, the data processing performed by the data generation unit comprises a process of imparting at least one invariance of rotation and inversion to the original image.
Therefore, when learning with the same number of sample images, the accuracy of image recognition by the hierarchical neural network can be improved as compared with the case where the original image is used as the input data as it is without performing the above data processing. Yes (see Figure 10). In addition, even a rotated or inverted object can be recognized accurately.

(2) In the image recognition device of the present invention, specifically, the data processing performed by the data generation unit includes the first processing and the second processing defined below.
First process: Process of generating an image filter having at least one invariance of rotation and inversion with respect to the original image Second process: Process of convolving the image filter generated in the first process into the original image

(3) More specifically, the image filter opens a color vector of an arbitrary pixel point defined by polar coordinates with a predetermined point of the original image as an origin at an arbitrary angle starting from the pixel point. It consists of elements contained in a plurality of color vectors divided in a direction.
The reason is that the plurality of color vectors obtained by dividing the color vectors of the pixel points in the polar coordinate display as described above remain equivalent even when rotated at an arbitrary angle around the origin, and are rotated and inverted with respect to the original image. This is because it has at least one invariance of.

(4) More specifically, the image filter sets a color vector of an arbitrary pixel point defined by polar coordinates with a predetermined point of the original image as the origin in the radial direction and the tangential direction starting from the pixel point. It preferably consists of elements contained in the two divided color vectors.
The reason is that if the color vector is decomposed in two directions, the radial direction and the tangential direction, the number of calculation parameters can be minimized and the processing load of the data generation unit can be reduced.

(5) In the image recognition device of the present invention, specifically, the hierarchical neural network comprises a convolutional neural network.
The reason is that the convolutional neural network can realize high performance for image recognition even in the hierarchical neural network.

(6) In the image recognition device of the present invention, the object whose type is recognized may be at least one of handwritten characters, humans, animals, plants, and products.
The reason is that the data processing that imparts at least one invariance of rotation and inversion to the original image, which is a feature of the present invention, applies to various objects regardless of the attributes of the objects contained in the original image. This is because it is considered applicable. Therefore, the scope of application of the image recognition device of the present invention is not limited to the recognition of a specific object.

(7) The computer program of the present invention relates to a computer program for operating a computer as the image recognition device according to any one of (1) to (6) above.
Therefore, the computer program of the present invention has the same effect as the image recognition device according to any one of (1) to (6) above.

(8) The image recognition method of the present invention relates to an image recognition method executed by the image recognition device according to any one of (1) to (6) above.
Therefore, the image recognition method of the present invention exerts the same effect as that of the image recognition device in any of the above-mentioned (1) to (6).

(9) In the product monitoring system of the present invention, a photographing device for photographing a plurality of products, a robot device for taking out one of the photographed products to the outside, and the product to be taken out are used as the robot device. A product monitoring system including a control device for instructing, wherein the control device includes the image recognition device according to any one of (1) to (6) above, and the image recognition device is a defective product. Instruct the robot device to take out the recognized product.

According to the product monitoring system of the present invention, the image recognition device instructs the robot device to take out the product recognized as a defective product. Therefore, by learning the image recognition device from an appropriate number of sample images, the defective product can be taken out. Can be done automatically and accurately.

The present invention can be realized not only as a system and an apparatus having the above-mentioned characteristic configuration, but also as a computer program for causing a computer to execute such a characteristic configuration.
Further, the present invention described above can be realized as one or more semiconductor integrated circuits that realize a part or all of a system and an apparatus.

According to the present invention, the accuracy of image recognition by a hierarchical neural network can be improved.

It is a block diagram of the image identification apparatus which concerns on embodiment of this invention. It is a schematic block diagram of CNN included in a CNN processing part. It is a conceptual diagram of the processing content of a convolution layer. It is a conceptual diagram of the structure of the receptive field. It is explanatory drawing of the 1st process by a data generation part. It is explanatory drawing of the 2nd process by a data generation part. It is explanatory drawing of the rotation point and the like which rotated arbitrary pixel point by a predetermined angle. It is a structural drawing of the deep CNN constructed in the CNN processing part. It is a figure which shows an example of the handwritten character used in the simulation experiment. It is a graph which shows the test result of the recognition accuracy for each character class. It is an overall block diagram of the product monitoring system which concerns on embodiment of this invention.

Hereinafter, the details of the embodiment of the present invention will be described with reference to the drawings. In addition, at least a part of the embodiments described below may be arbitrarily combined.

[Overall configuration of image processing device]
FIG. 1 is a block diagram of an image recognition device 10 according to an embodiment of the present invention.
As shown in FIG. 1, the image recognition device 10 of the present embodiment includes, for example, an arithmetic processing unit 1 and an image processing unit 2 mounted on a PC (Personal Computer) (not shown).

The arithmetic processing unit 1 includes a CPU (Central Processing Unit). The number of CPUs in the arithmetic processing unit 1 may be one or a plurality, and may include integrated circuits such as FPGA (Field-Programmable Gate Array) and ASIC (Application Specific Integrated Circuit).
The arithmetic processing unit 1 includes a RAM (Random Access Memory). The RAM is composed of memory elements such as SRAM (Static RAM) or DRAM (Dynamic RAM), and temporarily stores a computer program executed by a CPU or the like and data necessary for the execution thereof.

The image processing unit 2 includes a GPU (Graphics Processing Unit). The number of GPUs in the image processing unit 2 may be one or a plurality, and may include integrated circuits such as FPGA and ASIC.
The image processing unit 2 includes a RAM. The RAM is composed of a memory element such as an SRAM or a DRAM, and temporarily stores a computer program executed by the GPU or the like and data necessary for the execution thereof.

The arithmetic processing unit 1 includes a data generation unit 3 that generates an input image to the CNN processing unit 4 as a functional unit realized by the CPU executing a computer program for arithmetic processing recorded in the memory.
The data generation unit 3 performs the following first and second processes on the labeled sample image 7 or the photographed image to be identified (hereinafter, also referred to as “target image”) 8 to form a CNN processing unit. An input image for 4 (hereinafter, also referred to as “input data”) is generated.

First process: A process of generating an image filter 9 having rotation and inversion invariance with respect to the images 7 and 8 from the sample image 7 or the target image 8 (see FIG. 5).
Second process: A process of convolving the image filter 9 generated in the first process with respect to the sample image 7 or the target image 8 (see FIG. 6).

Hereinafter, the generic term of the "sample image" and the "target image" input to the data generation unit 3 is also referred to as an "original image". The original image refers only to the object area image to be processed (trained or recognized).
The data generation unit 3 inputs the input data obtained by performing the first and second processing on the original images 7 and 8 to the CNN processing unit 4 in the image processing unit 2 in the subsequent stage.

The image processing unit 2 includes a CNN processing unit 4, a learning unit 5, and a recognition unit 6 as functional units realized by the GPU executing a computer program for image processing recorded in the memory.
The CNN processing unit 4 recognizes the type of object included in the input data (for example, recognizes the type of character included in the input image), and the recognition result (specifically, the probability for each classification class, etc.) ) Is input to the learning unit 5 or the recognition unit 6.

Specifically, when training the CNN using the labeled sample image 7, the CNN processing unit 4 specifies the classification class of the sample image 7 and inputs the specified classification class to the learning unit 5.
On the other hand, when the trained CNN processing unit 4 is made to specify the classification class of the target image 8, that is, when the image processing unit 2 operates as a classifier, the CNN processing unit 4 recognizes the specified classification class. Enter in 6.

The learning unit 5 updates the parameters (weights and biases) held by the CNN processing unit 4 based on the input classification class, and stores the updated parameters in the CNN processing unit 4.
The recognition unit 6 outputs the recognition result based on the input classification class. Specifically, the classification class with the highest probability input from the CNN processing unit 4 is output as the classification class of the target image 8. The recognition result output by the recognition unit 6 is displayed on the display of the PC or the like to notify the operator of the PC.

[Processing content of CNN processing unit]
(CNN configuration example)
FIG. 2 is a schematic configuration diagram of a CNN included in the CNN processing unit 4.
As shown in FIG. 2, the CNN constructed in the CNN processing unit 4 has three arithmetic processing layers: a convolution layer (also referred to as a “downsampling layer”) C1 and C2, a pooling layer P1 and P2, and a fully connected layer F. And the final layer E, which is the output layer of the CNN.

The pooling layers P1 and P2 are arranged after the convolution layers C1 and C2, and the fully connected layer F is arranged after the last pooling layer P2. The final layer E of the CNN includes the same number of final nodes (10 in FIG. 2) as the preset number of classification classes.
FIG. 2 illustrates a case where there are two convolution layers C1 and C2 and corresponding pooling layers P1 and P2. However, the number of the convolution layer and the pooling layer may be three or more. Further, at least one fully connected layer F is arranged.

The j-th node in a certain layer C1, P1, C2, P2 _{receives inputs x i} (i = 1, 2, ... m) from m nodes in the immediately preceding layer, respectively, and biases them to the weighted sum. Calculate the intermediate variable u _{j by adding.} That is, the intermediate variable u _j is calculated by the following equation. Note that in the following _{equation, w ij} is the weight, _{b j} is the bias.

_{The response y j in which the intermediate variable u j} is applied to the activation function a (・), which is a non-linear function, that is, y _j = a (u _j ) becomes the output of the node of this layer, and this output is input to the next layer. Will be done.
For the activation function a, a "sigmoid function" or a (x _j ) = max (x _j , 0) is used. In particular, the latter activation function is called "ReLU (Rectified Linear Unit)". ReLU is often used in recent years because it contributes to good convergence and improvement of learning speed.

In the vicinity of the output layer of the CNN, one or more fully connected layers F in which all the nodes between adjacent layers are connected are arranged. The final layer E, which gives the output of the CNN, is designed in the same way as a normal neural network.
When the purpose is to classify the input image, the same number of nodes as the number of classification classes are arranged in the final layer E, and the "softmax function" is used for the activation function a of the final layer E.

Specifically, the following equation is calculated based on the inputs u _j (j = 1, 2, ... n) to the n nodes. At the time of recognition, the index j = argmax _j p _{j of the node in which p j} has the maximum value is selected as the estimation class.

(Processing content of convolution layer)
FIG. 3 is a conceptual diagram of the processing contents of the convolution layers C1 and C2.
As shown in FIG. 3, the inputs of the convolution layers C1 and C2 are in the form of N pixels (N channels) having a vertically long size of S × S pixels.
Hereinafter, an image of this format will be referred to as S × S × N. Further, the input of S × S × N is described as x _ijk (where (i, j, k) ∈ [0, S-1], [0, S-1], [1, N]).

In CNN, the number of channels of the first input layer (convolution layer C1) is N = 1 if the input image is grayscale, and N = 3 (3 channels of RGB) if the input image is color.
In the convolution layers C1 and C2, the calculation of convolving a filter (also referred to as "kernel") _{in the input x ijk is executed.}

This calculation includes filter convolution in general image processing, for example, a process of two-dimensionally convolving a small-sized image into an input image to blur the image (Gaussian filter), and a process of emphasizing edges (sharpening filter). ) Is basically the same process.
Specifically, a two-dimensional filter having an L × L size is convoluted into pixels having _{an input x ijk} size S × S of each channel k (k = 1 to N), and the result is obtained from all channels k = 1 to N. Add over. The calculation result is in the form of a 1-channel image _uij.

If the filter is _{defined as w ijk} (where (i, j, k) ∈ [1, L-1], [1, L-1], [1, N]), u _ij is calculated by the following equation. ..

However, _Pij is a square region of size L × L pixels having pixels (i, j) in the image as vertices. That is, _Pij is defined by the following equation.

b _k is a bias. In this embodiment, the bias is common to all output nodes for each channel. That is, b _ijk = b _k .
The filter may be applied at intervals of multiple pixels instead of all pixels. That is, for a given number of pixels _s, define a _{P ij} as _{follows, w p-i, q-} j, a _{k w p-si, q-} sj, to calculate the _{u ij} replaced with _k May be good. This pixel spacing s is called a "slide".

_{The u ij} calculated as described above is then passed through the activation function a (.) To become _{the output y ij of the convolution layers C1 and C2.} That is, y _ij = a (u _ij ).
As a result, for one filter w _{ijk , the output y ij} for one channel of S × S having the same vertical and horizontal sizes as the input x _ijk can be obtained.

If N'similar filters are prepared and the above calculations are performed independently for each, the output of S × S for N'channels, that is, the output of S × S × _N'size y ijk (however, (i, j, k) ∈ [1, S-1], [1, S-1], [1, N']) is obtained.
The output y _{ijk for this N'channel} becomes the input x ijk to the next layer. FIG. 3 shows the calculation contents for one of the N'filters.

The above calculation can be expressed as a single-layer network in which interlayer nodes are connected in a special form, as shown in FIG. 4, for example. FIG. 4 is a conceptual diagram of the structure of the receptive field. In the figure on the left, the receptive field is represented by a rectangle, and in the figure on the right, the receptive field is represented by a node.
Specifically, each node in the upper layer is connected to a part of each node in the lower layer (this is called a "local receptive field"). In addition, the join weight is common among each node (this is called "weight sharing").

(Processing content of pooling layer)
As shown in FIG. 2, the pooling layers P1 and P2 exist in pairs with the convolution layers C1 and C2. Therefore, the outputs of the convolution layers C1 and C2 are the inputs to the pooling layers P1 and P2, and the inputs of the pooling layers P1 and P2 are in the form of S × S × N.
The purpose of the pooling layers P1 and P2 is to discard some information about the position of the image where the filter response was strong, and to realize the invariance of the response to a slight change in the characteristics.

The nodes (i, j) of the pooling layers P1 and P2 have local receptive fields _{Pi and j} in the input side layer, similarly to the convolutional layers C1 and C2. Node pooling layer P1, P2 (i, j) is to aggregate local receptive field _{P i,} the internal nodes of the _{j (p, q) ∈P i} , the output _{y p} of the _{j, q} to one.
_{The sizes of the local receptive fields Pi and j} of the pooling layers P1 and P2 are set independently of those of the convolutional layers C1 and C2 (filter size).

When the input is a plurality of channels, the above processing is performed for each channel. That is, the number of output channels of the convolution layers C1 and C2 and the pooling layers P1 and P2 are the same.
Pooling is performed by thinning out in the vertical and horizontal directions (i, j) of the image. That is, two or more strides are set. For example, when s = 2, vertical and horizontal size of the output becomes half the vertical and horizontal size of the input, the number of output nodes of the pooling layer becomes 1 / s ² times the number of input nodes.

There are "average pooling" and "maximum pooling" as a method of collecting and aggregating the inputs from the nodes inside the receptive fields _{Pi and j.}
The average pooling is a method of outputting the average value _{of the input x pqk} from the nodes belonging to _{Pi and j as shown in the following equation.}

Maximum pooling is a method of outputting the maximum value _{of the input x pqk} from the nodes belonging to _{Pi and j as shown in the following equation.} Average pooling was predominant in CNN's early studies, but maximum pooling is now commonly adopted.

Unlike the convolution layers C1 and C2, the pooling layers P1 and P2 do not have weights that change due to learning, and the activation function is not applied.
In the CNN of the present embodiment, either the average pooling or the maximum pooling may be adopted, but in the implementation example of the CNN shown in FIG. 7, the maximum pooling is adopted.

[Processing content of the learning department]
In CNN training, "supervised learning" is the basis. Also in this embodiment, the learning unit 5 executes supervised learning.
Specifically, the learning unit 5 is executed by minimizing the classification error of each sample image for a set including a large number of labeled sample images as learning data. This process will be described below.

_{Each node of the final layer E of the CNN processing unit 4 outputs the probability pj} (j = 1, 2, ... n) for the corresponding class by normalization by the softmax function ([Equation 2] described above). .. This probability p _j is input to the learning unit 5.
Learning unit 5, so as to minimize the classification error calculated from the input probability p _j, it updates the parameters such weight set in the CNN processing unit 4.

Specifically, the learning unit 5 has ideal outputs d1, d2, ... dn (label) for the input sample and output p1. p2. The dissociation of pn is calculated by the cross entropy C of the following equation. This cross entropy C is the classification error.

The target outputs d1, d2, ... dn are set so that d _{j = 1 only in the correct answer class j and d k} = 0 in all other k (≠ j).
Learning unit 5, as in the above cross-entropy C decreases, the bias b _k coefficients w _ijk each node of the filter of each convolution layers C1, _C2, and total binding layer disposed on the output layer side of the CNN Adjust the weight and bias of F.

A stochastic gradient descent method is used to minimize the classification error C. Learning section 5, the error gradient related weights and biases the (∂C / ∂w _ij), calculated by backpropagation (BP method). The calculation method by the BP method is the same as in the case of a normal neural network.
However, in the back propagation when the CNN processing unit 4 adopts the maximum pooling, the value of which node in the pooling region is selected at the time of forward propagation for the training sample is memorized, and only that node is combined at the time of back propagation ( Combine with weight 1).

The evaluation of the classification error C by the learning unit 5 and the update of the parameters (weights, etc.) based on the evaluation may be performed for all the training samples. However, from the viewpoint of convergence and calculation speed, it is preferable to execute each set (mini-batch) of several to several hundred samples. The update amount Δw _ij of the weight w _{ij in} this case is determined by the following equation.

In the above equation, Δw _ij ^(t) is the current weight update amount, and Δw _ij ^(t-1) is the previous weight update amount. The first term of the above equation is a term representing the amount of correction of _wij for reducing the error by the gradient descent method, and ε is the learning rate.
The second term of the above equation is momentum. Momentum suppresses weight bias due to the selection of mini-patches by adding α (~ 0.9) times the previous update amount. The third term is weight decay. Weight attenuation is a parameter that prevents the weight from becoming excessive. The same applies to the update of the bias b _k.

[Processing content of image generator]
FIG. 5 is an explanatory diagram of the first process by the data generation unit 3.
As described above, the "first process" is a process of generating an image filter 9 having invariance of rotation and inversion with respect to the original images 7 and 8 from the sample image 7 or the target image 8.
In FIG. 5, the original image at the learning stage is the sample image 7, and the original image when the image processing device 2 is used as the classifier is the target image 8.

The original images 7 and 8 include RGB values (0 to 255) at each pixel point p (x, y) in Cartesian coordinates. Here, as shown in FIG. 5A, the data sequence (R, G, B) having the RGB value at the pixel point p as an element is referred to as a color vector “g”.
First, the data generation unit 3 extracts the center points c of the original images 7 and 8, and sets the extracted center points c as the origin of the coordinates. Next, as shown in FIG. 5B, the data generation unit 3 converts the pixel point p of the orthogonal coordinates (x, y) into polar coordinates with the center point c as the origin (polarization process).

The origin of the polar coordinates does not necessarily have to be the center point c of the original images 7 and 8, but may be a predetermined point located at a position slightly deviated from the center point c.

Next, the data generation unit 3 sets the color vector g = (R, G, B) of an arbitrary pixel point p included in the polar coordinates with the center point c as the origin, and the color vector gr in the radial direction at the pixel point p. And the color vector gt in the tangential direction is decomposed. The decomposition of the color vector g is executed by the following equation.

Here, "g ^T " is a transposed vector of the color vector g = (R, G, B). “R” is a unit vector in the radial direction at the pixel point p defined by the following equation. “T” is a unit vector in the tangential direction at the pixel point p defined by the following equation.

In the above equation, "R _θ " is a rotation vector that rotates the unit vector r by an angle θ. In the present embodiment, since the direction of the unit vector t is the tangential direction (the angle from the unit vector r is 90 degrees), the value of the angle θ of the _{rotation matrix R θ is θ = π / 2.}
By performing the above calculation on all the pixel points p included in the polar coordinates of the original images 7 and 8, a total of 6 types of elements (Rr, Rt, Gr, Gt, Br, Bt) are included in each pixel point p. A single channel image filter 9 is generated.

FIG. 6 is an explanatory diagram of the second process by the data generation unit 3.
As described above, the "second process" is a process of convolving the image filter 9 generated in the first process with respect to the original images 7 and 8.
In FIG. 6, the original image at the learning stage is the sample image 7, and the original image when the image processing device 2 is used as the classifier is the target image 8.

[Invariance of rotation and inversion of image filter]
Figure 7 is a conversion point p _theta rotating the pixel point p by a predetermined angle theta, is an illustration of a _theta 'multiple changes p rotated inverted (or reversed rotation)' is inverted reversal point p.
As shown in FIG. 7, a rotation point that is a point that advances counterclockwise by a predetermined angle θ with the same radius with respect to an arbitrary pixel point p is defined as “p _θ ”.
As shown in FIG. 7, the inverted inversion point is defined as "p'" with respect to an arbitrary pixel point p.
As shown in FIG. 7, the rotation point, which is a point advanced counterclockwise by a predetermined angle θ with the same radius with respect to the inversion point p'of an arbitrary pixel point p, is defined as "_p'θ". However, when there are a plurality of changes, the procedure for conversion may be reverse rotation or rotation reverse.
The rotation point p _theta, reversal point p ', the rotation inversion (or reversal rotation) point p' each color vector in _theta "g _theta", and "g '", "g' _theta ', rotating point p' _theta each "r _theta" a unit vector in the radial and tangential directions in, "r '", "r'_theta" and "t _theta", and "t '", "t' _theta '.

In this case, as shown in the following equation, the pixel point p, rotation point p _theta, color vector inversion point p ', more conversion points p' radial and tangential direction in the _theta ^{is, (g T r, g T} t) ^{_{, ((R θ g) T}} r, (R θ g) T t), ((Mg) T r, (Mg) T t), ((MR θ g) T r, (MR θ g) T t) Matches with.
However, in the following equation, M is an inversion matrix. M, since the diagonal elements are the diagonal matrix of 1 or ^-1, and ^M T M = I.

(In case of rotation)

(In the case of inversion)

(In the case of rotation reversal (or reversal rotation))

The above equation holds regardless of the value of the angle θ around the center point c. That is, the radial color vector gr and the tangential color vector gt at the pixel point p are invariant regardless of the angle θ around the center point c. Similarly, inversion and multiple changes also have invariance. The inversion includes both left-right inversion (y-axis symmetry) and up-down inversion (x-axis symmetry).
Therefore, if the processing of convolving the image filter 9 having the color vectors gr and gt in each direction as elements with respect to the color vectors g of the original images 7 and 8 configured as discrete information is executed, the input data after the processing will be obtained. , The input data has invariance with respect to rotation and inversion around the center point c.

[Recommended CNN structure example]
FIG. 8 is a structural diagram of the deep CNN constructed in the CNN processing unit 4.
As shown in FIG. 8, the CNN architecture for image recognition recommended by the inventors of the present application is based on a laminate of convolution layers C1 to C4 for converting an input volume into an output volume and fully coupled layers A1 to A3. It is configured.

Each layer C1-C4, A1-A3 of the CNN has neurons arranged three-dimensionally in width, height and depth.
The width, height and depth of the first input layer C1 are preferably 56 × 56 × 3. The neurons inside the convolutional layers C2-C4 and the fully connected layer A1 are connected only to the nodes in a small area called the receptive field of the previous layer.

The spatial size of the output volume can be calculated by the following equation.
W2 = 1 + (W1-K + 2P) / S
In the above equation, W1 is the size of the input volume. K is the field size of the nucleus (node) of the neuron in the convolution layer. S means stride, that is, the distance between the centers of the receptive fields of adjacent neurons in the kernel map. P means the amount of zero padding used on the border.

In the CNN of FIG. 8, in the first convolution layer C1, W1 = 56, K = 5, S = 2, P = 2. Therefore, the spatial size of the output volume of the second convolution layer C2 is W2 = 1 + (56-5 + 2 × 2) / 2 = 28.5 → 28.
The network of FIG. 8 includes seven layers with weights. The first four are convolution layers C1 to C4 and the remaining three are fully connected fully connected layers A1 to A3. Fully bonded layers A1 to A3 include dropouts.

The output of the last fully connected layer A3 is fed to 7-way SOFTMAX, which produces a distribution of 7 class labels, which is the final layer fully connected to this layer A3.
The neurons of the convolutional layers C2 to C4 and the fully connected layer A1 are connected to the receptive field of the previous layer, and the neurons of the fully connected layers A2 to A3 are connected to all the neurons of the previous layer.

The convolution layers C1 and C2 are followed by a batch normalization layer. Each batch normalization layer is followed by a pooling layer that performs the maximal pooling described above.
The nonlinear mapping function for the convolution layers C1 to C4 and the fully coupled layers A1 to A3 comprises a rectifying linear unit (ReLU).

The first convolution layer C1 filters a 56 × 56 × 3 input image (AGE image) with a 2-pixel stride by 64 kernels having a size of 5 × 5 × 3.
Stride is the distance between the centers of the receptive fields of adjacent neurons in the kernel map. The stride is set to 1 pixel in all convolution layers.

The input of the second convolution layer C2 is the output of the first convolution layer C1 that has been batch normalized and maximally pooled. The second convolution layer C2 filters the inputs with 128 kernels of size 3x3x64.
The third convolution layer C3 has 128 kernels of size 3x3x64, which are connected to the output of the second layer C2 (batch normalization and MAX pooling).

The fourth convolution layer C4 comprises 128 kernels having a size of 3x3x128. Fully connected fully connected layers A1 to A3 each include 1024 neurons.

[Recommended learning example]
The inventors of the present application actually trained (learned) the deep CNN of the structure of FIG. The training was conducted using "MATLAB", an open source numerical analysis software, for a PC equipped with an NVIDIA GTX745 4GB GPU.
In the CNN learning step, there are important settings such as weight attenuation, momentum, batch size, learning rate and parameters including learning cycle. This point will be described below.

In the training by the inventors of the present application, an asynchronous stochastic gradient descent method having a momentum of 0.9 and a weight attenuation of 0.0005 was adopted. The following equation is the update rule for the weight w adopted this time.

In the above equation, i is the number of iterations and m is the momentum variable. ε means the learning rate. The third term on the right side is the average value of the correction amount of the weight w for reducing the error L in wi with respect to the i-th batch Di.
Increasing the batch size provides a more reliable gradient estimate and can reduce the learning time, but it does not provide the maximum stable increase in the learning rate ε. Therefore, it is necessary to select a batch size suitable for the CNN model.

Here, the results of training (learning) using batch sizes of 64, 128, 256, and 512 for the convolution layers C1 to C4 were compared. As a result, it was found that the batch size of 256 was optimal for the CNN of FIG.
Equal learning rates were used for all layers and manually adjusted throughout the training. The learning rate was initialized to 0.1, and when the error rate stopped improving at the current learning rate, the learning rate was divided by 10. In the training, the network was trained in about 20 cycles.

[Experimental example: Effect of identifying handwritten characters]
The inventors of the present application used the images of the handwritten characters included in the "Manager's Circular 1" of the "Kanebo Material Database" held in the "Research Institute for Economics and Business Administration, Kobe University" to obtain the invention of the present application. A comparative experiment was conducted to test the significance.

The types of objects (handwritten characters) to be identified are "support", "distribution", "person", "work", "place", "chief", "kai", and "kai" included in the manager's turn 1. "Sha", "Ming" and "Ji". The number of samples of each character used for learning was 400 (56 × 56 pixels). FIG. 9 is a diagram showing an example of handwritten characters (original image) used in the comparative experiment.

FIG. 10 is a graph showing the test results of recognition accuracy for each character class. In FIG. 10, the graphs (ours) of ◯ show the recognition accuracy of the present invention. The graph (alexnet) of □ shows the recognition accuracy in the case of Non-Patent Document 1 (however, the input data is an RGB value).
The graph (vgg-vd-16) of ▽ shows the recognition accuracy in the case of Non-Patent Document 2 (the input data is an RGB value and the number of layers is 16). The graph (vgg-vd-19) of * shows the recognition accuracy in the case of Non-Patent Document 2 (the input data is an RGB value and the number of layers is 19).

As shown in FIG. 10, in the case of Non-Patent Documents 1 and 2 using the conventional RGB values as the input image, the recognition accuracy exceeding 90% cannot be achieved for all 10 types of character classes.
On the other hand, in the case of the present invention, which imparts invariance of rotation and inversion to the original image of handwritten characters, recognition accuracy of 90% or more was obtained for all 10 types of character classes.

As is clear from the graph of FIG. 10, in image recognition using a deep CNN (character recognition in the example of FIG. 10), if the original image is processed to give invariance of rotation and inversion to obtain CNN input data, The recognition accuracy is significantly improved as compared with the case where the conventional raw data (RGB image) is used as the input data.

[Effect of image recognition device]
As described above, according to the image recognition device 10 of the present embodiment, the input data to the CNN processing unit 4 is generated by convolving the image filter 9 having the invariance of rotation and inversion with respect to the original images 7 and 8. do. Therefore, even if a large number of sample images 7 are not collected, higher recognition accuracy can be exhibited as compared with other deep learning techniques.
For example, as shown in FIG. 10, when the number of sample images 7 is "400", higher recognition accuracy than the conventional deep learning technique can be obtained.

In the present embodiment, since the single-channel image filter 9 including 6 types of elements (Rr, Rt, Gr, Gt, Br, Bt) is convoluted into the sample image 7, the amount of data contained in one sample image 7 is at least It will be expanded 16 times.
Therefore, when 100 sample images 7 are required in the conventional deep learning, if about 6 (100/16 = 6.25) sample images 7 are collected, it is almost the same as the conventional deep learning. A degree of identification accuracy can be obtained.

According to the image recognition device 10 of the present embodiment, even a rotated or inverted object can be accurately recognized by convolution of the image filter 9.
Therefore, for example, it can be used as a device that accurately recognizes characters included in a photographed image such as an old document and converts them into text or word processor document data. It can also be used as a device that reads characters described in a document such as a form and converts them into text or word processor document data, or a device that recognizes characters handwritten on a touch panel in real time.

In addition, the image recognition device 10 of the present embodiment can be used for facial expression recognition of a human face, age recognition from a face image, recognition of all types of objects such as animals, plants, and products.
As described above, in the image recognition device 10 of the present embodiment, the object whose type can be recognized by the CNN processing unit 4 may be at least one of handwritten characters, humans, animals, plants, and products. It can be used to recognize any type of object.

[Other application examples of image recognition device]
FIG. 11 is an overall configuration diagram of the product monitoring system 20 of the present embodiment.
The product monitoring system 20 is a system that uses an image recognition device 10 (see FIG. 1) that recognizes the types of products 25 included in the captured image for sorting and taking out defective products.
As shown in FIG. 11, the product monitoring system 20 includes a photographing device 21, a drive control device 22, and a movable arm type robot device 23.

The photographing device 21 includes, for example, a digital camera that generates a digital image using a CCD (charge-coupled device). The photographing device 21 is suspended above the belt conveyor 24, and photographs a plurality of products 25 on the belt conveyor 24 traveling downstream (on the right side in FIG. 11) from above.
The photographing device 21 transmits a photographed image including a digital image including a plurality of products 25 to the drive control device 22. The captured image may be either a still image or a moving image.

The drive control device 22 includes a computer device that controls the operation of the robot device 23. The drive control device 22 includes a first communication unit 26, a second communication unit 27, a control unit 28, and a storage unit 29.
The first communication unit 26 includes a communication device that communicates with the photographing device 21 according to a predetermined I / O interface standard. The communication between the first communication unit 26 and the photographing device 21 may be either wired communication or wireless communication.

The second communication unit 27 includes a communication device that communicates with the robot device 23 according to a predetermined I / O interface standard. The communication between the second communication unit 27 and the robot device 23 may be either wired communication or wireless communication.

The control unit 28 is an information processing device having one or more CPUs, and includes a control device including the image identification device (see FIG. 1) of the present embodiment described above.
The storage unit 29 includes one or more RAMs and a storage device including a memory such as a ROM. The storage unit 29 functions as a temporary or non-temporary recording medium such as various computer programs executed by the control unit 28 and image data received from the photographing device 21 or the like.

In this way, the drive control device 22 is configured to include a computer. Therefore, each function of the drive control device 22 is exhibited by executing the computer program stored in the storage device of the computer by the CPU and GPU of the computer.
Such a computer program can be stored in a temporary or non-temporary recording medium such as a CD-ROM or a USB memory.

By reading and executing the computer program stored in the storage unit 29, the control unit 28 can realize communication control for the first and second communication units 26 and 27 and operation control of the robot device 23.
For example, the control unit 28 extracts an image portion of the product 25 (hereinafter, referred to as “product image”) from the captured image received by the first communication unit 26, and classifies the extracted product image (here, “normal”). ”Or“ defective ”).

When the control unit 28 detects a product image having a defective classification class, it calculates the time and position at which the product 25 passes under the robot device 23 from the shooting time, the traveling speed of the belt conveyor 24, and the like, and the calculated passage. The time and the passing position are transmitted to the second communication unit 27.
The robot device 23 includes an articulated robot arm 30 and an actuator 31 that drives the robot arm 30. The robot arm 30 has a hand portion (not shown) capable of gripping the product 25 on the conveyor 24.

The passing time and passing position of the defective product calculated by the control unit 28 are transmitted to the actuator 31. The actuator 31 drives each joint of the robot arm 30 so that the hand portion moves to the passing position at the passing time of the defective product, grasps the product 25, and takes it out to the outside.
The storage unit 29 stores a CNN having a predetermined structure (for example, FIG. 8) capable of performing a quality determination of the product 25, a learned weight and a bias with respect to the CNN, and the like. The control unit 28 determines the quality of the product image extracted from the captured image based on the learned CNN.

The work processes to be performed by the factory manager, which are necessary to realize the above-mentioned product monitoring system 20, are as follows.
Step 1) A worker 32 on the downstream side of the robot arm 30 determines a defective product from the products 25 flowing on the conveyor 24, and photographs the defective product with a digital camera (not shown).
Step 2) The captured image data is input to the storage unit 29 of the drive control device 22 as a sample image 7 of a defective product.

Step 3) The above steps 1 and 2 are repeated until the desired identification accuracy (for example, 99% or more) is obtained.
When the typical shape of the defective product is known in advance, the sample image of the defective product may be image data obtained by photographing a product other than the product 25 flowing on the conveyor 24.

In the learning stage in which steps 1 and 2 are repeated, since the sample image 7 of the defective product is small at the beginning of operation, the recognition accuracy of the defective product by the drive control device 22 is low.
However, as the number of sample images 7 of defective products increases, the recognition accuracy of the drive control device 22 improves, and defective products can be eliminated in a state close to 100%. Therefore, since the drive control device 22 learns from a predetermined number of sample images 7, visual monitoring by the worker 32 becomes unnecessary.

In particular, in the present embodiment, since a discriminator having high recognition accuracy is used even for a rotated or inverted object, as shown in FIG. 11, for example, even in the case of a product 25 mounted on a conveyor 24 in various orientations, almost at an early stage. Defective products can be identified in a form close to 100%.
As an example of the product 25 mounted on the conveyor 24 in various directions, foods such as sweets and paste products, molded products produced by a molding machine, and the like can be considered. Further, when the product 25 is currently conveyed on the conveyor 24 and the worker 32 visually observes and removes the defective product from the conveyor 24, the product monitoring system 20 of the present embodiment is adopted. Therefore, the effect of significantly reducing the number of the workers 32 who are performing the visual inspection can be expected.

[First modification]
In the above-described embodiment, the decomposition direction of the color vector g is not limited to the “radial direction” (r direction in FIG. 5) and the “tangential direction” (t direction in FIG. 5). That is, the decomposition direction of the color vector g may be any two directions that open at a predetermined angle with the pixel point p as the base point.
However, if the data is decomposed in two directions other than the radial direction and the tangential direction, the calculation formula of the color vector in each direction becomes complicated, and the processing load of the data generation unit 3 becomes large. Therefore, as described in the above embodiment, the decomposition direction of the color vector g is preferably the radial direction and the tangential direction.

[Second modification]
In the above-described embodiment, the three-dimensional polar coordinates with the center point c as the origin are defined, and the color vector g is decomposed into a radial direction and two tangential directions (three directions in total in the case of three dimensions). The original images 7 and 8 may be convoluted by a three-dimensional image filter.
By doing so, the image recognition accuracy is improved not only for the two-dimensional rotation or inversion with the center point c as the origin but also for the target image 8 tilted in the depth direction with the center point c as the origin. be able to.

[Third variant]
In the above-described embodiment, the image filter 9 having rotation and inversion invariance with respect to the original images 7 and 8 is adopted, but the image filter 9 has rotation-only invariance with respect to the original images 7 and 8. An image filter or an image filter having invariance of only inversion with respect to the original images 7 and 8 may be used. That is, the image filter 9 of the present invention may be an image filter having at least one invariance of rotation and inversion with respect to the original images 7 and 8.

[Other variants]
The embodiments disclosed this time (including modified examples) are examples in all respects and are not restrictive. The scope of rights of the present invention is not limited to the above-described embodiment, and includes all modifications within a range equivalent to the configuration described in the claims.
For example, in the above-described embodiment, the neural network is composed of a convolutional neural network (CNN), but it may be a hierarchical neural network having another structure that does not have a convolutional layer.

1 Arithmetic processing unit 2 Image processing unit 3 Data generation unit 4 CNN processing unit 5 Learning unit 6 Recognition unit 7 Sample image (original image)
8 Symmetric image (original image)
9 Image filter 10 Image recognition device 20 Product monitoring system 21 Imaging device 22 Drive control device (control device)
23 Robot device 24 Belt conveyor 25 Product 26 1st communication unit 27 2nd communication unit 28 Control unit 29 Storage unit 30 Robot arm 31 Actuator 32 Worker

Claims (13)

A data generator that generates input data by performing predetermined data processing on the original image,
An image recognition device including an image processing unit having a hierarchical neural network that recognizes the types of objects included in the generated input data.
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network, and when the original image is a recognition target image, the image processing unit performs the process of learning the parameters of the network. Performs the process of outputting the network recognition result and
Wherein the data processing data generating unit performs an image recognition apparatus wherein a treatment for imparting invariance of the rotation and reversing the original image. The image recognition device according to claim 1, wherein the data processing performed by the data generation unit includes a first process and a second process defined below.
First processing: rotation and generating an image filter having the invariance of the inversion processing second process on an original image: processing of convoluting the image filter produced in the first process to the original image A data generator that generates input data by performing predetermined data processing on the original image,
An image recognition device including an image processing unit having a hierarchical neural network that recognizes the types of objects included in the generated input data.
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network, and when the original image is a recognition target image, the image processing unit performs the process of learning the parameters of the network. Performs the process of outputting the network recognition result and
The data processing performed by the data generation unit is a process of imparting at least one invariance of rotation and inversion to the original image.
The data processing performed by the data generation unit includes the first processing and the second processing defined below.
The image filter is a plurality of color vectors obtained by dividing a color vector of an arbitrary pixel point defined by polar coordinates having a predetermined point of the original image as an origin in an arbitrary direction that opens at a predetermined angle starting from the pixel point. images recognizer ing than elements included in.
First process: A process of generating an image filter having at least one invariance of rotation and inversion with respect to the original image.
Second process: A process of convolving the image filter generated in the first process into the original image. A data generator that generates input data by performing predetermined data processing on the original image,
An image recognition device including an image processing unit having a hierarchical neural network that recognizes the types of objects included in the generated input data.
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network, and when the original image is a recognition target image, the image processing unit performs the process of learning the parameters of the network. Performs the process of outputting the network recognition result and
The data processing performed by the data generation unit is a process of imparting at least one invariance of rotation and inversion to the original image.
The data processing performed by the data generation unit includes the first processing and the second processing defined below.
The image filter includes two color vectors obtained by dividing a color vector of an arbitrary pixel point defined by polar coordinates with a predetermined point of the original image as an origin in a radial direction and a tangential direction starting from the pixel point. images recognizer ing from elements.
First process: A process of generating an image filter having at least one invariance of rotation and inversion with respect to the original image.
Second process: A process of convolving the image filter generated in the first process into the original image. The image recognition device according to any one of claims 1 to 4, wherein the hierarchical neural network is a convolutional neural network. The image recognition device according to any one of claims 1 to 5, wherein the object whose type is recognized is composed of at least one object of handwritten characters, humans, animals, plants, and products. A data generator that generates input data by performing predetermined data processing on the original image,
A computer program for operating a computer as an image recognition device including an image processing unit having a hierarchical neural network that recognizes objects included in the generated input data.
Wherein the data generating unit, as the data processing, and performing the treatment for imparting invariance of rotation and inverted with respect to the original image,
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network.
A computer program including a step of outputting a recognition result of the network when the original image is a recognition target image by the image processing unit. A data generator that generates input data by performing predetermined data processing on the original image,
An image recognition method executed by an image recognition device including an image processing unit having a hierarchical neural network that recognizes objects included in the generated input data.
Wherein the data generating unit, as the data processing, and performing the treatment for imparting invariance of rotation and inverted with respect to the original image,
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network.
An image recognition method including a step of outputting a recognition result of the network when the original image is a recognition target image by the image processing unit. A shooting device that shoots multiple products,
A robot device that takes out any of the above-mentioned multiple products that have been photographed, and
The control apparatus including an image recognition apparatus, a product monitoring system and a said control device to instruct the product to the robotic device to retrieve,
The image recognition device is
A data generator that generates input data by performing predetermined data processing on the original image,
An image processing unit having a hierarchical neural network that recognizes the types of objects included in the generated input data is provided.
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network, and when the original image is a recognition target image, the image processing unit performs the process of learning the parameters of the network. Performs the process of outputting the network recognition result and
The data processing performed by the data generation unit is a process of imparting at least one invariance of rotation and inversion to the original image.
The image recognition device is a product monitoring system that instructs the robot device to take out the product recognized as a defective product. A data generator that generates input data by performing predetermined data processing on the original image,
A computer program for operating a computer as an image recognition device including an image processing unit having a hierarchical neural network that recognizes objects included in the generated input data .
As the data processing, the data generation unit performs a process of imparting at least one invariance of rotation and inversion to the original image, and a step of performing the process.
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network.
When the original image is a recognition target image, the image processing unit includes a step of outputting the recognition result of the network.
The data processing performed by the data generation unit includes the first processing and the second processing defined below.
The image filter is a plurality of color vectors obtained by dividing a color vector of an arbitrary pixel point defined by polar coordinates having a predetermined point of the original image as an origin in an arbitrary direction that opens at a predetermined angle starting from the pixel point. A computer program consisting of the elements contained in.
First process: A process of generating an image filter having at least one invariance of rotation and inversion with respect to the original image.
Second process: A process of convolving the image filter generated in the first process into the original image. A data generator that generates input data by performing predetermined data processing on the original image,
A computer program for operating a computer as an image recognition device including an image processing unit having a hierarchical neural network that recognizes objects included in the generated input data .
As the data processing, the data generation unit performs a process of imparting at least one invariance of rotation and inversion to the original image, and a step of performing the process.
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network.
When the original image is a recognition target image, the image processing unit includes a step of outputting the recognition result of the network.
The data processing performed by the data generation unit includes the first processing and the second processing defined below.
The image filter includes two color vectors obtained by dividing a color vector of an arbitrary pixel point defined by polar coordinates having a predetermined point of the original image as an origin in a radial direction and a tangential direction starting from the pixel point. A computer program consisting of elements.
First process: A process of generating an image filter having at least one invariance of rotation and inversion with respect to the original image.
Second process: A process of convolving the image filter generated in the first process into the original image. A data generator that generates input data by performing predetermined data processing on the original image,
An image recognition method executed by an image recognition device including an image processing unit having a hierarchical neural network that recognizes objects included in the generated input data.
As the data processing, the data generation unit performs a process of imparting at least one invariance of rotation and inversion to the original image, and a step of performing the process.
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network.
When the original image is a recognition target image, the image processing unit includes a step of outputting the recognition result of the network.
The data processing performed by the data generation unit includes the first processing and the second processing defined below.
The image filter is a plurality of color vectors obtained by dividing a color vector of an arbitrary pixel point defined by polar coordinates having a predetermined point of the original image as an origin in an arbitrary direction that opens at a predetermined angle starting from the pixel point. An image recognition method consisting of the elements contained in.
First process: A process of generating an image filter having at least one invariance of rotation and inversion with respect to the original image.
Second process: A process of convolving the image filter generated in the first process into the original image. A data generator that generates input data by performing predetermined data processing on the original image,
An image recognition method executed by an image recognition device including an image processing unit having a hierarchical neural network that recognizes objects included in the generated input data.
A step before Symbol data generating unit, as the data processing, to perform the process of applying at least one invariance of rotation and inverted with respect to the original image,
When the original image is a sample image, the image processing unit performs a process of learning the parameters of the network based on the recognition result of the network.
When the original image is a recognition target image, the image processing unit includes a step of outputting the recognition result of the network.
The data processing performed by the data generation unit includes the first processing and the second processing defined below.
The image filter includes two color vectors obtained by dividing a color vector of an arbitrary pixel point defined by polar coordinates with a predetermined point of the original image as an origin in a radial direction and a tangential direction starting from the pixel point. An image recognition method consisting of elements.
First process: A process of generating an image filter having at least one invariance of rotation and inversion with respect to the original image.
Second process: A process of convolving the image filter generated in the first process into the original image.

Images