JP6701057B2 - Recognizer, program - Google Patents

Description

  The present disclosure relates to object recognition.

  Patent Literature 1 discloses the following contents as a method for detecting road topographic information. Performs high level spatial feature generation for selected locations called base points. Spatial feature generation for this base point is performed based on a continuous value-based reliability expression that captures the visual and physical features of the environment.

JP, 2013-073620, A

  In reality, the technique of Patent Document 1 is insufficient in accuracy. In particular, erroneous recognition is likely to occur at boundaries between objects. Such a problem is not limited to detection of road topography. Based on the above, the present disclosure has as a problem to be solved the recognition of road terrain and other objects with high accuracy.

One form of the present disclosure labels each pixel by inputting the information for each pixel acquired from an image to be recognized as input data to a deep neural network (13) including a plurality of intermediate layers. An initial recognition unit (S200) for acquiring an initial recognition value, and a scene identification intermediate that is at least a part of intermediate data output from at least one of the plurality of intermediate layers, for the scene to which the image to be recognized belongs. identifying the scene identification section on the basis of the data and (S300), targeting the initial recognition value, the a correcting unit that executes a correction based on the intermediate data (S500), the correction based on the identified scene And a correction unit (S500) including a correction execution unit (S740) to be executed .

  According to this aspect, since the label given by the initial recognition unit is corrected by using the intermediate data in the initial recognition, the labeling can be performed with high accuracy, and the recognition accuracy of the object is improved.

FIG. 3 is a functional block diagram of a vehicle equipped with a recognition device. The flowchart which shows a recognition apparatus. The figure which shows the mode of labeling by a deep neural network. The figure which overlapped the initial recognition value and the image. The figure which shows the mode of learning of a scene classifier. The figure which shows an example of a scene. The figure which shows the relationship between an error area and a patch. The flowchart which shows the narrowing down of the label using a patch. The figure which shows the mode of learning of a positional relationship feature-value and an appearance-distance feature-value. The figure which shows the spatial relationship between a patch and a receptor. The flowchart which shows narrowing down of a part if a pixel is used. The figure which shows a mode of appearance-distance feature-value production|generation. The figure which shows an example of the recognition result by an initial recognition value. The figure which shows an example of the recognition result by a final recognition value.

  As shown in FIG. 1, the recognition device 10 is mounted on the automobile 1. The recognition device 10 acquires the RGB values of each pixel from the camera 21 and the camera 22, respectively, as imaging data. The recognition device 10 recognizes the image represented by the imaged data by the recognition processing shown in FIG. 2, and inputs the recognition result to the control unit 25. The control unit 25 controls the operation of the automobile 1 based on the input recognition result.

  The cameras 21 and 22 are mounted so that the front of the automobile 1 is within the imaging range. The cameras 21 and 22 form a stereo camera.

  The recognition device 10 is configured as a well-known computer including a CPU 11 and a memory 12 such as a ROM and a RAM. The recognition device 10 uses the CPU 11 and the memory 12 to execute a recognition process by executing a program stored in the memory.

  The recognition device 10 repeatedly executes the recognition process while the vehicle can run. The recognition process is a process for realizing semantic segmentation in which labeling is performed for each pixel.

  When the recognition device 10 starts the recognition process, the recognition device 10 generates input data in S100. The input data is data having the hue, saturation, and distance of each pixel as parameters. The recognition device 10 generates input data from the imaged data input from the cameras 21 and 22. A known method is used to generate the input data.

  Next, the recognition device 10 obtains an initial recognition value by performing initial recognition in S200. The initial recognition is acquired by inputting input data to the deep neural network (hereinafter, DN) 13 shown in FIG. The DN 13 is stored in the memory 12 in advance.

  The DN 13 includes a plurality of intermediate layers. Specifically, the DN 13 includes a convolutional neural network and a deconvolutional neural network. The input data input to the DN 13 is converted into the intermediate data C1 having the number of pixels of 224×224 by the convolutional layer.

  The intermediate data C1 is converted by the pooling layer into the intermediate data P2 having 112×112 pixels. In this embodiment, MAX pooling is used. In this way, the outputs from the convolutional layer and the pooling layer are alternately repeated until the total coupling layer FC having the number of pixels of 1×1 is reached. The number of convolutional layers sandwiched between a certain pooling layer and the next pooling layer may be two or more.

  The fully concatenated layer FC is converted into intermediate data DC-FC having 7×7 pixels by the deconvolution layer. The intermediate data DC-FC is converted into intermediate data UP1 having 14×14 pixels by the ampling layer. In this way, when the outputs of the deconvolution layer and the ampling layer are alternately repeated and the intermediate data DC5 of the number of pixels 224×224 is output, the initial recognition value is obtained. FIG. 4 shows a state in which the initial recognition value and the captured image are superimposed.

  The initial recognition value is data in which a label is given to each of 224×224 pixels. In the label in this embodiment, the label K1 means a road, the label K2 means an obstacle, the label K3 means an empty space, and the label K4 means a ceiling. The ceiling is a ceiling in a tunnel or the like.

  The DN 13 has been learned by supervised learning using a combination of a large number of learning input data and the true value of the label, and each intermediate layer is finely adjusted. When input data is input to the DN 13, the confidence value of each label is output for each pixel. The initial recognition is realized by adopting the label with the highest confidence value for each pixel.

  After the initial recognition, the recognition apparatus 10 performs the scene identification in S300 and the error area identification in S400.

  The scene identification is realized by inputting the scene identification intermediate data to the scene identification unit 310. The scene identification intermediate data in this embodiment is intermediate data P5.

  The scene classifier 310 uses a random forest. The scene classifier 310 has been learned by the supervised learning shown in FIG.

  For example, the intermediate data P5a generated from the image shown in FIG. 6 is input. Then, the true value of the scene shown in FIG. 6 is “two lanes, the outer boundary is defined by two white solid lines, and the lane is divided by one broken line”. Is learned as scene a. Then, a large amount of data is prepared for each of the scene a and various other scenes (for example, a highway, a tunnel, etc.), and learning with a teacher is performed.

  Next, the error area will be described. The error area is an area in which the reliability of the initial recognition value is low. The confidence value output as the initial recognition is used to specify the error area. Specifically, when there is no label with a remarkably high confidence value, it is specified as a pixel included in the error region.

  In the present embodiment, when the reliability value for each pixel is standardized and the total value is set to 100%, an area in which any reliability value is less than a threshold value (for example, 80%) is specified as an error area. It

  For example, if the road reliability value assigned to a certain pixel is 99%, it is determined that the pixel is not included in the error region. On the other hand, if the road reliability value is 40%, the obstacle reliability value is 40%, the sky reliability value is 10%, and the ceiling reliability value is 10%, it is determined that the road area is included in the error area. In the case of the example shown in FIG. 7, the error areas E1, E2, E3 and E4 are specified. Each of the error areas E1, E2, E3, E4 is specified so as to form a closed area.

  Finally, the final recognition value is obtained by executing the correction process using the initial recognition value, the error area information, the identified scene (hereinafter, the identification scene), and the narrowing intermediate data. The narrow-down intermediate data in this embodiment is intermediate data C1 as shown in FIG.

  In the correction process, the patch level narrowing down in S600 and the pixel level narrowing down in S700 are performed.

  The patch in the present embodiment means each area preliminarily divided into 4×4 as shown in FIG. 7. Therefore, each patch has 56×56 pixels. A serial number is determined in advance for each patch. For example, the upper left patch is the patch p1, the patch p2 is on the right of the patch p1, and the patch p5 is immediately below the patch p1.

  As shown in FIGS. 2 and 8, the initial recognition value, the error area information, and the identification scene are used for narrowing down the patch level.

  Specifically, the recognition device 10 first identifies a patch including an error area in S610. In the case of the example shown in FIG. 7, the error area E1 is included in the patch p10. Further, the error area E2 is included in the patch p6, the error area E3 is included in the patch p11, and the error area E4 is included in the patch p12. Therefore, the patches p6, p10, p11, p12 are specified.

  Next, in S620, the recognition device 10 reads out the learned positional relationship feature amount for each identified patch based on the identification scene.

  Here, the positional relationship feature amount and its learning will be described. The positional relationship feature amount in the present embodiment is a feature amount that depends on the scene, and is also a feature amount related to the relative positional relationship between labels. As shown in FIG. 9, the positional relationship feature amount is learned by supervised learning for each scene.

  In learning, many sets as true values in supervised learning are given. Each set is composed of a true value of a scene and a label as a true value attached to each pixel (224×224) of input data belonging to the scene.

  Let us say that the true value of a scene included in a certain set is a scene s, and the spatial arrangement of labels as the true value included in that set is a label arrangement G, and learning of this set will be described below as an example. The learning of the positional relationship feature amount is realized by associating the positional relationship feature amount of the label arrangement G with the scene s.

  The calculation of the positional relationship feature amount of the label arrangement G first generates the label arrangement P divided into the patches described above and the label arrangement Q divided into four receptors, as shown in FIG. Therefore, each receptor has 112×112 pixels. The receptors included in the label arrangement Q are assigned serial numbers q1 to q4. The number of receptors included in the label arrangement Q is smaller than the number of patches included in the label arrangement P.

The positional relationship feature amount g is calculated for each of the patches p1 to p16. Specifically, it is calculated by the following formula.
g _pn ^s =[ω _pn ^K1 , ,ω _pn ^K2 ,ω _pn ^K3 ,ω _pn ^K4 ]・・・(1)
S in the above equation indicates that it is the positional relationship feature amount g of the scene s. pn indicates any one of p1 to p16. K1 to K4 represent labels. Hereinafter, any one of K1 to K4 is also referred to as Km.

ω _pn ^Km represents a geometrical relationship of (label Km in patch pn)-(all labels in receptors q1 to q4). The arrow shown in FIG. 10 indicates a vector forming ω _p16 ^K1 .

Notation of the above vector is as follows.
ω _pn ^Km = [v(pn,q1) _Km ^K1 ,…,v(pn,q4) _Km ^K1 ,…,v(pn,q1) _Km ^K4 ,…,v(pn,q4) _Km ^K4 ]・・・(2)
v(pn,q1) _Km ^K1 is a two-dimensional average space vector of (patch Km with patch pn)−(label K1 with receptor q1). v(pn,q4) _Km ^K1 is a two-dimensional average space vector of (label Km for patch pn)-(label K1 for receptor q4). v(pn,q1) _Km ^K4 is a two-dimensional average space vector of (patch Km with patch pn)−(label K4 with receptor q1). v(pn,q4) _Km ^K4 is a two-dimensional average space vector of ((patch mn is label Km)-(receptor q4 is label K4).

For example, the average space vector v(pn,q4) _Km ^K1 is calculated by the space vector between (label _Km- center of gravity of patch pn) and (all pixels constituting receptor q4 including label K1).

  The average space vector is an average value of all calculated space vectors and is represented by a two-dimensional size and an angle. The angle mentioned here is an angle formed with the horizontal axis.

v(pn,qi) _Km ^K1 becomes a zero vector [0,0] when the label K1 is not included in the receptor qi. i is any one of 1-4. For example, as shown in FIG. 10, since the label q1 is not included in the receptor q1, v(pn,q1) _Km ^K1 becomes a zero vector. Therefore, the vector indicating v(p16,q1) _K1 ^K1 is not shown in FIG.

Similarly, v(pn,qi) _Km ^K1 becomes a zero vector [0,0] when the label Km is not included in the patch pn. Therefore, from equation (2), ω _pn ^Km contains only a zero value if the label Km is not included in the patch pn. That is, when the label Km is not included in the patch pn, the equation (2) is as follows.
ω _pn ^Km ＝[0,…,0,…,0,…,0]・・・(3)

  The recognition device 10 has learned many sets of true values by the above method. As described above, the recognition device 10 reads the positional relationship feature amount g based on the identification scene for each patch identified as the error region in S620.

On the other hand, in S630, the recognition apparatus 10 calculates the positional relationship feature amount ρ _pn ^s (Km) by performing the same calculation as that at the time of learning for each patch identified as the error region. In S630, the positional relationship feature amount ρ _pn ^s for each of the labels K1 to K4 is calculated as Km. When calculating the label K1, the label K1 is calculated as a correction candidate. That is, when calculating the label K1, it is assumed that the dominant label in the patch to be calculated is the label K1. The same applies to the calculation for each of the labels K2 to K4.

  The dominant label in a patch is a label that is associated with a larger number of pixels in the patch than other labels.

  Subsequently, in S630, pruning based on the patch is executed, and the narrowed down label is output. The pruning based on the patch specifically means the following contents.

  For each patch, the difference in Euclidean distance between the positional relationship feature amount g read out in S620 and the positional relation feature amount ρ calculated in S630 is calculated for each of the labels K1 to K4. Labels for which this difference is greater than or equal to a threshold are excluded, and correction candidates are narrowed down to the remaining labels. In some cases, only one dominant label remains, and in some cases multiple labels remain.

  The pruning based on the patch is a process of eliminating the label that is not a correction candidate in each patch in this way.

  As shown in FIGS. 2 and 11, in the pixel level narrowing down, the final recognition value is output using the narrowing down intermediate data, the error area information, the identification scene, and the narrowed down label information.

  First, in S710, the recognition apparatus 10 identifies the pixel serving as the center of gravity of each error area (hereinafter, the center of gravity pixel) as a representative of each error area. If no pixel exactly coincides with the center of gravity, the pixel having the shortest distance from the center of gravity is specified as the center of gravity pixel.

  Subsequently, in S720, the recognition device 10 reads out the learned appearance-distance feature amount for the centroid pixel of each error region. The appearance-distance feature quantity is a scene-dependent feature quantity. In addition, the appearance-distance feature amount is a feature amount related to hue, saturation, and distance, and thus is a feature amount corresponding to the parameter included in the input data.

  As shown in FIG. 9, the appearance-distance feature amount is learned by supervised learning for each scene. In learning, many sets as true values in supervised learning are given. Each set is composed of the true value used for learning the positional relationship feature amount and the data obtained by performing convolution processing on the input data belonging to the scene. The convolution process here is a convolution process for obtaining the intermediate data C1 in the DN 13. That is, this data is the intermediate data C1 output by the DN 13.

  Although omitted in the description of the DN 13, the intermediate data C1 is composed of data composed of D 224×224 pixels. D is an integer of 2 or more. Therefore, for example, the pixel n shown in FIG. 12 can be associated with the D-dimensional feature vector obtained by the convolution processing. FIG. 12 shows the feature vector λ(h,w) corresponding to the pixel n.

  As shown in FIG. 12, in the present embodiment, a 3×3 filter is used in the convolutional layer for obtaining the intermediate data C1. Therefore, the feature vector λ is a feature amount that reflects the parameters of eight pixels around the pixel n.

  By averaging such feature vectors, the appearance-distance feature amount corresponding to the pixel n is determined and used as learning data. The appearance-distance feature amount is a feature amount generated from information on appearance and distance. Appearance here means saturation and hue. As the appearance-distance feature amount, the feature amount relating to hue, saturation, and distance is obtained because the input data is composed of these parameters.

  The average of the feature vectors is the average of the feature vectors acquired for the pixels at the same position in the input data belonging to the same scene, and the pixels labeled as the same true value. Therefore, as the appearance-distance feature amount, learning data of each label is obtained for each scene and each pixel. In S720, the appearance-distance feature amount of each label corresponding to the identification scene and the barycentric pixel is read.

  On the other hand, in S730, the appearance-distance feature amount is acquired for the center-of-gravity pixel of each error region from the intermediate data C1 as the intermediate data for narrowing down. In S730, the feature vector acquired by the same method as the above learning is acquired as the appearance-distance feature amount.

  Finally, as S740, the final recognition value is output. As shown in FIG. 11, S740 is executed based on the outputs of S720 and S730 and the label narrowed down by S630. In S740, the label correction based on the Euclidean distance is executed. Specifically, it is executed as follows.

  For each centroid pixel, the difference in Euclidean distance between the feature vector read in S720 and the feature vector acquired in S730 is calculated for each label. Labels whose difference is less than or equal to a threshold value are correction candidates for the entire error area including the centroid pixel. If any of the labels that are candidates for correction matches any of the labels narrowed down in S600, the label as the initial recognition value is corrected to that label.

  When a plurality of labels match, the label having the shortest sum of the Euclidean distances of the information feature amount and the positional relation feature amount is corrected. In this case, the information feature amount and the positional relationship feature amount may be appropriately weighted. The output containing the label thus corrected is the final recognition value.

  Compared with the initial recognition value shown in FIG. 13, in the case of the final recognition value shown in FIG. 14, the recognition accuracy is improved particularly in the area surrounded by the broken line. That is, in the case of the final recognition value, the boundary between the road and the obstacle can be recognized more accurately in the area surrounded by the broken line.

(1) The recognition accuracy is improved as described above because the intermediate data is used.
(2) The scene identification intermediate data, which is one of the intermediate data, is used for scene identification. Therefore, the correction using the identification scene can be executed.
(3) Since the intermediate data P5 as the intermediate data for scene identification is data whose information is compressed up to 7×7 by a plurality of pooling processes, it is suitable for scene identification.
(4) Since the scene identification is performed based on the comparison with the data learned by the supervised learning, the accuracy is high.

(5) By using the intermediate data C1 which is the intermediate data for narrowing down, it is possible to acquire the feature amount in which the influence of the surroundings is reflected at the pixel level. Then, the recognition accuracy is improved by comparing the feature vector, which is the feature amount, with the learning result.
(6) Since the intermediate data C1 is data obtained by performing the convolution process once on the input data, it is suitable as a feature amount indicating the relationship with the input data.
(7) Since the intermediate data C1 has not been subjected to pooling processing and has not been compressed, it is suitable as a feature amount indicating the relationship with the input data.

(8) Since the positional relationship feature amount is used in the correction process, the accuracy of correction is improved.
(9) The calculation of the positional relationship feature amount can be appropriately executed by using the patch and the receptor.

(10) In the correction process, since the correction is performed by integrating the information feature amount and the positional relationship feature amount, the accuracy of the correction is improved.

(11) Since the correction process is performed on the error region as a target, the label having a high confidence value in the initial estimation can be excluded from the correction target. Therefore, the processing load is reduced and the correction of the label that does not need to be corrected is suppressed.
(12) Since the patch not including the error area is excluded from the calculation target of the positional relationship feature amount, the processing load is reduced.
(13) Since the correction of each error area is executed by using the correction result of the pixel representing the error area, the processing load is reduced.
(14) Since the pixel representing the error area is the center-of-gravity pixel, the accuracy of correction is improved as compared with the case where the pixel located at the end of the error area is the representative pixel.

  The appearance-distance feature amount corresponds to the information feature amount. In addition, S200 is an initial recognition unit, S300 is a scene identification unit, S400 is a specification unit, S500 is a correction unit, S630 is a calculation unit, S710 is a centroid pixel acquisition unit, S730 is an information feature amount acquisition unit, and S740 is a correction execution unit. Corresponding to.

  The present disclosure is not limited to the embodiments, examples, and modified examples of the present specification, and can be realized in various configurations without departing from the gist thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each mode described in the column of the outline of the invention are to solve some or all of the above-mentioned problems, or In order to achieve some or all of the effects described above, they can be replaced or combined as appropriate. If the technical features are not described as essential in this specification, they can be deleted as appropriate. For example, the following are exemplified.

  Utilization of the intermediate data may not depend on the scene. For example, when capturing image data from a camera that is fixed for a location and performing recognition, the scene is fixed. In such a case, the intermediate data may be used to recognize a changeable object in the fixed scene.

  It may be used for other than the traveling control of the automobile. For example, as described above, it may be applied to a fixed camera, may be another transportation device (for example, a motorcycle), or may be a robot.

  Only one of the patch level narrowing in S600 and the pixel level narrowing in S700 may be executed. Even in this case, the improvement of the recognition accuracy by using the intermediate data is realized.

  It is not necessary to specify the error area. In this case, the patch level may be narrowed down to all patches, or the pixel level may be narrowed down to all pixels.

  The input data may not be composed of hue, saturation, and distance. For example, it may be composed of RGB values and a distance, or may be composed of a luminance value and a distance. The captured image may be monochrome if it is composed of a luminance value and a distance.

  The information on the distance included in the input data may be acquired from other than the stereo camera. For example, a depth sensor may be used, or a radar wave or the like may be used.

  In the above embodiment, some or all of the functions and processes realized by software may be realized by hardware. In addition, some or all of the functions and processes realized by hardware may be realized by software. As the hardware, for example, various circuits such as an integrated circuit, a discrete circuit, or a circuit module in which these circuits are combined may be used.

13 deep neural networks, 20 recognizers

Claims (14)

By inputting information for each pixel acquired from the image to be recognized as input data to the deep neural network (13) including a plurality of intermediate layers, each pixel is labeled and an initial recognition value is acquired. An initial recognition unit (S200),
A scene identification unit (S300) for identifying a scene to which the image to be recognized belongs, based on scene identification intermediate data that is at least a part of intermediate data output from at least one of the plurality of intermediate layers;
Targeting the initial recognition value, the a correcting unit that executes a correction based on the intermediate data (S500), the correct, the identified correction execution unit for executing, based on the scene (S740) and correction unit comprising ( S500) ,
A recognition device including. The recognition device according to claim 1 , wherein the intermediate data for scene identification is data subjected to at least one pooling process. The scene discrimination unit, an identification of the scene, and the intermediate data the scene identification, recognition apparatus according to claim 1 or claim 2 executes based on the comparison of the learned data by supervised learning. The correction unit further includes an information characteristic amount acquisition unit (S730) that acquires an information characteristic amount that is a characteristic amount corresponding to a parameter included in the input data from the narrowing intermediate data that is at least a part of the intermediate data. Prepare,
The correction execution unit, the correction, the acquisition information feature amount, from claim 1 to perform on the basis of a comparison between the learned information feature amount by supervised learning that the identified scene true value The recognition device according to claim 3 . The recognition device according to claim 4 , wherein the narrowing intermediate data is data obtained by performing convolution processing on the input data at least once. The intermediate data for narrowing the recognition device according to the input data, to claim 4 or claim 5 pooling process is data that has not been subjected. The correction unit further includes a calculation unit (S630) that calculates a positional relationship feature amount regarding a relative positional relationship between the labels in the initial recognition value.
The correction execution unit executes the correction based on a comparison between the acquired positional relation feature amount and the positional relation feature amount learned by supervised learning with the identified scene as a true value. The recognition device according to any one of claims 1 to 6 . The calculation unit calculates the positional relationship feature amount by using a plurality of patches configured by collecting labeled pixels included in the initial recognition value in a first size, and a label included in the initial recognition value. A plurality of receptors configured by collecting attached pixels with a second size larger than the first size is prepared, and a label included in each of the plurality of patches and a plurality of labels included in each of the plurality of receptors are provided. The recognition device according to claim 7, which is realized by deriving a relationship with a label. Claim wherein the correction execution unit, the said correction, the information features and claim 4 performed by Filter label as a correction candidate with the positional relationship characteristic amount, dependent on claim 5 or claim 6 The recognition device according to claim 7 or claim 8 . Further comprising a specifying unit (S400) for specifying an error area that is a closed area due to pixels whose reliability as the initial recognition value is less than a threshold value
The correction unit, the correct, the initial recognition either of said at least some of the labels applied to the pixels included in the specified error area claims 1 to run in the target to claim 9 as a value one The recognition device according to the item. The recognition device according to claim 10 , which is dependent on claim 8 , wherein the calculation unit calculates the positional relationship feature amount for the patch including the specified error region. 7. The correction execution unit applies the correction for a pixel that is a representative of each of the error regions to the correction for each of the error regions, and is dependent on any one of claims 4 to 6. The recognition device according to item 10 . The recognition device according to claim 12 , wherein the correction execution unit further includes a centroid pixel acquisition unit (S710) that obtains a pixel serving as a centroid of each of the error regions as the representative pixel. By inputting the information of each pixel acquired from the image to be recognized as input data to the deep neural network (13) including a plurality of intermediate layers, each pixel is labeled and the initial recognition value is acquired. ,
A scene to which the image to be recognized belongs is identified based on scene identification intermediate data that is at least a part of intermediate data output from at least one layer of the plurality of intermediate layers,
A program for causing a recognition device to correct the initial recognition value based on the identified scene .