JP6255125B2 - Image processing apparatus, image processing system, and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus , an image processing system, and an image processing method for generating a learning image for generating a dictionary used for image recognition processing for detecting an image of a target object from an input image.

  Conventionally, various research and development have been conducted on image recognition for detecting an image of a target object from an image obtained by imaging the target object. Image recognition technology has been applied to various fields and has been used for many real problems such as face recognition and parts recognition in factories.

  Such image recognition can be considered from the viewpoint of pattern recognition. Also in this pattern recognition, research is being made on a classifier that classifies input information as a class. As such research, for example, various methods such as a neural network, SVM (Support Vector Machine), and Randomized Tree have been proposed.

  In these methods, it is necessary to generate a dictionary for image recognition in advance, but a learning image is required when generating this dictionary. As visual recognition in recent industrial robots, there is a need to recognize a target object having a high degree of freedom in posture in three dimensions, such as a component picking process for detecting a desired component from a plurality of types of components stacked. In order to detect a three-dimensional posture in this way, learning images corresponding to various postures of the target object are required.

  As described above, in the recognition task for the purpose of picking a part by the robot, the posture information of the target object is extremely important. The posture corresponding to the learning image is expressed by parameters such as Euler angles and quaternions. However, it is difficult to prepare a learning image of a target object that makes such posture known as a live-action image. . Therefore, it is common to generate a CG image of an arbitrary posture based on CAD data and use it as a learning image.

JP 2010-243478 A

  As a conventional method for generating a learning image from CAD data, it is common to treat a joint portion of the CAD data represented by a polygon as an edge and generate a binarized edge image. In actual detection processing, edge extraction processing is performed on the live-action input image of the component group, and matching based on edge pixels is performed to identify the position and orientation of the target object. In such a method, the result of edge extraction processing on a real image greatly affects the detection performance. In general, the edge extraction process greatly varies due to the material of the target object, the influence of ambient light, and the like, and requires much labor for adjustment by the operator.

  On the other hand, a method of generating a learning image as an image close to a real image by rendering is also used. In this method, it is necessary to estimate what luminance value each surface of the target object has. If the BRDF (bidirectional reflectance distribution function) of the target object and the ambient light conditions are known, it is possible to generate CG by giving the estimated brightness value to the surface of the target object. It is. However, in order to accurately know the BRDF of the target object, measurement with special equipment is necessary, and it takes a considerable amount of work to accurately obtain the environmental light conditions in the actual environment as numerical values.

  There is also a method in which a sphere is placed in the environment to create an environment map related to the luminance value and the surface direction, and a learning image is generated according to the map. For example, in order to generate a learning image of a specular object, an environment map image is generated by placing a specular sphere in the environment (see Patent Document 1). However, for example, if the target object is an object made of a general material such as plastic, even if the material is the same, the reflection characteristics vary depending on the molding die and surface processing. It is very difficult to prepare a spherical object.

In view of the above problems, the present invention provides an image processing apparatus and an image processing system that easily generate a learning image that approximates the surface luminance of a target object by reflecting environmental conditions based on information obtained by photographing the target object in a real environment. And an image processing method.

As a means for achieving the above object, an image processing apparatus of the present invention comprises the following arrangement. That is, an image processing apparatus that generates a learning image of an object used to create a dictionary that is referred to in an image recognition process for detecting the object,
First acquisition means for acquiring a luminance value of the surface of the object in a plurality of regions of the luminance image from a luminance image including the plurality of objects having different postures ;
Second acquisition means for acquiring information relating to the orientation of the surface of the object in the plurality of regions from which the luminance values are acquired by the first acquisition means;
Correspondence relationship between the brightness value of the surface of the object in the plurality of areas acquired by the first acquisition means and information related to the orientation of the surface of the object in the plurality of areas acquired by the second acquisition means A third acquisition means for acquiring
A correspondence relationship acquired by the third acquisition means, based on the model information of the object, and generating means for generating a training image of the object,
It is characterized by having.

  According to the present invention, it is possible to easily generate a learning image that approximates the surface luminance of a target object by reflecting environmental conditions based on information obtained by photographing the target object.

In the image processing apparatus that performs the image recognition processing according to the first embodiment, in particular a block diagram showing a schematic configuration for generating a learning image, In the image processing apparatus of the first embodiment, a block diagram showing a schematic configuration for detecting a target object at runtime, A flowchart showing runtime processing and dictionary generation processing in the first embodiment, A diagram for explaining the surface properties of the target object, The figure which shows the example of observation of luminance distribution in the target object, The figure which shows the state of learning image generation with CG image, The figure which shows the example of observation of the luminance distribution in the target object which consists of plural colors, A block diagram showing a detailed configuration of a luminance estimation unit in the second embodiment, A flowchart showing details of luminance estimation processing in the second embodiment, The figure explaining the estimation method of the luminance distribution in 2nd Embodiment, The figure which shows the GUI example for the color classification specification in 2nd Embodiment, The figure which shows the example of matching of the luminance distribution function by color coding in 2nd Embodiment, A diagram showing an example of a predicted distribution of luminance values in the third embodiment, A block diagram showing a schematic configuration for generating a learning image in the fourth embodiment, 15 is a flowchart showing learning processing in the fourth embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention related to the scope of claims, and all combinations of features described in the present embodiments are essential for the solution means of the present invention. Is not limited.

<First Embodiment>
The present invention reflects a learning image used for generating a dictionary referred to in an image recognition process for detecting an image of a target object from an input image based on information obtained by photographing the target object in an actual environment and reflecting environmental conditions. It is generated so as to approximate the surface brightness of the target object.

Overview Configuration FIG. 1A shows an overview of a configuration for generating a learning image, in particular, in the image processing apparatus that performs image recognition processing in the present embodiment. The model setting unit 1010 sets the model of the target object and stores it in the model storage unit 1020. The image acquisition unit 1110 acquires a pre-acquired image by photographing the target object, and stores it in the pre-acquired image storage unit 1120. The observation data distribution acquisition unit 1130 acquires an observation data distribution of luminance values based on information obtained from the pre-acquired image stored in the pre-acquisition image storage unit 1120. The luminance estimation unit 1210 estimates the luminance distribution of the target object surface based on the data distribution obtained by the observation data distribution acquisition unit 1130. The image generation unit 1220 generates CG images of various postures of the target object based on the model stored in the model storage unit 1020 and the luminance distribution estimated by the luminance estimation unit 1210. The generated CG image is stored in the learning image storage unit 2010 as a learning image. The present embodiment is characterized in that a learning image based on the luminance distribution obtained by photographing the target object as described above is easily generated.

  A dictionary for image recognition is generated by learning processing using the learning image generated as described above. That is, the learning image setting unit 2020 reads a plurality of learning images stored in the learning image storage unit 2010, and the learning unit 2100 performs learning processing using these learning images to recognize the target object. Generate a dictionary. The generated dictionary is stored in the dictionary storage unit 2200.

  A runtime process is performed using the dictionary generated by the above configuration. Here, runtime refers to performing recognition (detection) processing of a target object on an actual input image using a dictionary obtained based on a learning image created according to the present embodiment. At runtime, the dictionary setting unit 3010 first reads a dictionary stored in the dictionary storage unit 2200 and inputs the read dictionary to the recognition unit 3100. On the other hand, the input image acquisition unit 3020 acquires an input image obtained by photographing the target object and inputs the input image to the recognition unit 3100. The recognition unit 3100 estimates the position and orientation of the target object for the input image according to the read dictionary. The position and orientation of the target object estimated by the recognition unit 3100 are presented as a recognition result by the recognition result output unit 3200 by a predetermined method.

Outline of Robot Work Hereinafter, an example in which the image processing apparatus according to the present embodiment configured as described above is applied to work by a robot will be described. First, FIG. 2 shows an apparatus configuration for detecting a target object at runtime. In FIG. 2, a target object 400 to be detected is arranged on a tray 500. Reference numeral 300 denotes an imaging device for obtaining image information and distance information from the shooting position, and corresponds to the input image acquisition unit 3020 in FIG. The imaging device 300 may be a stereo camera, a TOF sensor, or a device such as light cutting or spatial coding using a combination of a camera and a projector as long as distance information can be obtained together with image information at the time of shooting, and is limited in the present invention. It is not something. Further, as will be described later, when performing alignment with a model using a tracking technique, it is not necessary to obtain distance information, and the imaging apparatus 300 may be configured only by a camera. The imaging device 300 is connected to the computer 100 via a wired or wireless connection.

  In the computer 100, a configuration corresponding to the recognition unit 3100 and the recognition result output unit 3200 shown in FIG. 1 is incorporated as a program or a circuit. A hard disk (not shown) provided inside or outside of the computer 100 corresponds to the dictionary storage unit 2200 and stores a dictionary. Note that the recognition unit 3100, the recognition result output unit 3200, and the dictionary storage unit 2200 are not limited to the configuration described above. For example, it may be a computer or hard disk via a network, or a combination of a circuit incorporated in the camera and a flash memory.

  The computer 100 is connected to the robot controller 210. The robot controller 210 is connected to the robot arm 220, and the robot arm 220 operates in response to a command signal from the robot controller 210. The robot arm 220 is provided with an end effector 230 for performing a predetermined work such as a gripping work on a work object.

● Runtime Processing The runtime processing in the configuration shown in FIG. 2 will be described with reference to the flowchart of FIG. As described above, the runtime is to perform recognition (detection) processing of the target object on the actual input image using the dictionary based on the learning image created in the present embodiment. In the configuration example shown in FIG. 2, for example, the target object that is successively conveyed in an actual factory or the like is photographed to indicate the position and orientation of the object. However, the present invention is not limited to the runtime processing as described above. For example, when the present invention is applied to face recognition, a scene in which a camera is photographed toward a person and face recognition is performed. It can be said that this is a runtime process.

  In FIG. 3A, first in dictionary setting step S3010, the dictionary setting unit 3010 reads out a dictionary that has been generated in advance and stored in the dictionary storage unit 2200, and outputs it to the recognition unit 3100. Details of dictionary generation will be described later.

  In the input image acquisition step S3020, the target object 400 placed on the tray 500 is photographed by the camera 300, and the obtained image (luminance image) and distance information are sent to the computer 100. Next, in the recognition step S3100, the recognition unit 3100 performs image recognition processing on the input image using the dictionary read by the dictionary setting unit 3010, and estimates the position and orientation of the target object 400. The estimated position and orientation are sent to the recognition result output unit 3200 as a recognition result.

  The image recognition processing performed here is processing for classifying the position and orientation of the target object by a classifier, and the dictionary used at this time defines this classifier. The class classifier defined by the dictionary recognizes the position and orientation by determining to which class the target object shown in a part of the image applies. For example, a classifier based on a well-known technique such as SVM (Support Vector Machine) or RT (Randomized Trees) may be used. Further, as an input to the class classifier, data obtained by performing predetermined image processing on the input image may be input. Here, the image processing applied to the input image is a general term for processing for converting the input image into a format that can be easily handled by the classifier, and the processing content is not limited. For example, noise removal using a Gaussian filter, median filter, or the like, edge extraction processing using a Sobel filter, LoG filter, Laplacian filter, Canny edge detector, or the like can be considered. Further, preprocessing such as enlargement / reduction and gamma correction, and feature extraction processing such as HOG and SIFT are also considered as the image processing described here. Further, the image processing is not limited to the one that selectively performs only one of the above-described processes, and may be a series of combination processes in which edge extraction is performed using a Sobel filter after noise removal using a Gaussian filter, for example. .

  Next, in the recognition result output step S3200, the recognition result output unit 3200 encodes a command for the robot to perform a predetermined work from the estimated position and orientation of the target object 400, which is the recognition result, and outputs it to the robot controller 210. . The robot controller 210 decodes the sent command, operates the robot arm 220 and the end effector 230 in accordance with the command, and performs a predetermined work on the recognized work object (target object 400).

  If the recognition operation of S3100 is repeated at runtime, it is not necessary to repeat the dictionary setting step S3010 by holding the dictionary set in S3010 in a memory (not shown). That is, in this case, the processing after the input image acquisition step S3020 may be repeatedly executed.

Dictionary generation process (learning process)
In performing the runtime processing as described above, it is necessary to prepare a dictionary for detecting the target object 400 in advance. Hereinafter, processing for setting the dictionary will be described in detail. As described above, since the dictionary is reused in repeated runtime work, the dictionary generation process needs to be performed only once.

  The dictionary generation process in the present embodiment is performed by the configuration shown in FIG. Note that the image generation unit 1220, the learning unit 2100, and the recognition unit 3100 in FIG. 1 are all implemented as programs in the computer 100 shown in FIG. Also, description will be made assuming that the pre-acquired image storage unit 1120, the learning image storage unit 2010, the dictionary storage unit 2200, and the hard disk are mounted. The hard disk is connected to the inside or outside of the computer 100. However, the implementation form of the present invention is not limited to this example, and may be implemented in a computer or hard disk different from that used at runtime, or in a computer or memory installed in the camera. Further, the image acquisition unit 1110 is implemented as a program for controlling the imaging apparatus 300 in the imaging apparatus 300 or the computer 100 illustrated in FIG.

  Hereinafter, dictionary generation processing according to the present embodiment will be described with reference to the flowchart of FIG. First, in the model setting step S1000, the model setting unit 1010 sets the model of the target object 400 and stores it in the model storage unit 1020. Here, the model is model information including information necessary for generating a CG image of a target object as described later, and specifically refers to CAD data, a polygon model, and the like of the target object 400.

  Next, in the image acquisition step S1100, the target object 400 is placed on the tray 500, and the image acquisition unit 1110 takes an image with the imaging device 300, and the luminance image and distance information (distance image) of each pixel position in the luminance image. To get. The acquired luminance image and distance image are collectively written in the pre-acquired image storage unit 1050 as a pre-acquired image. Here, the pre-acquired image is an image used when the learning image is generated by the image generation unit 1220, and is captured under the same environmental conditions as at the time of runtime, that is, under the same environmental conditions as the input image acquisition step S3020. It is desirable. For example, it is desirable that the illumination condition at the time of capturing the pre-acquired image is substantially the same as that in S3020. Moreover, it is desirable that a plurality of target objects photographed as pre-acquired images be stacked in a random manner. In addition, at least one image is captured as a pre-acquired image, and about 5 images are captured in the present embodiment. Thus, when a plurality of pre-acquired images are captured, it is desirable that the arrangement of the target object 400 is different in each capturing state, that is, there are many variations in position and orientation. It is ideal to use the same imaging device as that used at the time of shooting for pre-acquired images, but if the positional relationship and lighting conditions between the imaging device and the tray are similar, use another imaging device. A pre-acquired image may be acquired. Further, as a pre-acquired image, a single target object may be photographed in various postures. In that case, it is desirable to photograph a larger number of images (for example, about 20 images) than when photographing in a stacked state.

  Next, in the observation data distribution acquisition step S1130, an observation data distribution indicating a luminance value distribution is acquired based on the pre-acquired image acquired in S1100. In the pre-acquired image stored in the pre-acquired image storage unit 1120, it is assumed that the camera coordinate system position is observed as a distance image at (Xj, Yj, Zj) for an arbitrary pixel j in the luminance image. Here, the camera coordinate system is a three-dimensional coordinate system corresponding to each pixel of the luminance image, and indicates an imaging space indicated by each axis of XYZ with the camera 300 as the origin. At this time, by summing up the camera coordinate system positions of the pixel j and its neighboring points (for example, a total of 9 pixels including the pixel j and the adjacent 8 pixels), the camera coordinate system position ( The surface normal vector Nj of Xj, Yj, Zj) is calculated. Observation data indicating the correspondence between the luminance value and the surface normal direction by calculating the corresponding surface normal vector for all pixels in the existing area of the target object 400 (for example, the inner area of the tray 500) in the pre-acquired image Distribution can be obtained. Here, although the observed luminance value is described as each pixel value, it is needless to say that it is generally a luminance value at a predetermined position on the image. Therefore, the observed luminance value does not have to be a luminance value in a single pixel, but may be an average value as a local region by a number of neighboring pixels, a value after passing through a noise removal filter, or the like.

  Next, in the luminance estimation step S1210, the surface luminance distribution of the target object 400 is estimated based on the observation data distribution obtained from the previously acquired image. In order to generate a learning image by CG from CAD data of the target object 400, it is necessary to model the surface luminance distribution of the target object 400 and estimate a parameter (luminance distribution parameter) in the surface luminance distribution model.

  Here, specific examples of the luminance distribution parameter are as follows. For example, assuming that the light source is a single parallel light and the surface of the part as the target object 400 is Lambertian reflection (diffuse reflection), it can be approximated by a relatively simple luminance distribution model. An example of this approximation will be described with reference to FIG. FIG. 4 shows a state in which the target object 400 is irradiated with light from the light source 600, and the camera 300 receives the reflected light. The light source direction L = (Lx, Ly, Lz) in the camera coordinate system of the light source axis 20 from the surface of the target object 400 toward the light source 600 and the camera optical axis direction V = (Vx, Vy, Vz) of the camera optical axis 10 The middle is the reflection center axis 30. Then, the direction vector H = (Hx, Hy, Hz) of the reflection center axis 30 is expressed by the following equation (1).

H = (L + V) / | L + V | (1)
When θ is an angle 50 formed by the normal vector N = (Nx, Ny, Nz) of the surface normal 40 at an arbitrary position on the surface of the target object 400 and the direction vector H of the reflection central axis 30, θ is (2).

θ = cos ^-1 (H ・ N / (| H || N |)) (2)
At this time, the luminance value J at an arbitrary position on the surface of the target object can be approximated by a Gaussian function as the following equation (3) as a function of θ.

J (θ) ＝ Cexp (-θ ² / m) (3)
Note that C and m in Equation (3) are luminance distribution parameters representing the intensity of the entire luminance distribution and the spread of the luminance distribution, respectively, and approximation of the luminance distribution model is performed by estimating this.

  Here, since it is assumed that the light source is single, the surface normal vector Nj of the pixel having the maximum luminance value among the obtained observation values is the light source direction L = (Lx, Ly, Lz) It is estimated that. At this time, it goes without saying that luminance values may be averaged by neighboring pixels in consideration of observation errors, luminance saturation, and the like. Of course, when the light source direction L is known, it is not necessary to estimate the light source direction.

  When approximating the luminance distribution with a Gaussian function like the above equation (3), the direction vector H of the reflection central axis is first calculated according to the above equation (1) with respect to the light source direction L obtained as described above. . Thereby, the angle θj from the reflection center axis in each pixel j is obtained from the above equation (2). Hereinafter, a set of the angle θj and the luminance value Jj in the pixel j is represented as an observation point pj = (θj, Jj). By calculating the observation points pj for all the pixels j, an observation distribution as shown in FIG. 5 can be obtained. In FIG. 5, a data point B100 is an observation point pj of the angle θj and the luminance value Ij.

  An estimated model of the surface luminance distribution of the target object 400 can be obtained by performing maximum likelihood estimation fitting on the observed distribution shown in FIG. Specifically, first, the error function E is defined as the sum of squares of the difference between the estimated value and the observed value as shown in the following equation (4). In the equation (4), Σ represents the total sum for j.

E = Σ {J (θj) −Jj} ² (4)
The maximum likelihood estimation fitting is considered as a minimization problem of the error function E. Then, since the error function E is a quadratic function convex downward with respect to the parameter C, an equation for updating the parameter C can be obtained as in equation (6) by solving the following equation (5).

      ∂E / ∂C = 0 (5)

  The parameter m is solved as an optimization problem of γ by setting γ = 1 / m to simplify the calculation. Here, since the error function E is not a convex function with respect to γ, the error function E is decomposed for each data as shown in the following equation (7) and solved for each.

Ej = {J (θj) −Jj} (7)
When equation (7) is solved in a steepest descent, the sequential update equation becomes the following equation (8), which is called the Robbins-Monro procedure. Note that the coefficient η in equation (8) is a constant defined by a positive value, and is generally given as the reciprocal of the number of observation data.

  As described above, when the surface of the target object is diffusely reflected, the luminance distribution parameter C, m is estimated to allow approximation with the luminance distribution model using the Gaussian function. On the other hand, when the specular reflection component on the surface of the target object is considered, a Torrance-Sparrow luminance distribution model as shown in the following equation (9) may be applied.

J (θ, α, β) = K _d cosα + K _s (1 / cosβ) exp (-θ ² / m) (9)
In Equation (9), K _d , K _s , and m are luminance distribution parameters in this model. When this model is applied to FIG. 4, θ is the same as θ in the equation (2), that is, an angle 50 formed by the direction vector N of the surface normal 40 and the direction vector H of the reflection central axis 30. Α is the angle 70 formed by the direction vector N of the surface normal 40 and the direction vector L of the light source axis 20, and β is the angle 60 formed by the direction vector N of the surface normal 40 and the direction vector V of the camera optical axis 100. Are expressed by the following equations (10) and (11), respectively.

α = cos ^-1 (L ・ N / (| L || N |)) (10)
β = cos ^-1 (V ・ N / (| V || N |)) (11)
The angles αj and βj in the equation (9) corresponding to each observation pixel j can be obtained from the above equations (10) and (11), whereby the observation distribution of the luminance value Jj corresponding to θj, αj, βj is obtained. Can be obtained. An estimated model of the surface luminance distribution of the target object 400 can be obtained by fitting the model of equation (9) to the observed distribution by maximum likelihood estimation.

  When there are multiple light sources or ambient light such as ambient light, the brightness distribution is approximated by a nonparametric regression model J (N) with the surface normal direction vector N as input and the brightness value J as output. Also good. A luminance distribution estimation function is obtained by learning a predetermined nonparametric model with the luminance value Jj as a teacher value for the surface normal direction vector Nj for each pixel j in the observed value. As the nonparametric regression model, various methods such as SVM, SVR (Support Vector Regression), and neural network can be used. When using these non-parametric models, it is not necessary to estimate the light source direction in advance before fitting.

  Also, given the camera coordinate system position (X, Y, Z) as an argument of the regression model and approximating J (N, X, Y, Z), the luminance distribution estimation function considering the difference in lighting conditions depending on the position You can also get In addition, when the luminance values are obtained with multiple channels, the luminance distribution for each channel may be estimated separately. The case of multi-channel is a case where, for example, an RGB color image or an invisible light image using infrared light or ultraviolet light is included as additional information.

  As described above, when the surface luminance of the target object 400 is estimated in the luminance estimation step S1210, a plurality of learning images necessary for generating a dictionary are set in the next image generation step S1220. The learning image is generated as a CG image based on the model (for example, CAD data) of the target object 400 set in S1000. For example, if the surface properties of the target object 400 represented by BRDF and the light source information in the work environment are known, the appearance of the target object 400 in various postures can be reproduced from the model as a CG image using a known rendering technique. Can do. Here, FIG. 6 shows a learning image generated from a CG image. As shown in FIG. 6, a learning image is generated with each viewpoint 403 in the geodetic dome 401 centered on the object center 404 of the target object 400 and a variation by the in-image rotation 402 at the viewpoint 403, and each has a posture class. An index is given. For example, when learning images are generated with variations of 72 viewpoints and in-plane rotation every 30 degrees, 72 × (360/30) = 864 classifiers are learned as a dictionary. Specifically, projective transformation corresponding to each of the above postures is performed on the model of the target object 400 stored in the model storage unit 1020, and the method of the point on the model corresponding to each pixel after the projection transformation is performed. The line direction (surface normal direction) is calculated. Then, according to the result obtained in the luminance estimation step S1210, a learning image corresponding to each posture is generated by giving a luminance value corresponding to the normal direction.

  Next, in the learning step S2000, a dictionary is generated according to the class classifier format used in the recognition unit 3100, using the learning images of a plurality of postures generated in the image generation step S1220. The generated dictionary is stored in the dictionary storage unit 2200.

  As described above, according to the present embodiment, environmental conditions such as lighting are reflected based on luminance information and distance information obtained by photographing a target object in a piled state or a single product state in a plurality of postures in an actual environment. Thus, a luminance image approximating the surface luminance of the target object is easily generated. This approximated luminance image is used as a learning image when generating a dictionary.

<Modification>
In the first embodiment, an example in which a luminance image and a distance image including a target object are acquired by photographing with the imaging device 300 has been described, but the present embodiment also applies when the imaging device 300 does not have a ranging function. Is applicable. Hereinafter, a modification example in which the first embodiment is applied when a distance image cannot be acquired by the imaging apparatus 300 will be described. In this modification, the distance image is generated by estimating the position and orientation of the target object using a known tracking technique. FIG. 1 (b) shows a configuration for learning image generation in the present modification, but the same number is assigned to the same configuration as in FIG. 1 (a), and the description is omitted. That is, the distance image is generated by the distance image generation unit 1140 in FIG. In this case, the user sets the approximate position and orientation by superimposing the model of the target object on the image, and matches the detailed position and orientation using the existing tracking technology. Include. By calculating the estimated position in the camera coordinate system of each pixel related to the region where the target object is captured from the correspondence with the fitted model, it can be handled as an estimated distance image.

<Second Embodiment>
Hereinafter, a second embodiment according to the present invention will be described. The first embodiment described above has been described on the assumption that the target object is a single color object. However, there are of course cases where the target object is composed of a plurality of colors. For example, when a component supplied as an assembly of a plurality of components is a target object, the luminance characteristics may be different depending on the part, such that a part of the target object is made of black plastic and part of it is made of white plastic. It is done. In the second embodiment, an example in which a learning image is generated for a target object having a plurality of luminance characteristics will be described.

  The basic configuration for performing image recognition processing in the second embodiment is the same as that shown in FIG. 1A shown in the first embodiment. In particular, the operation of the model setting unit 1010, the model storage unit 1020, the image acquisition unit 1110, the pre-acquired image storage unit 1120, and the observation data distribution acquisition unit 1130 is the same as that of the first embodiment, and thus description thereof is omitted. . In the second embodiment, the details of the processing in the luminance estimation unit 1210 that estimates the surface luminance of the target object from the observation data distribution are different from those in the first embodiment. FIG. 8 shows a detailed configuration of the luminance estimation unit 1210 in the second embodiment, which will be described below. Note that the processing after the image generation unit 1220 and the processing at runtime are the same as in the first embodiment, and thus description thereof is omitted.

  The luminance estimation unit 1210 of the second embodiment is subdivided into an initialization unit 1211, a data allocation unit 1212, an approximation unit 1213, and a convergence determination unit 1214. First, a plurality of functions that approximate the luminance distribution are initialized by the initialization unit 1211 with respect to the input observation data distribution, and then the observation data distribution is converted into one of the plurality of functions by the data allocation unit 1212. assign. Then, the approximation unit 1213 fits the luminance distribution function to the assigned observation data distributions, and the convergence determination unit 1214 determines whether the luminance distribution estimation calculation has converged.

Dictionary generation process (learning process)
Also in the second embodiment, as in the first embodiment, a dictionary for detecting a target object is generated from a learning image generated as described later. The outline of the dictionary generation process in the second embodiment is the same as the flowchart of FIG. 3B shown in the first embodiment, but the details are different. In FIG. 3B, the model setting step S1000, the image acquisition step S1100, and the observation data distribution acquisition step S1130 are the same as those in the first embodiment, and thus the description thereof is omitted.

  In the luminance estimation step S1210 of the second embodiment, the surface luminance of the target object is estimated based on the observation data distribution obtained from the previously acquired image. First, an observation data distribution relating to the correspondence between the luminance value and the surface normal direction is obtained from the image stored in the pre-acquired image storage unit 1120. Here, an approximate example of the luminance distribution model will be described with reference to FIG. 4 on the assumption that the light source is a single parallel light and the surface of the component undergoes Lambertian reflection.

  In FIG. 4, an intermediate between the light source direction L in the camera coordinate system of the light source axis 20 from the surface of the target object 400 toward the light source 600 and the camera optical axis direction V of the camera optical axis 10 is defined as a reflection center axis 30. Then, the direction vector H of the reflection center axis 30 is expressed by the equation (1) as in the first embodiment. Further, the angle θ formed by the normal vector N of the surface normal 40 and the direction vector H of the reflection center axis 30 is also expressed by equation (2), as in the first embodiment. At this time, it is assumed that the surface luminance characteristic of the target object 400 is represented by a combination of T types of characteristics. The t-th (t = 1,..., T) luminance distribution function Jt (θ) can be approximated by a Gaussian function as shown in the following equation (12).

Jt (θ) ＝ Ct ・ exp (-θ ² / mt) (12)
Note that Ct and mt in equation (12) are parameters representing the intensity of the entire luminance distribution and the spread of the luminance distribution, respectively.

  The luminance distribution characteristic of the target object 400 is approximated by T luminance distribution functions Jt (θ) (t = 1,..., T). An example of the luminance distribution function when T = 2 is shown in FIG. In FIG. 7, curves B210 and B220 are curves based on the luminance distribution function estimated at t = 1 and 2, respectively. Note that the value of T can be determined in advance as the number (fixed value) if the number that can be decomposed as parts of different materials is known beforehand by CAD data or the like. In the following description, it is assumed that T is set in advance, but processing when T is unknown will also be described later.

  In order to estimate the parameters Ct, mt of each luminance distribution function Jt (θ), the luminance estimation step S1210 in the second embodiment is subdivided as shown in FIG. 9 (a). FIG. 10 shows a conceptual diagram of this parameter estimation. Hereinafter, the procedure of parameter estimation processing will be described with reference to these drawings.

  First, in initialization step S1211, parameters Ct and mt are initialized. Specifically, the initial value may be selected at random, or after estimating maximum likelihood with T = 1, a different value slightly shifted from the estimation result is set as the initial value of a plurality of Ct and mt. , Etc. are conceivable.

  Next, in the data assignment step S1212, each observation point pj = (θj, Jj) is assigned to each closest brightness distribution function Jt (θ). Specifically, for example, among the plurality of luminance distribution functions Jt (θ), the estimated luminance value obtained when the surface normal direction of the observation point is input is the luminance distribution function closest to the observation point luminance value. The observation point may be assigned. That is, a data set St for estimating the luminance distribution function Jt (θ) is defined as the following equation (13).

  Equation (13) corresponds to labeling each observation point with the index of the luminance distribution function. An example of this is shown in FIG. In the figure, curves B210-a and B220-a are luminance distribution functions initialized with different parameters. On the other hand, each observation point is assigned to the closest luminance distribution function. That is, white circles such as point B110-a are observation point groups assigned to the curve B210-a, and black circles such as point B120-a are observation point groups assigned to the curve B220-a.

  Next, in the approximation step S1213, each observation point group St assigned to each luminance distribution function is used to update each luminance distribution function Jt (θ) by maximum likelihood estimation fitting. An example of this is shown in FIG. Curves B210-b and B220-b in the figure represent curves obtained by updating the curves B210-a and B220-a shown in FIG. 10 (a) with observation point groups assigned to themselves.

  Then, after updating all (two in this case) luminance distribution functions Jt (θ), the luminance distribution function closest to each observation point pj is specified again. If the luminance distribution function specified for all the observation points pj is the same as the luminance distribution function already assigned to each observation point, it is assumed that the luminance estimation step S1210 has converged and the next image generation step S1220 Proceed to On the other hand, if there is an observation point where the specified luminance distribution function is different from the already assigned luminance distribution function, the process returns to the data allocation step S1212 and the above processing is repeated.

  Here, when the specular reflection model is incorporated in the luminance distribution of the target object, the Torrance-Sparrow luminance distribution model is applied as in the first embodiment. In that case, the t-th luminance distribution function Jt (θ, α, β) may be approximated as in the following equation (14).

Jt (θ, α, β) = K _dt cosα + K _st (1 / cosβ) exp (-θ ² / mt) (14)
In Equation (14), K _dt , K _st , and mt are parameters in this model, and α and β are expressed by Equations (10) and (11) above. These parameters are also estimated by function fitting as in the first embodiment. Even when the luminance values are obtained with multiple channels, the luminance distribution for each channel may be estimated separately as in the first embodiment.

  The case where the number T of the luminance distribution functions constituting the luminance distribution characteristic of the target object is known has been described above (T = 2 in the above example). Hereinafter, an estimation example in the case where T is unknown will be described. When T is unknown, first, estimation is performed by setting a plurality of T, and among them, the one with the best separation is selected. The process in the luminance estimation step S1210 in this case will be described with reference to FIG. In FIG. 9 (b), the processing contents in the initialization step S1211, the data allocation step S1212, and the approximation step S1213 are the same as those in FIG. 9 (a). That is, in FIG. 9B, the processing of S1211 to S1213 similar to FIG. 9A is performed for a plurality of T variations T = 1,..., Tmax, and then the separation degree evaluation step S1214 is performed. It is characterized by. Tmax is the upper limit number of colors constituting the target object, and for example, Tmax = 5 is set.

In the separation degree evaluation step S1214, the separation evaluation value λ _T for each T is defined as in the following equations (15), (16), and (17).

In the equation (17), εt is a square error with respect to the estimated value, and ζt in the equation (16) represents a skewness centered on the estimated value. As the data set St assigned to each luminance distribution function Jt becomes closer to a normal distribution with respect to Jt, the value of the skewness ζt approaches 0. Therefore, in this case, the value of _{T with} the smallest separation evaluation value λ _T is set as the estimated value of the number T of luminance distribution functions.

  As described above, even if the number T of the luminance distribution functions of the target object is known or unknown, if the luminance distribution function is estimated in the luminance estimation step S1210, then in the image generation step S1220, A learning image based on the estimated luminance distribution function is generated. The learning image generation process in the second embodiment will be described below. In generating a learning image, first, it is necessary to associate a luminance distribution function with each part of the target object. This association can be performed by the following method.

For example, the correlation can be automatically performed by comparing the diffuse reflection components of the luminance distribution. In the approximate expression shown in the above equation (12), the luminance distribution function corresponding to each of the bright material portion and the dark material portion is determined depending on the luminance value in the portion where the value of θ is large (for example, θ = 1 rad). Judgment and association can be made. Further, in the approximate expression shown in the above expression (14), the parameter K _dt represents the intensity of the diffuse reflection component, and therefore the luminance distribution function may be associated with the magnitude thereof.

  Further, when a multi-channel luminance distribution function is estimated for a color image or the like of the target object, the diffuse reflection components of characteristic channels may be compared. For example, when a red part and a green part are associated, they can be easily associated by comparing the intensity of diffuse reflection components in the R channel and the G channel.

  Further, the user may designate a plurality of points in the pre-acquired image, and the association may be performed by detecting the luminance distribution function contributed by the pixel corresponding to the designated point. For example, as shown in FIG. 11, a pre-acquired image T500 indicating a piled state of target objects is displayed on the GUI, and a part T100 that is a part of a bright luminance part and a part T200 that is a part of a dark luminance part are displayed with a cursor. Specify with T300. By this designation, the luminance distribution functions are associated as shown in FIG. That is, it is assumed that the observation data at the designated part T100 is C100 in FIG. 12, and the observation data at the part T200 is C200. In FIG. 12, it can be determined from the above equation (13) that the observation data C100 and C200 belong to the luminance distribution functions indicated by the curves B210 and B220, respectively.

  In the image generation step S1220, if each part of the target object is associated with the luminance distribution function as described above, a learning image can be generated thereafter as in the first embodiment. In addition, since the process which produces | generates a dictionary using a learning image in subsequent learning process S2000 is the same as that of 1st Embodiment, description is abbreviate | omitted.

  As described above, according to the second embodiment, even when the target object surface is composed of a plurality of colors, a learning image approximating the surface luminance can be generated.

<Third embodiment>
The third embodiment according to the present invention will be described below. In the first and second embodiments described above, the surface texture of the target object has been described as being stable with respect to the surface normal direction. However, the surface texture is not necessarily stable. For example, if the surface of the target object is processed like a satin finish, the luminance value changes depending on the portion even on the surface of the target object facing the same direction. Even if there is no intentional surface treatment, the luminance value may change in the same manner depending on the roughness of the mold at the time of molding. In order to reproduce the instability of the luminance value of such a target object with unstable surface normality, it is conceivable to add noise at the time of learning image generation. That is, the third embodiment is characterized in that the luminance distribution of the target object is estimated in consideration of noise added at the time of learning image generation.

Dictionary generation process (learning process)
Also in the third embodiment, as in the first embodiment, a dictionary for detecting the target object is generated from a learning image generated as described later. This dictionary generation processing is also a flowchart of FIG. 3 (b). Follow. In FIG. 3B, the model setting step S1000, the image acquisition step S1100, and the observation data distribution acquisition step S1130 are the same as those in the first embodiment, and thus the description thereof is omitted.

  In the luminance estimation step S1210 of the third embodiment, the surface luminance of the target object is estimated as follows based on the observation data distribution obtained from the previously acquired image.

  Consider that the luminance distribution is expressed by a linear Gaussian kernel model y (θ, w) as shown in the following equations (18), (19), and (20).

y (θ, w) = w ^T Φ (θ) (18)
w = (w ₁ ,…, w _h ,…, w _M ) ^T (19)
Φ (θ) = (φ ₁ (θ), ..., φ _h (θ), ..., φ _M (θ)) ^T ... (20)
Here, θ is an angle 50 formed by the direction vector N of the surface normal 40 described in FIG. 4 and the direction vector H of the reflection central axis 30. M is the number of kernels and is defined by the user. Since the angle θ falls within the range of 0 to 90 deg, for example, if M = 9 is set, the kernels can be arranged at intervals of about 10 deg. The parameter w is an M-dimensional vector, and the element w _h (h = 1,..., M) is a positive real value. The vector φ is an M-dimensional vector, and its element φ _h (h = 1,..., M) is a Gaussian kernel defined as in equation (21).

φ _h (θ) = exp {-(θ-μ _h ) / 2S ² } (21)
Here, mu _h represents the central position of the Gaussian kernel phi _h. The kernel center μ _h may be arranged within the definition area of the angle θ. For example, when M = 9 is defined, μ _h may be set every 9 degrees.

  When the luminance distribution is approximated by such a linear Gaussian kernel model, a predicted luminance distribution defined as the following equation (22) is considered.

p (J | R, θ) = ∫p (J | w, θ) p (w | R, θ) dw (22)
Here, R is a set vector of observed luminance values, and when the total number of observed pixels is N, it is represented by a column vector as shown in the following equation (23).

R = (J ₁ , ..., J _j , ..., J _N ) ^T ... (23)
Here, J _j is an observed value of the luminance value at pixel j of the observed data.

  The first term on the right side of the above equation (22) is a conditional distribution of luminance values, and is represented by a normal distribution as shown in the following equation (24).

p (J | w, θ) = N (J | y (θ, w), ε ² ) (24)
Here, ε is an accuracy parameter, and a mean square error between the estimated luminance function J (θj) and the observed luminance value Jj in the first and second embodiments as shown in the following equation (25) is used. In the equation (25), Σ represents the total sum for j.

ε ² = (1 / N) Σ {Jj-J (θj)} ² ... (25)
The above equation (24) is considered as a likelihood function of the weight w, and this conjugate prior distribution is assumed to be a Gaussian distribution having an expected value m ₀ and a covariance S ₀ as shown in the following equation (26).

p (w) = N (w | m ₀ , S ₀ ) (26)
At this time, the second term on the right side of the posterior distribution (22) can be represented by a normal distribution as shown in the following expressions (27), (28), and (29).

p (w | R, θ) = N (w | m _N , S _N ) (27)
m _N = S _N (S ₀ ^-1 m ₀ + 1 / ε ² Φ ^T R) (28)
S _N ^-1 = S ₀ ^-1 + 1 / ε ² Φ ^T Φ (29)
Here, Φ is called a design matrix and is determined from the kernel and observation data as shown in the following equation (30).

  At this time, if the linear Gaussian kernel model of the above equation (18) is approximated along the least square method, the predicted luminance distribution of the equation (22) is finally expressed as the following equations (31) and (32): It has been known.

p (J | R, θ) = N (J | m _N ^T φ (θ), σ _N ² (θ)) (31)
σ _N ² (θ) = ε ² + φ (θ) ^T S _N φ (θ) (32)
Equation (32) is the variance of the predicted luminance distribution, and σ _N (θ), which is the square root, is the standard deviation of the predicted luminance distribution.

  As described above, when the predicted luminance distribution of the target object is estimated in the luminance estimation step S1210, in the next image generation step S1220, a learning image based on the estimated predicted luminance distribution is generated. As in the first embodiment, the generation of the learning image in the third embodiment calculates a luminance value with respect to the normal direction on the model corresponding to each pixel when the CAD model in each posture is projectively transformed, and the pixel A learning image is generated by giving a luminance value to. Specifically, with respect to the angle θk formed by the normal direction of the surface projected onto the pixel k in the generated learning image and the reflection central axis, the predicted distribution p (J | R, θk) can be obtained.

Here, FIG. 13 shows an example of a predicted distribution of luminance values obtained in the third embodiment. In FIG. 13, the solid line B300 is the center of the prediction distribution in the variable θ, and the broken line B310 and the broken line B320 are the prediction distributions represented by the standard deviation σ _N (θ) obtained by giving θ as an argument to the above equation (32). Indicates the width. A curve B330 shows the predicted distribution in the luminance direction at θk, and its width B340 is the standard deviation σ _N (θ). In the third embodiment, a learning image can be generated by generating a random number according to the predicted luminance distribution and giving the obtained value to the pixel k.

  As described above, according to the third embodiment, the luminance value for each pixel on the target object surface is determined from the variance of the luminance distribution estimated for the target object, thereby taking into account variations in the surface luminance of the target object. Learning images can be generated.

<Fourth embodiment>
The fourth embodiment according to the present invention will be described below. In the first embodiment described above, an example has been shown in which the luminance of the surface of the target object is approximated by the luminance distribution model according to the above equations (3) and (9). In the fourth embodiment, a plurality of parameter candidates are prepared as parameters (luminance distribution parameters) for the luminance distribution model. An example in which a dictionary for recognizing a target object is created based on each parameter candidate, and an optimal parameter candidate is selected using a recognition result obtained by applying the dictionary to an input image obtained by actual shooting as an evaluation value. Here, the luminance distribution parameter is C and m in equation (3) or K _d , K _s , m in equation (9).

  FIG. 14 shows a basic configuration for performing image recognition processing in the fourth embodiment. In the same figure, the same components as those in FIG. 1A shown in the first embodiment are denoted by the same reference numerals, and description thereof is omitted. That is, the fourth embodiment is characterized in that a parameter setting unit 1230 and a selection unit 2110 are added except for the configuration for luminance estimation from the configuration shown in FIG.

Dictionary generation process (learning process)
The dictionary generation process in the fourth embodiment follows the flowchart of FIG. In FIG. 15A, in the model setting step S1000 and the image acquisition step S1100, the CAD model of the target object and a plurality of pre-acquired images are set as in the first embodiment.

  In an image generation parameter candidate setting step S1215, K pattern candidates for image generation parameters for generating a learning image are prepared. Here, the image generation parameter is a luminance distribution parameter estimated in the first embodiment described above.

  Then, in the image generation step S1220, learning images corresponding to the prepared K pattern image generation parameter candidates are generated by the same method as in the first embodiment. Hereinafter, among all the K patterns, a set of learning images having various postures generated by the kth image generation parameter candidate is referred to as a learning image set Sk. In the learning step S2000, K dictionaries are generated using each of the K learning image sets Sk (k = 1,..., K).

  In the selection step S2110, evaluation is performed on all the pre-acquired images set in S1100 using the generated K dictionaries, and optimal dictionaries and image generation parameter candidates are selected based on the evaluation results.

  Here, details of the selection step S2110 will be described with reference to the flowchart of FIG.

  First, in the recognition step S2111, similarly to the recognition process at runtime described in the first embodiment (S3100), the recognition process using the dictionary is performed on the pre-acquired image, and the position of the target object in the pre-acquired image and Calculate the estimated value of the posture.

  Next, in the evaluation step S2112, the recognition result obtained in the recognition step S2111 is evaluated as follows. First, a CG image of the target object is generated from the model set in S1000 based on the estimated position and orientation obtained as a recognition result. At this time, a CG image may be directly generated for the estimated position and orientation, but a CG image may be generated for a result of more detailed matching using a well-known tracking technique. That is, the estimated position and orientation of the target object obtained as a recognition result are used as initial values, and the estimated position and orientation after matching in advance with a pre-acquired image using a known tracking technique are used to calculate CG. It is also possible to generate an image. Then, the generated CG image and the pre-acquired image are binarized by edge extraction processing, and the distance is calculated by comparing the edge point positions obtained by both, and the sum or square sum is detected. The error is calculated as an evaluation value. That is, in the evaluation step S2112, the evaluation value of the recognition result is calculated based on the difference in the corresponding part between the model image (CG image) generated from the recognition result and model information and the image of the target object in the pre-acquired image. .

  Alternatively, evaluation using a distance residual may be performed using a distance image. That is, based on the estimated position and orientation obtained as a recognition result, the distance of the part surface at the position and orientation is calculated from the model. Then, by comparing with distance information corresponding to the pre-acquired image, an error due to the sum of the residual distance or the sum of squares on the object surface is calculated as an evaluation value.

  Alternatively, the similarity between the learning image and the input image may be evaluated. In that case, the similarity between the CG image generated based on the estimated position and orientation in the recognition result and the existing area of the target object in the pre-acquired image is compared using normalized correlation etc., and the similarity is calculated as an evaluation value. .

  Furthermore, the user may visually check the generated CG image and the input pre-acquired image and evaluate the difference. For example, an error of positional deviation or posture deviation is defined in several stages (for example, about five stages), and the user inputs subjectively as an evaluation value. Moreover, it is good also as one evaluation value by the combination of the said various evaluation values, for example, a linear sum.

  In the evaluation step S2112, when evaluation values using K dictionaries are obtained for each pre-acquired image as described above, the value obtained by accumulating the values obtained from all the pre-acquired images is evaluated for each dictionary. And

  Then, in the dictionary selection step S2113, the dictionary having the best evaluation value calculated in S2112 is selected as the optimum dictionary. Here, a good evaluation value is a smaller value if the evaluation value is a detection error such as an edge error or a distance residual, and a larger value if the evaluation value is a similarity, and depends on the definition of the evaluation value.

  In the image generation parameter selection step S2114, the image generation parameter candidate used when generating the dictionary selected in the dictionary selection step S2113 is selected as an optimal image generation parameter for generating a learning image. That is, the image generation parameter candidate used when generating the learning image set Sk corresponding to the selected dictionary in the image generation step S1220 is selected.

  As described above, according to the fourth embodiment, it is possible to determine a parameter (luminance distribution parameter) for generating an optimal learning image for generating an optimal dictionary by evaluating an actual recognition result. . Therefore, by creating an optimal dictionary using the learning image generated based on the determined parameters, optimal recognition processing using the dictionary can be performed.

<Other embodiments>
In the first to fourth embodiments described above, an input image at runtime may be used as a pre-acquired image. By doing so, it becomes possible to dynamically generate a learning image that is dynamically suitable for environmental changes during runtime.

  Further, the present invention provides a storage medium storing a program code of software for realizing the functions of the above-described embodiments (for example, processing shown by a flowchart in which processing of each unit described above is associated with each process), a system or an apparatus It can also be realized by supplying to. In this case, the function of the above-described embodiment is realized by the computer (or CPU or MPU) of the system or apparatus reading and executing the program code stored in the storage medium so that the computer can read it.

  1120: Prior acquisition image storage unit 1110: Image acquisition unit 1020: Model storage unit 1010: Model setting unit 1130: Observation data distribution acquisition unit 1210: Luminance estimation unit 1220: Image generation unit 2010: Learning image storage unit 3010: Dictionary setting unit 2200: Dictionary storage unit 2100: Learning unit 2020: Learning image setting unit 3020: Input image acquisition unit 3100: Recognition unit 3200: Recognition result output unit

Claims (26)

An image processing apparatus for generating a learning image of an object used for creating a dictionary referred to in an image recognition process for detecting the object,
First acquisition means for acquiring a luminance value of the surface of the object in a plurality of regions of the luminance image from a luminance image including the plurality of objects having different postures ;
Second acquisition means for acquiring information relating to the orientation of the surface of the object in the plurality of regions from which the luminance values are acquired by the first acquisition means;
Correspondence relationship between the brightness value of the surface of the object in the plurality of areas acquired by the first acquisition means and information related to the orientation of the surface of the object in the plurality of areas acquired by the second acquisition means A third acquisition means for acquiring
A correspondence relationship acquired by the third acquisition means, based on the model information of the object, and generating means for generating a training image of the object,
An image processing apparatus comprising: The first acquisition means acquires a plurality of luminance images including a plurality of the objects having different postures, acquires a luminance value of the surface of the object in a plurality of regions from each luminance image,
The second acquisition means acquires information relating to the orientation of the surface of the object in the plurality of regions for each of the luminance images,
The third acquisition unit is configured to acquire, for each of the luminance images, a correspondence relationship between information relating to the orientation of the surface of the object in the plurality of regions and a luminance value corresponding to each region. The image processing apparatus according to claim 1. The second acquisition unit acquires distance information with respect to the object , and acquires information regarding the orientation of the surface of the object in the plurality of regions based on the acquired distance information. Or the image processing apparatus of 2. The second acquisition means acquires distance information of each pixel of the luminance image, and acquires information related to the orientation of the surface of the object in the plurality of regions based on the acquired distance information. The image processing apparatus according to claim 3. Luminance image including a plurality of objects having the different posture, the image according to any one of claims 1 to 4, wherein the plurality of objects are captured image in a state of being arranged randomly Processing equipment. Luminance image including a plurality of objects having the different posture, the image processing apparatus according to any one of claims 1 to 5, wherein the plurality of objects are captured image in a state of being piled . Based on the correspondence acquired by the third acquisition means, further comprising an estimation means for estimating a luminance distribution on the surface of the object;
The generation unit, based on the model information of the object and luminance distribution estimated by the estimating means, the image processing according to any one of claims 1 to 6, characterized in that to generate the learning image apparatus. The image processing apparatus according to claim 7 , wherein the generation unit determines a luminance value according to information related to a direction of a surface of the object from a variance of the luminance distribution. The image processing apparatus according to claim 7 , wherein the estimation unit estimates the luminance distribution based on the correspondence acquired by the third acquisition unit and a predetermined luminance distribution model. The estimation means includes a means for acquiring a plurality of luminance distribution models, an assigning means for assigning the correspondence acquired by the third acquisition means to the plurality of luminance distribution models, and the plurality of luminance distribution models as the luminance distribution. The image processing apparatus according to claim 9 , further comprising an update unit configured to update based on a correspondence relationship assigned to the model. The assigning means is the brightness distribution in which a brightness value obtained by inputting information relating to the orientation of the surface of the object indicated by the correspondence acquired by the third acquisition means is closest to the brightness value obtained by the correspondence The image processing apparatus according to claim 10 , wherein the correspondence relationship is assigned to a model. 11. The apparatus according to claim 10 , further comprising: a unit that causes the allocation unit and the update unit to repeatedly perform processing until the allocation destinations of the correspondence relationship acquired by the third acquisition unit before and after the update match. The image processing apparatus described. The generating means performs projective transformation corresponding to a plurality of postures of the object on the model information of the object, and in the orientation of the surface of the object indicated by the model information after the projective transformation, giving a brightness value corresponding to the orientation of the surface, the image processing apparatus according to any one of claims 1 to 12, characterized in that to generate the learning images for each of the posture. The model information is CAD data, the learning image is an image processing apparatus according to any one of claims 1 to 13, characterized in that a CG image. The image processing according to any one of claims 1 to 14 , wherein the information related to the orientation of the surface of the object acquired by the second acquisition means is information on a normal of the surface of the object. apparatus. The image processing apparatus according to any one of claims 1 to 15, further comprising a distance image generating means for generating a distance image from the model information of the said luminance image object. The image processing apparatus according to claim 16 , wherein the distance image represents a position of a surface of the object. Based on the generated training image by the generation unit, an image processing apparatus according to any one of claims 1 to 17, further comprising means for generating a classifier for recognizing said object . Means for acquiring an image including the object;
The image processing apparatus according to claim 18 , further comprising a recognition unit configured to recognize the object from an image including the object based on the classifier. Luminance image including a plurality of objects having the different posture, the image processing apparatus according to any one of claims 1 to 19, wherein the a recognized image taken under the same lighting conditions of the object. An image processing method for generating a learning image of an object used for creating a dictionary referred to in image recognition processing for detecting an object,
A first acquisition step of acquiring a luminance value of the surface of the object in a plurality of regions of the luminance image from a luminance image including the plurality of objects having different postures ;
A second acquisition step of acquiring information relating to the orientation of the surface of the object in the plurality of regions from which the luminance value is acquired in the first acquisition step;
Correspondence relationship between the luminance value of the surface of the object in the plurality of areas acquired in the first acquisition step and information related to the orientation of the surface of the object in the plurality of areas acquired in the second acquisition step A third acquisition step of acquiring
A correspondence relationship acquired by the third acquisition step, based on the model information of the object, a generation step of generating a training image of the object,
An image processing method comprising: Further comprising the step of inputting the luminance image;
In the first acquisition step, the luminance value of the surface of the object in each of the plurality of regions of the input luminance image is acquired;
The image processing method according to claim 21 , wherein in the second acquisition step, information relating to a direction of a surface of the object in each of the plurality of regions is acquired. The distance information with respect to the object is acquired in the second acquisition step, and information related to the orientation of the surface of the object in the plurality of regions is acquired based on the acquired distance information. Or the image processing method of 22. In the second acquisition step, distance information of each pixel of the luminance image is acquired, and information related to the orientation of the surface of the object in the plurality of regions is acquired based on the acquired distance information. The image processing method according to claim 23. An image processing system for generating a learning image used for generating a dictionary referred to in object recognition,
Photographing means for photographing a luminance image including a plurality of the objects having different postures;
First acquisition means for acquiring a luminance value of the surface of the object in a plurality of regions of the luminance image from a luminance image including the plurality of objects having different postures ;
Second acquisition means for acquiring information relating to the orientation of the surface of the object in the plurality of regions from which the luminance values are acquired by the first acquisition means;
Correspondence relationship between the brightness value of the surface of the object in the plurality of areas acquired by the first acquisition means and information related to the orientation of the surface of the object in the plurality of areas acquired by the second acquisition means A third acquisition means for acquiring
A correspondence relationship acquired by the third acquisition means, based on the model information of the object, and generating means for generating a training image of the object,
An image processing system comprising: By being executed by a computer device, a program to function as each unit of the image processing apparatus according to the computer device in any one of claims 1 to 20.

Images