JP2002117403A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor capable of efficiently and accurately performing processing for extracting an image of a specific subject from among photographed images. SOLUTION: In an object extraction processing using distance information and luminance information obtained from images including the background and the like, a scope of extraction is first and automatically limited based on distance information, a boundary part between an extracted object and the other region is set, and the set scope corresponds to a luminance image. Threshold value comparison processing is done in the extraction scope set based on profile tracking based on luminance in a boundary region set in the luminance image or the distance information of the luminance image. Since the profile tracking processing and the threshold value comparison processing are done in a region set in the luminance image only, an object region can be efficiently and accurately extracted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing method for measuring the shape of a subject, and more particularly, to an image processing apparatus for imaging a desired object together with a background, and more particularly, to extract a desired object area from a background area. The present invention relates to an image processing apparatus capable of performing the above. The present invention relates to an image processing device and an image processing method.

[0002]

2. Description of the Related Art A person is photographed with a background as a background, or only a person is taken from an image of a building or the like.
Alternatively, a so-called image segmentation method of cutting out an image of only a building is used in various fields.

As a conventional technique for extracting only an object region from an image including an object and a background, for example, the following method is known. As a first method, there is a method of manually performing a clipping process of a specific target object using an auxiliary tool. Specifically, a method is used in which a desired object is imaged together with a background using a mouse. In this method, a background portion in an image is selected, and the background portion is erased to obtain an object region.

As a second method, there is a method using a chroma key technique. Specifically, a desired object is imaged together with the background, and the image is obtained by dividing the area using the color of the background as a key, and the object area is divided. It is a method to extract.

A third method uses a simple difference. Specifically, an image obtained by imaging a desired object together with a background and a background image obtained by imaging only the background are used. And a difference process is performed to obtain a difference, and a region where the absolute value of the difference is equal to or larger than a threshold value is extracted as an object region.

As a fourth method, there is a method using distance image information obtained by a depth distance image obtaining apparatus. Specifically, an image (luminance image) obtained by imaging the desired object together with the background and a distance image having depth distance information corresponding to each pixel of the image are acquired, and when the depth distance information is equal to or less than the threshold value. This is a method of extracting a region in a luminance image corresponding to a certain region as an object region.

[0007] However, the first method described above has a problem that a great deal of labor is required since the work is performed manually. The second and third methods have a problem that it is extremely difficult to determine an appropriate threshold for identifying an area in the entire image. This is particularly difficult when there is an area of a similar color to the background of the desired object or when the desired object has a texture (pattern).

According to the fourth method, there is a problem that a part of the background image is erroneously extracted because the depth difference is large near the boundary between the desired object and the background and accurate depth information cannot always be obtained. In addition, it is necessary to acquire depth distance information corresponding to each pixel for an image obtained by imaging a desired object together with the background. There is also the problem of such a situation. In addition, when performing the process of lowering the resolution of a captured image in accordance with the resolution of the distance measuring device, there is a problem that the resolution of the extracted image is also lowered.

As a conventional method for improving the above problem, there is a method disclosed in Japanese Patent Application Laid-Open No. 7-121710, for example. This device first specifies one pixel on the contour of the object in the image data from the outside, and further specifies from the outside the direction in which to search the contour of the object from the specified pixel.
A method of extracting the contour of an object by searching for and determining the outline of the object from a designated pixel in a designated direction is disclosed. Thus, the contour of the desired object can be extracted from the image obtained by imaging the desired object together with the background in accordance with the input.

[0010] However, the method described in Japanese Patent Application Laid-Open No. 7-121710 has a problem that it requires manual input by an operator and is troublesome. Further, when the contour has a complicated shape, or when there is a texture (pattern) in the object or the background, if the pixel is initially separated from an externally specified pixel, accurate object extraction may not be performed. In addition, when the search pixel moves away from the object contour, the operator must manually execute the correction process of returning the search pixel to the contour again and again, and the operation is complicated and requires skill.

[0011]

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems of the prior art, and is intended to be applied to a case where an object to be extracted has a portion similar to a background color or an object to be extracted. Even when there is a texture in the object or the background, the processing range is limited in advance based on the distance information and the extraction processing is executed by the luminance information. It is an object of the present invention to provide an image processing apparatus and an image processing method capable of accurately extracting an object region by automatic processing that does not require confirmation and correction.

[0012]

Means for Solving the Problems The present invention has been made to achieve the above object, and a first aspect of the present invention is a process for extracting a specific subject image from an image obtained by an imaging means. A luminance image obtained by capturing an image including a subject image to be extracted, and a distance image based on distance information corresponding to pixels included in the luminance image. Means, based on the distance information, the subject image to be extracted,
An extraction area extracting means for determining a boundary area with another area and setting an extraction area in the luminance image, and determining an area of a subject image to be extracted in the extraction area set by the extraction area extracting means. And a region extracting means for executing processing.

Further, in one embodiment of the image processing apparatus of the present invention, the area extracting means includes: a subject image to be extracted obtained by the extracted area extracting means;
In a boundary area with another area, the image processing apparatus further includes a contour extraction unit that performs a tracking process of a contour of the subject image to be extracted, and the contour extraction unit includes a contour extraction unit that calculates a pixel included in the boundary area in the luminance image. It is characterized in that it has a configuration for executing processing for obtaining a boundary line between the subject image to be extracted and another region based on a luminance value.

Further, in one embodiment of the image processing apparatus of the present invention, the area extracting means includes a memory storing background image information excluding a specific subject image in the image acquired by the imaging means or luminance information of the background image. And threshold processing means for comparing the luminance information of each pixel in the extraction area set by the extraction area extraction means with the background image information or the luminance information of the background image stored in the memory. The threshold value processing means includes a luminance value of a pixel included in the extraction area set by the extraction area extraction means and a luminance obtained from the background image information or the luminance information of the background image stored in the memory. It is characterized in that it has a configuration in which a difference from the value is compared with a predetermined threshold value and a process of extracting the subject image to be extracted is performed.

Further, in one embodiment of the image processing apparatus according to the present invention, the processing means generates a distance image indicating distance information for each grid-like area on the image as distance information, and the extraction area extracting means A subject image to be extracted based on the luminance information associated with each of the lattice regions,
It is characterized in that a boundary region with another region is obtained and an extraction region is set in the luminance image.

Further, in one embodiment of the image processing apparatus of the present invention, the extraction area extracting means sets an extraction area to a luminance image based on a difference in luminance value between a grid point and a neighboring grid point in the luminance image. The configuration is characterized by

Further, in one embodiment of the image processing apparatus according to the present invention, the processing means captures an image of a subject together with a light projection pattern captured by the first imaging means and the second imaging means from different directions. The distance information of the photographed image is obtained based on the obtained image, and the first image pickup means changes the photographing pattern in a region where the amount of change is equal to or more than a predetermined value.
A new code corresponding to a photographing pattern by the image pickup means is assigned, and distance information generation is performed by a re-encoding method for generating the first distance information from a photographing pattern by the second image pickup means based on the new code. It is characterized by being.

Further, a second aspect of the present invention is an image processing method for executing a process of extracting a specific subject image from an image acquired by an image capturing means, and captures an image including a subject image to be extracted. A luminance image obtained by
An image generation processing step of generating a distance image based on distance information corresponding to a pixel included in the luminance image, and a subject image to be extracted based on the distance information,
An extraction region extraction step of obtaining a boundary region with another region and setting an extraction region in the luminance image, and within the extraction region set in the extraction region extraction step,
An area extraction step of executing a process of determining an area of a subject image to be extracted.

Further, in one embodiment of the image processing method according to the present invention, the area extracting step includes: in a boundary area between the subject image to be extracted obtained in the extracting area extracting step and another area; A contour extraction step of performing a contour tracking process of the contour of the subject image to be extracted, wherein the contour extraction step includes extracting the contour of the subject based on a luminance value of a pixel included in the boundary area in the luminance image. A process for obtaining a boundary line between the subject image and another region.

Further, in one embodiment of the image processing method of the present invention, the area extracting step includes: luminance information of each pixel in the extraction area set in the extraction area extracting step; and background image information stored in a memory. Or a threshold value processing step of executing a comparison process with luminance information of a background image, wherein the threshold value processing step includes: And comparing a difference between the background image information and a luminance value obtained from luminance information of the background image with a predetermined threshold value, and extracting the subject image to be extracted. And

Further, in one embodiment of the image processing method according to the present invention, in the image generation processing step, a distance image indicating distance information for each grid-like area on the image is generated as distance information, and the extraction area extraction is performed. The step of obtaining a boundary region between the subject image to be extracted and another region based on the luminance information associated with each of the lattice regions, and setting an extraction region in the luminance image. And

Further, in one embodiment of the image processing method according to the present invention, the extraction region extracting step sets the extraction region to a luminance image based on a difference in luminance value between a lattice point and a neighboring grid point in the luminance image. It is characterized by doing.

Further, in one embodiment of the image processing method according to the present invention, the image generation processing step includes a light projection pattern photographed by the first imaging means and the second imaging means from different directions with respect to the subject. The distance information of the photographed image is obtained based on the photographed image, and a new code corresponding to the photographing pattern by the first imaging unit is changed for an area in which the amount of change of the photographing pattern by the first imaging unit is a predetermined value or more. Allocation, second based on the new code
Wherein the distance information is generated by a re-encoding method for generating the first distance information from a photographing pattern by the imaging means.

[0024]

According to the configuration of the present invention, the extraction range is automatically limited without the need to specify the range, so that the extraction processing of the specific target can be realized by a simple operation. By limiting the range in which the extraction process is performed, the load of the calculation process is reduced, and the object region can be accurately extracted. Further, even when the number of distance measurement points is small, the object region can be accurately extracted with high resolution from the luminance image.

[0025]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image processing apparatus and an image processing method according to the present invention will be described in detail with reference to the drawings. FIG.
1 shows an image processing apparatus according to a first embodiment of the present invention, in which a projector 11 for projecting a projection pattern coded with respect to intensity onto a subject 50 as an extraction target, and a half provided on an optical path of the projection pattern. A mirror 12, a first camera 13 provided coaxially with the optical axis of the projector 11 to capture an image reflected by the half mirror 12, and a second camera 13 to capture an object from a direction different from the optical axis direction of the projector. The image information captured by the camera 14 and the first camera 13 and the second camera 14 is input to the image processing unit 20.

An image processing unit 20 includes a processing unit 21 for acquiring a distance image and a luminance image based on image information captured by the first camera 13 and the second camera 14, and a distance image memory for storing the distance image. 22, a luminance image memory 24 for storing a luminance image, and a pixel correspondence memory 23 for storing a pixel position where each pixel of the distance image corresponds to a pixel of the luminance image.
Extraction area extraction unit 25 and extraction area extraction unit 2 for inputting a luminance image and a distance image to extract an area where extraction processing is to be performed.
And an area extraction unit 26 for extracting a desired area from the luminance image based on the extraction area input from Step 5. The extracted image area data is displayed on the display 29.

The processing section 21 inputs a photographing pattern obtained by photographing a slit pattern as a projection pattern projected from the projector 11 on the optical axis of the projector 11 with the first camera 13. This photographing pattern is compared with the projection pattern in the processing unit 21 and the intensity difference is compared to generate a distance image.

The comparison of the intensity difference is performed by, for example, decomposing the slit pattern and the photographing pattern into cells of several pixels, and comparing the average intensity of the cells. When the intensity difference is equal to or greater than the threshold value and has a new intensity, the re-encoding unit generates and assigns a new intensity code, thereby re-encoding the cell.
When the strength is not the new strength, the code of the corresponding strength is assigned, and the cell is recoded.
The recoding process will be described later in detail.

The region extracting unit 26 has a contour extracting unit 27 and a region determining unit 28. The contour extracting unit 27 traces a boundary line in the extracted region input from the extracted region extracting unit 25 to form a contour. The region extraction unit 28 extracts
Based on the contour data input from step 7, an enclosed area is extracted as an extraction area.

Here, a description will be given of an example in which a re-encoding method described below is applied as one method of acquiring a distance image. However, in the configuration of the present invention, another method of obtaining a distance image is applied. It is also possible. For example, a laser method that emits laser light or ultrasonic waves to measure the amount of light reflected from the object and the arrival time, and extracts depth information, or a special pattern light source such as slit light, is used to generate the geometric shape of the target surface pattern. The distance information of the subject is obtained by a method such as a pattern projection method that does not apply recoding, which estimates a target shape from image information such as geometric deformation, or a method in which contour lines are formed by moire fringes by optical processing to obtain three-dimensional information. The configuration to be obtained may be applied.

The recoding process executed in the processing unit 21 will be described. FIG. 2 is a block diagram showing a detailed configuration of the recoding processing unit. FIG. 3 shows the positional relationship between the light source and the image sensor.

In FIGS. 2 and 3, cameras 3103 are coaxial with the light projector for projecting the pattern light.
Cameras 101 from different directions on both sides
And the cameras 2 and 102 photograph the subject. As shown in FIG. 1, a single camera combination may be used. The processing of an image obtained from only the camera 3 and the camera 1 and the processing of an image captured by the camera 3 and the camera 2 are the same. In this example, a method of acquiring distance data of a subject by recoding processing in a configuration including three cameras 101 to 103 and a projector 104 will be described.

As shown in FIG. 3, the distances I1, I2, and I3 shown in the figure are made equal so that the distance relationship between the cameras is uniform. The cameras 3 and 103 and the projector 104 are arranged using a half mirror 105 as a beam splitter so that the optical axes coincide. Camera 1, 101, Camera 2,
102 is on both sides of the cameras 3 and 103 and the projector 104,
They are arranged so that their optical axes are different from them. The distance between the central optical axis and the optical axes on both sides is the base length L.

The light projector 104 includes a light source 106, a mask pattern 107, an intensity pattern 108, and a prism 10.
9 is provided. Here, as the light source 106, a light source in an invisible region using infrared light or ultraviolet light can be used. In this case, each camera is configured as shown in FIG. That is, the incoming light 310 is split in two directions by the prism 301, one of which is in the invisible region (infrared or ultraviolet).
An imaging device (for example, a CCD) through the transmission filter 302
Camera 303, and the other light passes through an invisible region (infrared and ultraviolet) cutoff filter 304 and enters an imaging device 305.

The light source 106 shown in FIG. 3 is not limited to a visible region or an invisible region, but may be a light source in a wavelength band capable of capturing an image. In this case, a progressive scan type CCD camera is used for the cameras 3 and 103, and the cameras 1 and 101 and the cameras 2 and 102 are
The configuration is not particular. However, considering the correspondence with the cameras 3 and 103, CCD cameras having the same configuration are desirable. The pattern is projected from the light source 106, and the three cameras 1 to 3 (101 to 103) shoot simultaneously. Then, each camera obtains the light that has passed through the filters 304 and 305 (see FIG. 4) by the imaging devices 303 and 305 to collectively acquire images.

The process of acquiring distance information by the recoding method will be described with reference to FIG. As shown in the figure, the cameras 1 and 101 store luminance information obtained by photographing in a luminance value memory 1.
21 and stores the photographing pattern in the pattern image memory 12.
Stored in 2. Similarly, the cameras 2 and 102 store the luminance information in the luminance value memory 123 and the photographing pattern in the pattern image memory 124. Cameras 3 and 103
The luminance information is stored in the luminance value memory 125, and the photographing pattern is stored in the pattern image memory 126. Floodlight 104
, Each slit is divided into cells on a square lattice and stored in the frame memory 127 in order to refer to a coded pattern created in advance.

Using the stored photographing pattern and luminance information, distance information is obtained as follows. The following operation is common to both the combination of the cameras 1 and 101 and the cameras 3 and 103 and the combination of the cameras 2 and 102 and the cameras 3 and 103. Therefore, here, the combination of the cameras 1 and 101 and the cameras 3 and 103 is taken as an example. explain.

In FIG. 2, an area dividing section 128 divides an area of a photographing pattern photographed by the cameras 3 and 103. Then, an area where the intensity difference between adjacent slit patterns is equal to or less than the threshold is extracted as an area 1 where light from the projector does not reach, and an area where the intensity difference between the slit patterns is equal to or more than the threshold is extracted as an area 2. I do.
The re-coding unit 129 re-codes the extracted area 2 using the photographing pattern stored in the pattern image memory 126 and the projection pattern stored in the frame memory 127.

FIG. 5 is a flowchart for performing recoding. First, each slit pattern is vertically divided for each slit width (step 1001) to generate a square cell. An average value of the intensity is obtained for each of the generated cells, and the average value is set as the intensity of each cell (step 100).
2). In order from the center of the image, the intensity between each corresponding cell of the projection pattern and the photographing pattern is compared, and the intensity between the cells is equal to or greater than the threshold because the pattern has changed due to factors such as the reflectance of the object and the distance to the object. It is determined whether they are different (step 1003). If they do not differ by more than the threshold value, the re-encoding ends for all the photographed cells (step 1007).

If the difference is not less than the threshold value, it is determined whether or not the cell has a new strength (step 1004). When the cell has a new strength, a new code is generated and assigned (step 1005). If the cell does not have a new strength, the cell is coded using a sequence of slit patterns that can be distinguished from other appearing parts (step 1006). This ends the recoding (step 1007).

FIG. 6 shows an example of coding of the slit pattern. FIG. 6A shows a projection pattern coded by the arrangement of the slits.
(Strong), 2 (medium), and 1 (weak) are assigned. In FIG. 6B, a code 0 is newly assigned because the intensity has changed in the third cell from the left and a new code has appeared. In the same figure (c), 3 from the left
Since an existing code appears in the second cell from the top, the code is recoded as a new code from the cell arrangement, such as [232] in the vertical arrangement and [131] in the horizontal arrangement. This recoding is equivalent to projecting a complicated pattern such as a two-dimensional pattern onto a portion where the shape of the object is rich in change, and projecting a simple pattern onto a portion where the shape changes little. This process is repeated, and re-encoding is performed by assigning a unique code to all cells.

Next, description will be made returning to FIG. Camera 1, 1
The code decoding unit 130 on the 01 side stores the pattern image memory 1
The projection pattern is extracted from 22 and divided into cells in the same manner as described above. Then, the code of each cell is detected using the code re-coded by the re-coding unit 129 first, and the slit angle θ from the light source is calculated based on the detected code. FIG. 7 is a diagram showing a method of calculating a distance in the spatial coding, wherein a slit angle θ of a cell to which each pixel belongs, an x coordinate on an image taken by the camera 1, a focal length F as a camera parameter, and a base length L Then, the distance Z is calculated by the following (Equation 1).

[0043]

## EQU1 ## Z = (F × L) / (x + F × tan θ) (Equation 1)

The calculation of the distance Z is similarly performed in the code decoding unit 131 of the camera 2 or 102. Further, for the above-mentioned region 1, the distance is calculated as follows. In the area 1, pattern detection based on the projected pattern cannot be performed, so the corresponding point search unit 132 uses the brightness value memories 121, 123,
The parallax is detected by using the luminance information read from 125, and the distance is calculated based on the parallax.

The above is a method of acquiring distance information using the re-coding method. The recoding method is one of the distance information obtaining methods applicable to the image processing apparatus and the image processing method of the present invention. However, as described above, the distance information can be obtained without being limited to the recoding method. For example, a laser method, a pattern projection method to which recoding is not applied, and a method in which contour lines are formed by moire fringes by optical processing to obtain distance information as three-dimensional information can be applied.

Hereinafter, in the image processing apparatus and the image processing method of the present invention, after obtaining a distance image as distance information of a subject and a luminance image as luminance information of the subject,
Details of the processing executed by the image processing unit 20 in FIG. 1 will be described with reference to a specific processing example shown in FIG.

The processing example shown in FIG. 8 is a processing example in the case where a luminance image and a distance image of a person who has a mobile phone as a subject and talks are acquired based on the pattern projection method. 2
One camera, that is, a first camera 13 that is provided coaxially with the optical axis of the light projector 11 in FIG. 1 and captures a reflection image reflected by the half mirror 12, and an object that is different from the optical axis direction of the light projector A second camera 14 for photographing is used.

The camera angle is adjusted so that the object (a person who has a mobile phone and talks) fits within the photographable range of the two cameras, and a projection pattern is projected from the projector and installed coaxially with the optical axis of the projector 11. The first camera 13 shoots an image. The second camera 14 photographs the caller from a direction different from the optical axis direction of the projector 11. The image signals of the first camera 13 and the second camera 14 are image-processed by the processing unit 21 to obtain a distance image and a luminance image. For the acquisition of the distance image, for example, the above-described recoding method is applied.

The procedure for measuring the distance by the pattern projection method will be described below. FIG. 8A shows a luminance image obtained by capturing an object. An image obtained by capturing an object and a background with the first camera 13 that captures an object from the same direction as the optical axis direction of the projector 11 is stored in the luminance image memory 22 as a luminance image. This luminance image is obtained by classifying the luminance of light entering the light receiving portion of the first camera 13 from the surface of the object for each pixel.

FIG. 8B shows a distance image obtained by photographing the object. The processing section 21 photographs the projection pattern projected on the background and the object by the second camera 14 in a direction different from the optical axis direction of the light projector 11. Then, the distance image is formed by decomposing the image into regions corresponding to the distance based on the change amount of the projection pattern based on the distance, and assigning a distance code to the pixels of each classified region. This distance image is two-dimensionally arranged with each pixel having a distance from the light receiving portion of the second camera 14 to each point of the object. In FIG. 8B, the target person is located at the forefront position in the paper surface direction, and the background area is located behind it. As a method of expressing these on the screen, for example, the brightness can be expressed by the magnitude of the luminance, the luminance in the area on the near side of the paper with a short distance is increased, and the luminance is reduced in the depth direction of the paper at a distance. May be expressed as

The number of distance values obtained depends on the fineness of the stripe to be projected, and one column of distance data can be obtained for each stripe. The number of stripes to be projected is smaller than the number of pixels of the camera. Further, as in the above-described equation (1), the x value on the image corresponds to the distance value Z to each measurement point. The coordinates on the luminance image correspond to each pixel in the distance image, and information is recorded in the pixel correspondence memory.

FIG. 8C is a diagram in which, based on the distance image stored in the distance image memory 22, a range where the difference between the values of adjacent points is larger than the threshold value is extracted, and the range between the points is extracted. The range is indicated by a mesh line 801. That is, an area where the difference in distance value between adjacent pixels is large is extracted, and this is identified as a boundary line between the subject and the background.
The threshold value is set in advance according to the distance difference between the measurement target and the surrounding background or object. When it is assumed that the distance difference between each of the photographed objects is small, such as when one object is extracted from an image in which a plurality of objects are photographed, the threshold value is set to be small, and FIG. In the case where it is assumed that the distance difference between the subject and the background is large, as in the case of extracting a subject (person) from the background, as in the example of, the threshold value can be increased.

FIG. 8D shows a region extraction process of a luminance image executed as a process of the extraction region extraction unit 25. The extraction range set based on the distance image (the range indicated by the shaded portion 801 in FIG. 8C) corresponds to the luminance image stored in the luminance image memory 24 based on the information in the pixel correspondence memory 23. Then, the range of the luminance image is extracted as the region extraction region. FIG. 8D shows an extraction result of the luminance image.
The distance measurement points are indicated by grid points in the luminance image, and the extraction range is indicated by a mesh line 802. The extraction region extraction unit 25 extracts a range of the luminance image as a region extraction region based on a difference in luminance value between a lattice point in the luminance image and an adjacent lattice point.

FIG. 8E is a diagram showing the result of the processing executed in the contour extracting section 27. The shaded area 803 identified as the boundary between the subject and the background in the luminance image of FIG. The contour line 804 extracted by tracking the boundary line portion based on the luminance value is shown, and the contour line 804 is indicated by a thick line. In the configuration of the present invention, since the outline tracking processing range is limited to the hatched area 803 identified as the boundary between the subject and the background in the luminance image, the tracking point does not largely deviate from the extraction target, and
Since the tracking range is narrow, efficient tracking is possible.
The contour tracking is performed based on the luminance value in the shaded area 803 identified as the boundary line between the subject and the background in the luminance image.

FIG. 8F is an output area image acquired by the area determining section 28. A region separated by the contour line 804 obtained as a result of the contour tracking process executed by the contour extraction unit 27 is defined as an extraction region. Contour 804
Are displayed on the display 29 as an output area (an area corresponding to a person as a subject indicated by a shaded line 805).

According to the image processing apparatus of the first embodiment, even when the number of distance measurement points is small, it is possible to accurately and accurately extract a specific subject area to be extracted from a luminance image. In addition, since the extraction process, that is, the boundary portion between the extraction target and the other portion is selected in advance and the contour tracking is performed within the range, the processing range is limited, the load of the calculation process is reduced, and the efficiency is reduced. The object region can be accurately extracted.

FIG. 9 shows an image processing apparatus according to a second embodiment of the present invention, in which an area extracting section 26 includes a threshold processing section 30, a threshold information memory 31, and a background luminance information memory 3.
2 is different from the image processing apparatus shown in FIG. 1, and other configurations and functions are the same.

The threshold value processing section 30 includes the extraction area extraction section 2
5, the difference between the luminance of each pixel and the luminance stored in the background luminance information memory 32 exceeds the threshold based on the threshold 31 in the threshold information memory. The pixels are set as the pixels of the extraction region, and a set of the pixels is set as the extraction region. The background luminance information memory 32 stores, for example, a sample point luminance value of the background image or an average luminance value of the entire background image.

The area determination section 28 combines the area input from the threshold value processing section 30 and the area inside the background extraction area as an extraction area and displays it on the display 29.

According to the image processing apparatus of the second embodiment, object extraction is performed as a process of extracting only pixels different from background luminance by comparing with background luminance information. The tracking process becomes unnecessary, and the object region can be extracted by a light calculation process. By limiting the region to be compared, the object is less affected by the texture of the object, and the object can be accurately extracted. That is, the extraction range is automatically limited based on the distance information, the extraction range is associated with the luminance image, and threshold processing is performed within the limited range as a comparison process with the luminance value of the background image. Is performed, the load of the calculation processing is reduced, and the object region can be accurately extracted.

FIG. 10 shows an image processing apparatus according to a third embodiment of the present invention, in which an area extracting unit 26 includes a threshold processing unit 30, a threshold information memory 31, and a background image memory 33.
1 is different from the image processing apparatus shown in FIG. 1, and other configurations and functions are the same.

The threshold value processing section 30 includes the extraction area extraction section 2
5, the difference between each pixel and the luminance at the same pixel position stored in the background image memory 33 is determined based on the threshold value in the threshold value information memory 31 in the extraction area. Pixels exceeding the value are defined as pixels in the extraction area, and a set of the pixels is defined as the extraction area. The background image memory 33 stores, for example, an image of only the background captured by the first camera 13, and a luminance value is obtained based on the image.

The area determination section 28 combines the area input from the threshold value processing section 27 and the area inside the background extraction area as an extraction area and displays it on the display 29.

According to the image processing apparatus of the third embodiment, the object is extracted as a process of extracting only pixels different from the background luminance by comparing with the luminance information of the background image. Such an outline tracking process becomes unnecessary, and an object region can be extracted by a light calculation process. By limiting the region to be compared, the influence of the texture of the object is reduced, and the object can be accurately extracted.

As described above, in each of the first to third embodiments, the image processing apparatus of the present invention inputs distance information and luminance information obtained from an image including an extraction target and other backgrounds, and The extraction range is automatically limited based on the distance information acquired from the distance image, the boundary between the extracted object and another region is set, and the set range is associated with the luminance image. Further, within the boundary area set in the luminance image, extraction of a specific object is performed by contour tracking based on luminance or threshold processing based on a difference from the luminance value of the background image. Since the contour tracking processing or threshold processing performed as the object extraction processing is performed only within the limited area set in the luminance image, the load of the calculation processing is reduced, and the object processing is efficiently and accurately performed. Regions can be extracted.

The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

[0067]

As described above, according to the image processing apparatus and the image processing method of the present invention, in the subject extraction (segmentation) process using the distance information and the luminance information obtained from the subject image including the background and the like, first, A configuration that automatically limits the extraction range based on the distance information, sets a boundary portion between the extracted object and other regions to an extremely narrow region, and executes a contour tracking process based on a luminance image within the range. Therefore, the load of the calculation processing is reduced, and the object region can be accurately extracted. In addition, since the contour tracking is performed using the luminance image, the object region can be extracted based on the high-resolution image of the luminance image, so that accurate region extraction can be performed. Is not so required, and an inexpensive distance measuring device can be used.

Further, according to the image processing apparatus and the image processing method of the present invention, in the subject extraction processing using the distance information and the luminance information obtained from the subject image including the background and the like,
First, the extraction range is automatically limited based on the distance information,
The extraction range is associated with the luminance image, and within the limited range, the threshold processing is performed as the comparison processing with the luminance value of the background image, so that the calculation processing load is reduced, The object region can be accurately extracted.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram showing an image processing apparatus according to a first embodiment of the present invention;

FIG. 2 is a block diagram illustrating an example of a configuration of acquiring distance information by recoding processing applicable to the image processing apparatus according to the present invention.

FIG. 3 is a block diagram illustrating a configuration example of a camera in the image processing apparatus according to the present invention.

FIG. 4 is a diagram illustrating an imaging configuration applicable to the image processing apparatus of the present invention.

FIG. 5 is a diagram showing a processing flow of a recoding process applicable in the image processing apparatus of the present invention.

FIG. 6 is a diagram showing an example of coding of a projection pattern in a re-coding process applicable to the image processing apparatus of the present invention.

FIG. 7 is a diagram illustrating a distance calculation method using a space coding method applicable to the image processing apparatus of the present invention.

FIG. 8 is a diagram illustrating a processing example in the image processing apparatus of the present invention.

FIG. 9 is an explanatory diagram showing an image processing apparatus according to a second embodiment of the present invention.

FIG. 10 is an explanatory diagram showing an image processing apparatus according to a third embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 11 Projector 12 Half mirror 13 1st camera 14 2nd camera 20 Image processing part 21 Processing part 22 Distance image memory 23 Pixel correspondence memory 24 Brightness image memory 25 Extraction area extraction part 26 Area extraction part 27 Outline extraction part 28 Area determination Unit 29 display 30 threshold processing unit 31 threshold information memory 32 background luminance information memory 33 background image information memory 101 camera 1 102 camera 2 103 camera 3 104 floodlight 105 half mirror 106 light source 107 mask pattern 108 intensity pattern 109 prism 121 , 123, 125 Luminance value memory 122, 124, 126 Pattern image memory 127 Frame memory 128 Area division unit 129 Recoding unit 130, 131 Code decoding unit 133 Distance information integration unit 134 3D memory Li 301 prism 302 pass filter 303, 305 imaging device

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G01B 11/24 FF Term (Reference) 2F065 AA12 AA56 BB05 CC16 FF07 GG01 HH07 JJ03 JJ05 JJ26 LL00 LL21 LL46 LL46 QQ06 QQ13 QQ24 QQ36 QQ42 5B057 BA02 CA01 CA08 CA13 CA16 CB01 CB08 CB12 CB16 CE09 DA08 DB03 DB06 DB09 DC17 DC22 DC30 5C023 AA16 BA02 DA04 DA08 5L096 AA09 CA05 CA17 FA06 FA14 FA19

Claims (12)

[Claims] An image processing apparatus for executing a process of extracting a specific subject image from an image obtained by an imaging unit, comprising: a brightness image obtained by capturing an image including a subject image to be extracted; Processing means for generating a distance image based on distance information corresponding to the pixels included in the luminance image, based on the distance information, the subject image to be extracted, and a boundary area between the other areas, An extraction region extraction unit that sets an extraction region in a luminance image, and within the extraction region set by the extraction region extraction unit,
An image processing apparatus comprising: an area extracting unit configured to execute a process of determining an area of a subject image to be extracted. 2. A tracing process of a contour of the subject image to be extracted in a boundary area between the subject image to be extracted obtained by the extracted region extracting means and another region. The contour extraction unit performs a boundary extraction between the subject image to be extracted based on the luminance values of the pixels included in the boundary region in the luminance image and another region. The image processing apparatus according to claim 1, wherein the image processing apparatus has a configuration for performing a process of obtaining 3. A memory storing background image information or background image luminance information excluding a specific subject image in an image obtained by an imaging unit, wherein the extraction unit sets an extraction set by the extraction region extraction unit. Threshold value processing means for performing a comparison process between the luminance information of each pixel in the region and the background image information stored in the memory or the luminance information of the background image; The difference between the luminance value of the pixel included in the extraction area set by the extraction area extraction means and the luminance value obtained from the background image information stored in the memory or the luminance information of the background image is determined in advance by a predetermined threshold value. 2. The apparatus according to claim 1, further comprising: executing a process of comparing and extracting the subject image to be extracted.
An image processing apparatus according to claim 1. 4. The processing means generates a distance image indicating distance information for each grid-like area on the image as distance information, and the extraction area extracting means generates a brightness associated with each grid-like area. 2. The image processing apparatus according to claim 1, wherein a boundary area between the subject image to be extracted and another area is obtained based on information, and an extraction area is set in the luminance image. . 5. The apparatus according to claim 4, wherein said extraction area extracting means sets an extraction area in the luminance image based on a difference in luminance value between a grid point and an adjacent lattice point in the luminance image. An image processing apparatus according to claim 1. 6. The processing means according to claim 1, wherein distance information of the photographed image is obtained based on an image photographed together with a light projection pattern photographed by said first imaging means and said second imaging means from different directions with respect to a subject. A new code corresponding to the pattern obtained by the first imaging means is assigned to a region where the amount of change of the imaging pattern by the first imaging means is equal to or greater than a predetermined value, and the second imaging means is assigned based on the new code. From the shooting pattern, the first
The image processing apparatus according to claim 1, wherein distance information is generated by a re-encoding method for generating distance information. 7. An image processing method for executing a process of extracting a specific subject image from an image obtained by an imaging unit, comprising: a luminance image obtained by capturing an image including a subject image to be extracted; An image generation processing step of generating a distance image based on distance information corresponding to a pixel included in the luminance image; and obtaining a boundary area between the subject image to be extracted and another area based on the distance information. An extraction area extraction step of setting an extraction area in the luminance image; and an extraction step of executing a process of determining an area of a subject image to be extracted in the extraction area set in the extraction area extraction step. An image processing method comprising: 8. The area extracting step includes: tracking a contour of the subject image to be extracted in a boundary area between the subject image to be extracted obtained in the extracting area extracting step and another area. A contour extraction step of executing a boundary line between the subject image to be extracted and another region based on a luminance value of a pixel included in the boundary region in the luminance image. The image processing method according to claim 7, further comprising: 9. The area extracting step executes a process of comparing luminance information of each pixel in an extraction area set in the extraction area extracting step with background image information or luminance information of a background image stored in a memory. A threshold value processing step, wherein the threshold value processing step includes a luminance value of a pixel included in the extraction region set in the extraction region extraction step and a background image information or a luminance of the background image stored in the memory. 8. The image processing method according to claim 7, wherein a difference from a luminance value obtained from the information is compared with a predetermined threshold value, and a process of extracting the subject image to be extracted is executed. 10. The image generation processing step generates a distance image indicating distance information for each lattice area on the image as distance information, and the extraction area extraction step associates each of the lattice areas with each other. 8. The image processing method according to claim 7, wherein a boundary area between the subject image to be extracted and another area is obtained based on the obtained luminance information, and an extraction area is set in the luminance image. 11. The method according to claim 10, wherein said extracting area extracting step sets an extracting area to a luminance image based on a difference in luminance value between a grid point and a neighboring grid point in the luminance image.
The image processing method according to 1. 12. The image generation processing step includes the steps of: a distance of a photographed image based on an image photographed together with a light projection pattern photographed by a first image pickup means and a second image pickup means from different directions with respect to a subject; Information is obtained, and a new code corresponding to the shooting pattern by the first imaging means is assigned to an area where the amount of change of the shooting pattern by the first imaging means is equal to or greater than a predetermined value, and the second imaging is performed based on the new code. From the photographing pattern by means
8. The image processing method according to claim 7, wherein distance information is generated by a re-coding method for generating distance information.