KR101086274B1 - Apparatus and method for extracting depth information - Google Patents

Abstract

Disclosed are a depth information extraction apparatus and an extraction method for generating final depth information corrected from initial depth information by using initial depth information on a reference view image and previous frames of the reference view image. According to this technology, the image input unit for receiving and storing the reference viewpoint image and the target viewpoint image; A depth information input unit configured to receive and store initial depth information of the reference viewpoint image; An image dividing unit dividing the reference viewpoint image into a plurality of areas by using the color similarity of the reference viewpoint image and the initial depth information; And a first depth information generator that corresponds to each of the divided regions to correspond to the target viewpoint image, and generates final depth information of which the initial depth information is corrected using color similarity between corresponding regions. An information extraction apparatus is provided.

Depth, variation, stereo

Description

Depth information extraction device and extraction method {APPARATUS AND METHOD FOR EXTRACTING DEPTH INFORMATION}

The present invention relates to a depth information extraction apparatus and an extraction method for generating more improved depth information for a reference view image.

The present invention is derived from a study conducted as part of the IT source technology development project of the Ministry of Knowledge Economy. [Task management number: 2008-F-011-01, Task name: Next-generation DTV core technology development]

The method for obtaining accurate disparity / depth information about images obtained from a plurality of cameras having different viewpoints has been studied for a long time in the field of computer vision, and many studies have been made. In general, disparity / depth information about an image is extracted through stereo matching. Stereo matching refers to a process of finding a correspondence point between a reference view image and a target view image acquired from a plurality of cameras having different viewpoints.

The stereo matching will be described in more detail below. When one of the images obtained from two cameras is set as the reference viewpoint image, and the other image is set as the target viewpoint image to which the reference viewpoint image is compared, the photographing object is represented as pixels in the reference viewpoint image and the target viewpoint image. The pixels with respect to are present at different positions in the reference view image and the target view image due to differences in viewpoints. The position of the photographing object in the reference viewpoint image and the target viewpoint image is acquired through stereo matching, and the degree of difference in the position of the photographing object in the reference viewpoint image and the target viewpoint image is called disparity. Three or more cameras may be used to perform stereo matching, and the reference view image and the target view image may be changed according to a setting.

Depth represents the distance between the photographing object and the camera and is closely related to the variation. The farther away the object is from the camera, the smaller the variance value and the greater the depth value. The closer the object is to the camera, the larger the variance value and the smaller the depth value. As a result, it can be said that the depth and the variation are inversely related, and the depth information can be obtained from the variation information, or the variation information can be obtained from the depth information. Meanwhile, the disparity / depth information stored in the form of an image is called a disparity / depth map.

Depth information obtained through stereo matching has a problem that reliability of depth information is low due to the characteristics of an image, that is, a change in lighting conditions, a texture, and a presence of a shielding area. In order to solve this problem, a lot of researches have been conducted to divide an image into several segments having similar colors and perform stereo matching based on a region corresponding to a reference viewpoint image and a target viewpoint image. These techniques are carried out on the assumption that the depth changes in any area equally or continuously, where the depth value changes drastically, such as a textured area or a boundary point where discontinuities in the depth value appear, for example, the boundary point between the object and the background. In contrast, the error of depth information may increase.

 In addition, the conventional method of extracting the variation / depth information extracts the variation / depth information independently for each frame of the image, so that the result may be severely distorted even though the actual depth information between the previous frame and the current frame should be similar. have.

The present invention has been proposed to solve the above problems, and an object thereof is to improve an error of depth information due to discontinuity of a depth value in an area according to color similarity in a reference view image.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention, which are not mentioned above, can be understood by the following description, and more clearly by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

The present invention for achieving the above object is an image input unit for receiving and storing the reference viewpoint image and the target viewpoint image; A depth information input unit configured to receive and store initial depth information of the reference viewpoint image; An image dividing unit dividing the reference viewpoint image into a plurality of areas by using the color similarity of the reference viewpoint image and the initial depth information; And a first depth information generator that corresponds to each of the divided regions to correspond to the target viewpoint image, and generates final depth information of which the initial depth information is corrected using color similarity between corresponding regions. Provided is an information extraction apparatus.

In addition, the present invention for achieving the above object is a step of receiving and storing the reference viewpoint image and the target viewpoint image; Receiving and storing initial depth information of the reference view image; Dividing the reference view image into a plurality of areas by using color similarity of the reference view image and the initial depth information; And generating final depth information of which the initial depth information is corrected by using each of the divided regions to correspond to the target viewpoint image, and using color similarity between corresponding regions. to provide.

According to the present invention, the reference view image is divided into a plurality of regions using color similarity with respect to the reference view image and depth information of the reference view image that can be obtained from the depth camera. Therefore, the boundary point where the depth value is discontinuous in the region according to the color similarity can be subdivided, and the error of depth information due to the discontinuity of the depth value in the region can be improved.

First, the concept of the depth information extraction apparatus according to the present invention will be briefly described.

An apparatus for extracting depth information according to the present invention divides a reference view image into a plurality of regions and generates depth information of the reference view image for each divided region. At this time, the depth information extracting apparatus according to the present invention divides the reference view image into a plurality of regions using color similarity, but further divides the image using the initial depth information. Here, the initial depth information may be obtained from a depth camera or the like. Therefore, when the depth information extraction apparatus according to the present invention has a discontinuity of the depth value in the divided region according to the color similarity, for example, the depth information extracting apparatus may repartition the region according to the discontinuity of the depth value. As a result, according to the present invention, since the boundary point at which the discontinuity of the depth value in the region appears is subdivided by the initial depth information, the error of the depth information according to the discontinuity of the depth value of the divided region may be improved according to the color similarity.

The depth information extracting apparatus according to the present invention generates final depth information of the current frame by using depth information of a previous frame of the reference view image when generating final depth information of the reference view image. Here, the final depth information is depth information of which the initial depth information is corrected and is finally generated by the depth information extraction apparatus according to the present invention. Therefore, according to the present invention, when the previous frame and the current frame are almost the same as the background area, continuity of depth information of the previous frame and the current frame can be maintained. Here, the continuity of the depth information means that the depth information of the previous and current frames that are substantially the same maintains the same.

DETAILED DESCRIPTION Hereinafter, the most preferred embodiments of the present invention will be described with reference to the accompanying drawings so that those skilled in the art may easily implement the technical idea of the present invention.

1 is a view showing a depth information extraction apparatus according to an embodiment of the present invention.

As shown in FIG. 1, the apparatus for extracting depth information according to the present invention includes an image input unit 101, a depth information input unit 103, an image divider 105, and a first depth information generator 107.

The image input unit 101 receives and stores a reference view image and a target view image. The reference view image and the target view image may be obtained from a camera and input, and the image input unit 101 may store the frames for each frame in synchronization with the input reference view image and the target view image. According to the present invention, the image input unit 101 may receive a plurality of target viewpoint images, and the image input unit 101 may store the frames for each frame in synchronization with the reference viewpoint image and the plurality of target viewpoint images.

The depth information input unit 103 receives and stores initial depth information about the reference view image. As described above, the initial depth information may be acquired and input in the form of an image from the depth camera. The initial depth information is acquired at the same time as the reference viewpoint image, and the depth information input unit 103 preferably stores the frames for each frame in synchronization with the reference viewpoint image and the initial depth information. Here, when the initial depth information is acquired at the same time as the reference view image, for example, a depth value for the first frame of the reference view image is included in the first frame of the initial depth information image.

The image divider 105 divides the reference view image into a plurality of regions by using color similarity and initial depth information of the reference view image. The image splitter 105 may split the reference view image using an average shift algorithm and a split-and-merge algorithm.

In more detail, the image divider 105 first divides the reference view image into a plurality of regions according to the color similarity. The image splitter 105 may repartition the segmented region using initial depth information when there is a boundary point where the depth value is discontinuous in the segmented region according to the color similarity. In this case, the image splitter 105 may subdivide the divided region according to the average value of the initial depth information of the divided region, and the difference between the initial depth information and the average value of the initial depth information of the pixels of the divided region may be preset. If it is larger than the threshold value TH1, the divided region may be subdivided.

The first depth information generator 107 associates each of the plurality of regions divided by the image divider 105 with the target viewpoint image. The first depth information generator 107 generates final depth information with initial depth information corrected using color similarity between corresponding areas.

Initial depth information provided by the depth camera is limited in resolution. That is, it is difficult for a depth camera to provide high resolution depth information and to provide approximate depth information for a reference view image. The depth information extracting apparatus according to the present invention finally generates final depth information in which the initial depth information is corrected. Since the first depth information generator 107 generates final depth information having higher reliability than the initial depth information through stereo matching, the first depth information generator 107 may correct the initial depth information to generate final depth information on the reference view image. The first depth information generator 107 will be described later in more detail with reference to FIG. 2.

In summary, the depth information extracting apparatus according to the present invention divides the reference view image by using color similarity and initial depth information of the reference view image. Therefore, when there is a boundary point where the depth value is discontinuous in the divided region according to the color similarity, the divided region according to the color similarity may be subdivided by the initial depth information, and the depth information according to the discontinuity of the depth value existing in the region. The error of can be improved.

Meanwhile, as described above, the depth information generated by the depth information extracting apparatus according to the present invention may be converted into variation information according to the relationship with the variation information.

The apparatus for extracting depth information according to the present invention may further include a second depth information generator 109, which will be described in detail with reference to FIG. 2.

FIG. 2 is a diagram illustrating in detail the first depth information generator 107 shown in FIG. 1.

As shown in FIG. 2, the first depth information generation unit 107 according to the present invention includes a reference depth information selection unit 201, a similarity determination unit 203, and a stereo matching unit 205.

The reference depth information selecting unit 201 selects the depth value of the region of the previous frame of the reference viewpoint image as reference depth information according to the degree of movement between the previous frame and the current frame of the reference viewpoint image. Here, the degree of movement between the previous frame and the current frame means a position difference between an object or a background scene existing in each of the previous and current frames.

At this time, the reference depth information selector 201 may determine the degree of motion for each corresponding area of the previous frame and the current frame of the reference view image. As an embodiment, the degree of motion may be determined according to Equation 1 below. You can judge.

On the other hand, the reference depth information is a depth value as a reference in determining the color similarity for each corresponding area of the previous frame and the current frame in the similarity determination unit 203 to be described later. Although described below in detail, the similarity determining unit 203 determines the color similarity between a predetermined region of the virtual viewpoint image and the target viewpoint image within a range added or subtracted by a predetermined value from the reference depth information.

1) The pixel ratio of the area S of the previous frame to the area S ′ of the current frame> a preset threshold value TH2 (eg, 85%).

2) Color difference between the area S of the previous frame and the area S ′ of the current frame < the preset threshold TH3

3) Difference in Initial Depth Information (Value) between Pixels in the Area S of the Previous Frame and the Area S ′ of the Current Frame <Preset Threshold TH4

In Equation 1, the area S ′ of the current frame is an area corresponding to the area S of the previous frame. That is, the reference depth information selector 201 may determine the degree of motion by comparing the pixel ratio, color difference, and depth value difference between the previous frame and the current frame with a preset threshold. When the condition of Equation 1 is satisfied, the reference depth information selection unit 201 is because the region S 'of the current frame and the region S of the previous frame have a high probability of being a static background, for example. May select initial depth information of the previous frame as reference depth information. In addition, the reference depth information selector 201 may select a value obtained by averaging initial depth information of the previous frame and initial depth information of the current frame as reference depth information, or select final depth information of the previous frame as reference depth information. At this time, the final depth information for the previous frame may be provided from the stereo matching unit 205 to be described later.

The reference depth information selecting unit 201 may determine the degree of movement in consideration of all three conditions of Equation 1, or may determine the degree of movement in consideration of only a part thereof.

On the other hand, as a result of determining the movement degree, if the condition of [Equation 1] is not satisfied or the initial depth information for the previous frame does not exist, the reference depth information selecting unit 201 uses the reference depth information for the initial depth information for the current frame. To select.

As described above, the similarity determination unit 203 determines the color similarity between the reference viewpoint image and the target viewpoint image for each region within a range added or subtracted by a predetermined value in the reference depth information. The color similarity may be determined by a similar function such as a sum of squared distance (SSD) or a normalized cross correlation (NCC). The range (search_depth (x, y)) added or subtracted by a predetermined value in the reference depth information may be expressed by Equation 2 below.

Base_depth (x, y)-error_rate * Base_depth (x, y) <search_depth (x, y) <Base_depth (x, y) + error_rate * Base_depth (x, y)

In Equation 2, Base_depth (x, y) represents reference depth information selected by the reference depth information selecting unit 201, and error_rate is a value smaller than 1 set in consideration of an error of the reference depth information.

Hereinafter, the similarity determination unit 203 will be described in detail the process of determining the color similarity using the reference depth information.

For example, when the camera providing the base view image and the target view image is parallel to the X axis and the variation information (value) between the base view image and the target view image converted from the reference depth information is '4', the similarity determination unit ( 203 determines color similarity between the reference view image and the target view image within a range of 4-0.5 * 4 <search_depth (x, y) <4 + 0.5 * 4. That is, when the coordinate value of the predetermined region of the reference viewpoint image is (3, 3), the similarity determination unit 203 may reference the reference viewpoint within the range of (1, 3) to (5, 3) coordinate values of the target viewpoint image. The image and color similarity may be compared and determined. Since the depth information and the variation information are closely related, the variation information is described in the above example. The reference viewpoint image and the target viewpoint image are images captured at different viewpoints. Therefore, the reference depth information obtained from the reference viewpoint image cannot be applied to the target viewpoint image as it is, and the similarity determination unit 203 determines the color similarity within a predetermined range.

On the other hand, when a pixel having an incorrect depth value is included in the region due to shielding, an error may exist in the color similarity determination result of the similarity determination unit 203. Accordingly, the similarity determination unit 203 may determine the color similarity by excluding the pixel having the largest deviation from the average of the difference of the color values of the pixels in each region in the color similarity determination process in order to reduce errors in the color similarity determination. have. At this time, the similarity determination unit 203 repeats until the standard deviation for the difference in the color value of the pixel is smaller than the preset threshold TH5, or there is no change in the standard deviation, or the maximum number of repetitions is reached. Color similarity can be determined. In addition, the similarity determination unit 203 may reduce the error of the color similarity determination through the planar approximation process and the cost function optimization.

The stereo matching unit 205 determines the color similarity by the similarity determination unit 203, and generates final depth information by stereo matching between the regions having the highest color similarity.

Through stereo matching, a corresponding region (pixel) of the reference viewpoint image and the target viewpoint image is extracted and final depth information is generated. At this time, the stereo matching unit 205 compares the color similarity obtained through the initial depth information of the current frame of the reference view image or the depth information obtained from the previous frame, and then performs the stereo matching between the regions corresponding to the region having the highest similarity. Final depth information of the view image may be generated.

In addition to the stereo matching method, the stereo matching unit 205 includes a silhouette-based method, a volume-based modeling method of voxel coloring, and a motion for calculating three-dimensional information about a multi-view object photographed by camera movement. Final depth information can be generated based on a shape shape tracking method. The final depth information may be converted into a point cloud or a three-dimensional model in the three-dimensional space.

In summary, the depth information extracting apparatus according to the present invention generates the depth information of the current frame based on the depth information of the previous frame. Therefore, according to the present invention, when the previous frame and the current frame are almost the same as the background area, the continuity of the depth information of the previous frame and the current frame can be maintained.

Meanwhile, the second depth information generator 109 illustrated in FIG. 1 projects the final depth information of the reference viewpoint image to the target viewpoint image by using information of a camera providing the reference viewpoint image and the target viewpoint image. Depth information about the viewpoint image is generated.

The apparatus for extracting depth information according to the present invention may further include a camera correction unit (not shown). The camera correction unit performs camera calibration using a reference viewpoint image and a target viewpoint image inputted to the image input unit 101. By calculating the matrix based on the camera information such as the focal length and the positional relationship between each viewpoint. The second depth information generator 109 may project the final depth information of the reference view image generated by the first depth information generator 107 to the target view image based on the base matrix.

As a result of the projection, when a hole is generated due to a difference in viewpoint between the reference viewpoint image and the target viewpoint image, the second depth information generator 109 interpolates the hole using the depth value of the neighboring pixel to depth information about the target viewpoint image. Can be generated.

Although the present invention has been described in terms of the apparatus, the operation of each component constituting the depth information extracting apparatus according to the present invention can be easily understood from the process point of view. Therefore, the operation of the components included in the depth information extraction apparatus according to the present invention can be understood as the steps included in each depth information extraction method according to the principles of the present invention. Hereinafter, a method of extracting depth information according to the present invention will be described with reference to FIGS. 3 and 4.

3 is a flowchart illustrating a method of extracting depth information according to an embodiment of the present invention.

As shown in Fig. 3, the depth information extraction method according to the present invention starts from step S301. In operation S301, the reference view image and the target view image are input and stored. The reference view image and the target view image may be obtained from a camera, input, and synchronized and stored.

In operation S303, initial depth information of a reference view image is input and stored. The initial depth information may be obtained from the depth camera and input, and may be obtained at the same time as the reference view image.

In operation S305, the color similarity and the initial depth information of the reference viewpoint image are used to divide the reference viewpoint image into a plurality of regions. In more detail, when there is a discontinuous boundary point of the depth value in the divided region according to the color similarity in step S305, it may be re-divided by the initial depth information. In this case, when the difference between the average value of the initial depth information and the initial depth information of the pixels of the divided region is greater than the preset threshold TH1, the divided region may be re-divided.

Each of the plurality of divided regions in step S307 corresponds to the object viewpoint image, and final depth information is generated by correcting initial depth information by using color similarity between the corresponding regions. In step S307, a method such as stereo matching may be used to generate final depth information. Step S307 will be described later in more detail with reference to FIG.

Meanwhile, the method for extracting depth information according to the present invention may further include step S309. Step S309 will be described along with step S307 in FIG. 4.

FIG. 4 is a flowchart showing step S307 shown in FIG. 3 in more detail.

As shown in Fig. 4, step S307 starts from step S401 in more detail.

In operation S401, the depth value of the region of the previous frame of the reference viewpoint image is selected as reference depth information according to the degree of movement between the previous frame and the current frame of the reference viewpoint image. The degree of motion may be determined by comparing the pixel ratio, color difference, and depth information (value) difference between regions of the previous frame and the current frame according to Equation 1 above with a preset threshold. At this time, the initial depth information of the previous frame or the value obtained by averaging the initial depth information of the previous frame and the initial depth information of the current frame may be selected as the reference depth information.

In step S403, the color similarity between the reference viewpoint image and the target viewpoint image is determined for each region within a range added or subtracted by a predetermined value from the reference depth information. The range search_depth (x, y) subtracted by a predetermined value from the reference depth information may be expressed by Equation 2 above. Since the reference viewpoint image and the target viewpoint image are images captured at different viewpoints, reference depth information obtained from the reference viewpoint image cannot be directly applied to the target viewpoint image, and color similarity is determined within a predetermined range in step S403.

At this time, in step S403, in order to reduce the error of the color similarity determination by the shielding area or the like, the pixel having the largest deviation from the average with respect to the difference in the color values of the pixels in each area is excluded from the color similarity determination process and the color similarity is determined. Can be judged. In step S403, the color similarity may be determined until the standard deviation of the difference in color values of the pixels becomes smaller than the preset threshold value TH5, or the standard deviation is unchanged or the maximum number of repetitions is reached.

In step S405, as a result of the color similarity determination in step S403, final depth information is generated by stereo matching between areas having the highest color similarity. Through stereo matching, a corresponding region (pixel) of the reference viewpoint image and the target viewpoint image is extracted and final depth information is generated.

 Meanwhile, in step S309 illustrated in FIG. 3, information of a camera providing a reference viewpoint image and a target viewpoint image is used to project final depth information of the reference viewpoint image to the target viewpoint image, and to the target viewpoint image. Depth information is generated.

In step S309, the reference matrix image and the target viewpoint image input in step S301 are used to calculate a matrix based on camera information such as a focal length and a mutual positional relationship for each viewpoint through camera calibration. In operation S309, final depth information on the reference view image generated in operation S307 may be projected onto the target view image based on the base matrix.

As a result of the projection, when holes are generated due to a difference in viewpoints between the reference viewpoint image and the target viewpoint image, final depth information on the target viewpoint image may be generated by interpolating the holes using depth values of surrounding pixels in step S309. Depth information about the reference view image and the target view image may be recorded as a digital image, and the digital image may be converted into a point cloud or a 3D model in a 3D space.

On the other hand, the depth information extraction method according to the present invention as described above can be created by a computer program. And the code and code segments constituting the program can be easily inferred by a computer programmer in the art. In addition, the written program is stored in a computer-readable recording medium (information storage medium), and read and executed by a computer to implement the method of the present invention. The recording medium includes all types of computer-readable recording media (CD, DVD), as well as intangible media such as carrier waves.

In addition, although the present invention has been described by way of limited embodiments and drawings, the present invention is not limited thereto and is equivalent to the technical spirit and claims of the present invention by those skilled in the art to which the present invention pertains. Of course, various modifications and variations are possible within the scope.

1 is a diagram showing an apparatus for extracting depth information according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating the first depth information generator 107 shown in FIG. 1 in more detail.

3 is a flowchart illustrating a method of extracting depth information according to an embodiment of the present invention;

4 is a flowchart showing in more detail the step S307 shown in FIG.

Claims (10)

An image input unit which receives and stores a reference view image and a target view image; A depth information input unit configured to receive and store initial depth information of the reference viewpoint image; The reference viewpoint image is divided into a plurality of segments by using the color similarity of the reference viewpoint image and the initial depth information, and optionally, the divided regions are divided into the initial depth information and the initial depth information. An image divider for re-segmenting according to a difference in average values; And A first depth information generation unit that corresponds to each of the plurality of regions to the target viewpoint image, and generates first final depth information in which the initial depth information is corrected through stereo matching between regions having high color similarity between corresponding regions; Depth information extraction device comprising a. delete The method of claim 1, The first depth information generator A reference depth information selector which selects a depth value of the previous frame area as reference depth information according to a degree of movement between a previous frame and a current frame of the reference view image; A similarity determination unit that determines a color similarity between the reference viewpoint image and the target viewpoint image for each region within a range added or subtracted by a preset value from the reference depth information; And As a result of the color similarity determination, the stereo matching unit generates the first final depth information by stereo matching between areas having the highest color similarity. Depth information extraction device comprising a. The method of claim 3, wherein The reference depth information selection unit Determining the degree of movement by comparing the pixel ratio, color difference, and depth value difference for each region of the previous frame and the current frame of the reference viewpoint image with a preset threshold value Depth information extraction device. The method of claim 3, wherein Generating depth information on the target viewpoint image by projecting the first final depth information on the reference viewpoint image to the target viewpoint image using information of the reference viewpoint image and a camera providing the target viewpoint image 2nd depth information generator Depth information extraction device further comprising. Receiving and storing a reference view image and a target view image; Receiving and storing initial depth information of the reference view image; The reference viewpoint image is divided into a plurality of regions by using the color similarity of the reference viewpoint image and the initial depth information, and the divided regions are divided according to a difference between the average value of the initial depth information and the initial depth information. Subdividing; And Generating each of the plurality of regions corresponding to the target viewpoint image, and generating final depth information of which the initial depth information is corrected through stereo matching between regions having high color similarity between corresponding regions; Depth information extraction method comprising a. delete The method of claim 6, Generating the final depth information Selecting a depth value of an area of the previous frame as reference depth information according to a movement degree between a previous frame and a current frame of the reference view image; Determining a color similarity between the reference viewpoint image and the target viewpoint image for each region within a range subtracted by a predetermined value from the reference depth information; And Generating final final information by stereo matching between areas having the highest color similarity as a result of the color similarity determination; Depth information extraction method comprising a. The method of claim 8, The step of selecting the reference depth information Determining the degree of movement by comparing the pixel ratio, color difference, and depth value difference for each region of the previous frame and the current frame of the reference viewpoint image with a preset threshold value Depth information extraction method. The method of claim 8, Generating depth information on the target viewpoint image by projecting final depth information on the reference viewpoint image to the target viewpoint image using information of the reference viewpoint image and a camera providing the target viewpoint image. Depth information extraction method further comprising.

Images