JP6148154B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing program for estimating the depth of a scene from multi-viewpoint images and depth information.

  In the field of video media, in recent years, the flow of high reality and personalization has been rapidly progressing. 3D video that enables a highly realistic video experience by viewing the video three-dimensionally has been widely used in movies, televisions, game machines, and the like. In addition, one of the factors that accelerate the trend toward higher realization is the increase in screen size, such as 4K video with 4 times the resolution of full HD and Super Hi-Vision video that is currently being developed. In addition, the spread of video-on-demand services via a network is remarkable, and it can be seen that personalization of video for viewing a favorite content on a favorite device by a user such as a television or a smartphone is progressing.

  In such a background, one of the new video media that further accelerates the realization and personalization is a free viewpoint video that allows the user to enjoy the video while moving the viewpoint that the user wants to see. A free viewpoint video is a video that is synthesized from an arbitrary viewpoint, which is the viewpoint that the user wants to see, by estimating the depth of the scene using videos taken from multiple viewpoints. In many methods, the depth of a scene is obtained by image processing such as stereo matching using parallax of a multi-viewpoint video, or three-dimensional shape measurement by a laser range sensor.

  However, it is difficult to obtain a depth image without a defect. For example, a depth image estimated by stereo matching has a high resolution, but it is difficult to completely prevent an estimation error. On the other hand, in the measurement using the laser range sensor, although the estimation error is low and the measurement can be performed with high accuracy, the problem is that the resolution is low. Equipment that can be measured with high resolution is very expensive and requires large equipment, so realistic operation is difficult. In addition, an encoding error caused by encoding a depth image is one of the causes of loss of the depth image. In addition to the above, when viewed from an arbitrary viewpoint, it is difficult to restore a depth image even in a shielded area that is not captured by any observation camera, and this is one of the missing depth images. I can say that.

  One method for repairing a defect in a depth image is a method of filtering the depth image (see, for example, Non-Patent Document 1). Bilateral filters are used to remove only noise while leaving the original edges in the depth image. However, since the shape of the real scene is complex, depth boundaries may be mixed and only two-dimensional filtering is performed. Then, there is a defect that cannot be dealt with. In addition, a method for repairing a depth image using color information of a camera has been proposed (see, for example, Non-Patent Document 2).

  However, since it is performed independently for each camera depth image, it does not depend on the arbitrary viewpoint position, and there is a possibility of outputting inconsistent depth images for each viewpoint. Application is difficult.

T. Matsuo, N. Fukushima, Y. Ishibashi, "Weighted Joint Bilateral Filter with Slope Depth Compensation Filter for Depth Map Refinement," Proc. International Conference on Computer Vision Theory and Applications (VISAPP 2013), Feb. 2013 Color based depth up-sampling for depth compression M.O.Wildeboeer, T. Yendo, M.P.Tehrani, T. Fujii, M. Tanimoto Proceeding of Picture Coding Symposium 2010 (PCS)

  The video quality of an arbitrary viewpoint image largely depends on the video quality of a depth image. However, it is difficult to obtain the depth image accurately because the depth image is actually lost due to a depth estimation error due to stereo vision, a low resolution depth image due to the laser range sensor, or an encoding error due to the depth image encoding. In addition, there is a problem that the arbitrary viewpoint video deteriorates due to the loss of the depth image.

  The present invention has been made in view of such circumstances. When generating an arbitrary viewpoint image from a multi-viewpoint image and a depth image, the present invention focuses on the arbitrary viewpoint while detecting and repairing a missing portion of the depth image. An object is to provide an image processing apparatus, an image processing method, and an image processing program capable of generating an arbitrary viewpoint image.

  According to the present invention, an area dividing unit that divides each captured image at a plurality of viewpoints into small areas, and the pixels of the captured image are projected onto an arbitrary viewpoint image plane according to the depth value of each depth image, and the projected image is Generating and projecting the pixels of the depth image onto the arbitrary viewpoint image plane to generate a depth projection image; and for each pixel on the arbitrary viewpoint image, the projection image or the depth projection image Using the reliability measurement means for calculating the suitability of the luminance value of the pixel of the arbitrary viewpoint image, and the reliability of the pixel on the arbitrary viewpoint image using the suitability calculated by the reliability measurement means A pixel that is determined to be reliable by the determination unit and the luminance value of the pixel of the arbitrary viewpoint image from the luminance values of the plurality of pixels of the projection image. A layer image group storage unit that stores a value and a layer image in which the depth value of the pixel is determined; and a pixel that is determined to be unreliable by the determination unit is a pixel on the arbitrary viewpoint image The depth value is corrected using the reliability and depth value of other pixels in the small region to which the image belongs, and the processing by the pixel projecting unit, the reliability measuring unit, and the determining unit is all performed on the captured image. Area correction means that repeats until a sufficient area is projected, and a plurality of the arbitrary viewpoint image planes are filled with a hidden area due to a shielding area that may be generated when the layer image group is projected, or insufficiently reliable Layer correction means for expanding a composite region on the layer image by filling a region that has not been completed using already synthesized layer pixels, and one arbitrary viewpoint image plane On the other hand, after projecting each of the layer images, a matching process is performed on a portion where the depth is discontinuous, and an image synthesis means for outputting the arbitrary viewpoint image by synthesizing the images is provided. .

  According to the present invention, the reliability measurement means indicates depth connectivity indicating whether or not the projected images are not significantly different in depth from the neighboring pixels in the pixels on the arbitrary viewpoint image and the known pixels on the arbitrary viewpoint image. The suitability is calculated by calculating.

  In the present invention, the reliability measurement unit may indicate luminance affinity indicating whether the projected images are not significantly different from the neighboring pixels in the pixels on the arbitrary viewpoint image and the known pixels on the arbitrary viewpoint image. The suitability is calculated by calculating.

  In the present invention, when the captured image is a temporally continuous image, the reliability measuring unit may compare the pixel on the arbitrary viewpoint image and the pixel on the arbitrary viewpoint image between the projected images as compared to the immediately preceding frame. The suitability is calculated based on whether or not the luminance value of the pixel in the arbitrary viewpoint image is not significantly different from the luminance value of the pixel on the arbitrary viewpoint image.

  The present invention is an image processing program for causing a computer to function as the image processing apparatus.

  According to the present invention, when generating an arbitrary viewpoint image from a multi-viewpoint image and a depth image, it is possible to generate an arbitrary viewpoint image centered on an arbitrary viewpoint while detecting and repairing a missing portion of the depth image. An effect is obtained.

It is a block diagram which shows the structure of 1st Embodiment of this invention. It is a figure which shows the result of having performed area division. It is explanatory drawing which shows the processing operation which the pixel projection part 8 shown in FIG. 1 performs. It is explanatory drawing which shows operation | movement of the reliability measurement part 11 shown in FIG. It is a block diagram which shows the structure of 2nd Embodiment of this invention.

<First Embodiment>
Hereinafter, an image processing apparatus according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes a multi-viewpoint image storage unit that stores multi-viewpoint images. Reference numeral 2 denotes a multi-view depth image storage unit that stores multi-view depth images. Reference numeral 3 denotes a camera parameter storage unit that stores camera parameters. Reference numeral 4 denotes an arbitrary viewpoint parameter storage unit that stores arbitrary viewpoint parameters. Reference numeral 5 denotes a layer image group storage unit that stores a layer image group. Reference numeral 6 denotes an arbitrary viewpoint image storage unit that stores an image of an arbitrary viewpoint. Reference numeral 7 denotes an area dividing unit that performs area division on an image. Reference numeral 8 denotes a pixel projection unit that projects pixels. Reference numeral 9 denotes a depth connectivity measuring unit that measures depth connectivity. Reference numeral 10 denotes a luminance affinity measuring unit that measures luminance affinity. The depth connectivity measurement unit 9 and the luminance affinity measurement unit 10 constitute a reliability measurement unit 11. Reference numeral 12 denotes a suitability determination unit that determines suitability. Reference numeral 13 denotes an area correction unit that corrects an area, and reference numeral 14 denotes a layer correction part that corrects a layer. Reference numeral 15 denotes an image composition unit that performs image composition.

Next, the operation of the image processing apparatus shown in FIG. 1 will be described. First, the input elements, each memory unit in the stored N images (multi-view image) I _n, the depth image at the same viewpoint position and each of the multi-view image I _n, each camera capturing the multi-view image These are camera parameters (internal parameters and external parameters) and arbitrary viewpoint parameters.

First, the area dividing unit 7 performs area division in each image I _n input. For example, luminance clustering by K-means is used as the region division method, but other region division methods may be used. The result of area division is shown in FIG. FIG. 2 is a diagram showing the result of area division (actually a color image). As a parameter used at the time of division, a parameter depending only on luminance (color information) (R, G, B) is used, but a parameter depending on (R, G, B, D) combining depth information and color information. You may divide by. Here, D is depth information. In addition, the area division may not be divided as shown in FIG. 2, but may be divided into simple rectangles.

  When region segmentation using both luminance and depth is used, it is highly possible that all the pixels included in each region are composed of the same object. Therefore, more natural video composition in subsequent processing operations It can be performed. In the case of region division using only luminance, the number of locations where the region is divided so as to straddle the boundary between objects of different depths that are configured with similar colors increases, and the accuracy of the synthesized video is lower than the former. When a rectangle is used, since the area shape is not considered at all, it is difficult to be affected by the adjacent area, and the accuracy of the synthesized video is further deteriorated. However, since the processing time required for area division can be shortened, high-speed processing is possible. The size of the divided area and the rectangle varies depending on the scene, but it is desirable that the divided area is divided as finely as possible and one area is constituted by one type of object. In the confirmation by experiment, the full HD size image of the subject was divided into areas of about 20 × 20 pixels.

Next, pixel projection portion 8, each pixel of each image I _n of the multi-view image, the three-dimensional point in accordance with the camera parameters of the depth image D _n same pixel depth value and the camera n the (n is the identification number of the camera) And the projected image I ′ _n is obtained by projecting onto the arbitrary viewpoint image plane according to the arbitrary viewpoint parameters. At the same time, the pixel projection unit 8 also projects each depth image D _n to an arbitrary viewpoint, converts the depth image D _n into a depth value from the arbitrary viewpoint, and obtains a depth image D ′ _n after projection. As shown in FIG. 3, the pixels of each camera are projected onto the layer d corresponding to the value of D (depth). In other words, each pixel is projected according to the depth value of each viewpoint on the layer space formed by the arbitrary viewpoint. FIG. 3 is an explanatory diagram showing processing operations performed by the pixel projection unit 8 shown in FIG. The above processing is performed according to a mask image for indicating whether or not the processing is completed for each pixel. The update of the mask image is already performed based on the determination result of the suitability determination unit 12 in the following processing.

The reliability measurement unit 11 measures the reliability of each pixel by comparing the luminance and depth values of I ′ _n projected on the arbitrary viewpoint image plane (see FIG. 4). FIG. 4 is an explanatory diagram showing the operation of the reliability measuring unit 11 shown in FIG. As shown in FIG. 4, two indices measured by the depth connectivity measuring unit 9 and the luminance affinity measuring unit 10 shown in FIG. 1 are used for the measurement. Based on these indices, the determined pixel is determined based on the target pixel p and the projection pixel in the target range W. It is also possible to perform processing using only one index. In the case where only one index is used, what kind of defect is suppressed with respect to a depth image having a defect changes.

Depth connectivity measurement unit 9, shooting viewpoint to each other is installed between, projected from the camera n on the layer L _d that is set to the position of the depth d from the center position of the arbitrary viewpoint is the starting point for attempting to synthesize Depth connectivity E _depth (n, d, p) at pixel p has been defined as follows. Of the plurality of pixels q included in the neighborhood range W on the image plane among the plurality of layers before and after δ of the target pixel p, the depth value of the pixel q that has already been adopted as the layer, and the target pixel p The depth value of is used. f (p, q) is a function that is proportional to the reciprocal of the distance between two vectors p and q indicating the pixel position on the image, and the further away from p, the less affected by the function g. g (q, d, l) is a function that returns the reciprocal of the distance as a depth value with d, which is the depth at an arbitrary viewpoint of the target pixel p, only when the pixel q of the layer L _l has already been determined. . In both functions f (p, q) and g (q, d, l), a minute value ε is introduced to prevent zero division. This time, such a one-dimensional and two-dimensional Euclidean distance is used, but it may be used by replacing it with an index such as another-dimensional Euclidean distance.

The luminance affinity measuring unit 10 calculates the luminance affinity _Ecolor (n, d, p) at the pixel p projected from the camera n on the layer L _d set at the position of the depth d from the center of the arbitrary viewpoint as follows. Define as follows. A plurality of pixels included in the vicinity range W on the image plane in a plurality of layers before and after δ of the target pixel p are set as comparison pixels q. f (p, q) is a function that is proportional to the reciprocal of the secondary Euclidean distance on the image plane between the target pixel p and the comparison pixel q, and is affected by other function values as q is far from p. It is difficult. h (S (p), T (q)) is the reciprocal of the difference in luminance value between the RGB luminance value S (p) of the pixel p in the image S and the RGB luminance value T (q) of the pixel q in the image T. A small value ε is introduced to prevent zero division, which is a function that returns a proportional value. h (I ′ _n (p), L _l (q)) is the luminance value I _n ′ (p) of the pixel on the projection image and the luminance value L _l (q of the pixel already employed in the layer image. ) Mh (I ′ _n (p), I ′ _m (q)) is proportional to the luminance value I ′ _n (p) of the pixel on the projection image and on the projection image other than itself. A value that is proportional to the reciprocal of the difference from the luminance value I ′ _m (q) of the pixel is summed for all cameras, and γ is introduced for weighting.

The suitability determination unit 12 performs threshold processing on the following E (n, d, p) in each pixel, and determines a pixel on the layer if it is equal to or greater than the threshold. Processing is performed by the correction unit 13.
The luminance value L _d (p) at the pixel p on the layer L _{d at the} depth d is determined by the following equation. Here, δ is a coefficient for reducing an error caused by a difference in sampling interval between depth images. For 256 gradations used as the number of gradations of a general depth image, δ is about 2. . This time constants are used, but function values may be used. In that case, if the layer interval is not equal, it is necessary to make δ smaller as it goes farther in the depth direction.

  The region correction unit 13 changes the depth of the pixels that have not been adopted on the layer by using the depth values of the member pixels other than the own pixel in the region to which the pixel belongs and in the adjacent region. Among the member pixels in each region, an intermediate value of a set of depth values of pixels not yet adopted in the layer is calculated, and this intermediate value is assigned as a new depth value to the pixels that are not yet adopted in the layer. . In addition to the intermediate value, a neighborhood depth value and an average value of the intermediate value may be used. If only pixels equal to or less than the preset threshold number remain as member pixels, the layer correction unit determines the depth value as a pixel whose depth value is unknown, and the process is suspended.

Since the layer correction unit 14 may cause a defect on the arbitrary viewpoint image due to occlusion or a defect in the depth image, the layer correction unit 14 performs an interpolation on the layer image, so that the arbitrary viewpoint image that is three-dimensionally consistent can be obtained. Is synthesized. Among the pixels that have not been projected from any of the images I ′ _n due to occlusion of the objects and the pixels whose processing is suspended while the depth value is unknown, when the layer is projected to an arbitrary viewpoint In order to interpolate the luminance value for the pixels that remain unprojected, the layer is corrected. Among the neighboring pixels of the pixel p whose luminance value is unknown on the arbitrary viewpoint image, among the pixels whose luminance has already been determined, the pixels that are not in contact with the boundary region in the region divided image before projection, In addition, the pixel q having the farthest depth value is selected, and the same depth as the pixel q is given to the pixel p. If all of the neighboring pixels are in contact with the boundary region in the region-divided image before projection, the depth value of the farthest pixel in the arbitrary viewpoint image is given as the depth of the pixel p. The luminance value of the pixel p gives the same luminance as the pixel q. Although the same brightness is given this time, texture synthesis using a divided region including the pixel q may be performed. Further, interpolation processing such as Joint Bilateral Upsampling may be used.

  The image composition unit 15 projects the layer image group sequentially from the distant layer onto the arbitrary viewpoint image plane. For multiple pixels that are projected onto the same pixel, the brightness is mixed if the depth value is within a predetermined range, and if it is outside the range, only the brightness value of the pixel at the foremost depth value is used. Is adopted. The luminance value of the arbitrary viewpoint image is determined, and at the same time, the depth value of the adopted pixel is recorded to generate the depth image at the arbitrary viewpoint. Finally, the following matching process is performed to suppress unnaturalness on the boundary line based on the depth discontinuity between objects. First, an image obtained by applying a Sobel filter to a depth image of an arbitrary viewpoint image is stored as an edge image. Other methods may be used for edge detection. By applying a low-pass filter to an arbitrary viewpoint image so that the degree of blurring increases as the edge strength of the edge image increases, unnatural boundary lines at locations where the depth values between subjects are different can be suppressed. . Although such a matching method is used this time, another similar matching method may be used.

Second Embodiment
Next, an image processing apparatus according to a second embodiment of the present invention will be described. FIG. 5 is a block diagram showing the configuration of the embodiment. In FIG. 5, the same parts as those in the apparatus shown in FIG. The apparatus shown in FIG. 5 is different from the apparatus shown in FIG. 1 in that a luminance continuity measuring unit 16 is provided. If the input is not a multi-viewpoint image but a multi-viewpoint video (a series of temporally continuous sets of multi-viewpoint images), discontinuous composition of arbitrary viewpoint video can be performed by using continuity between frames. To avoid this, the following processing can be performed.

The reliability measuring unit 11 measures the reliability of each pixel by comparing the luminance and depth values of I ′ _n projected on the arbitrary viewpoint image plane. The measurement uses three indices measured by the depth connectivity measuring unit 9, the luminance affinity measuring unit 10, and the luminance continuity measuring unit 16.

The luminance continuity measuring unit 16 calculates the luminance continuity E _connection (n, d, p) at the pixel p projected from the camera n on the layer L _d set at the position of the depth d from the arbitrary viewpoint center as follows. Define as follows. The luminance L ″ _l (q) at the pixel q of the layer image at the depth l synthesized one frame before the processing frame is used. By comparing q, which is a neighboring pixel of the target pixel p, with the preceding and following frames, the previous frame as much as possible In this case, only a simple luminance change value is used, but a luminance value change index in the time direction similar to this may be used.

The suitability determination unit 12 performs threshold processing on the following E (n, d, p) in each pixel, and if it is equal to or greater than the threshold, a pixel on the layer is determined. The process is performed.

  As described above, in the situation where only one frame such as a still image is targeted, it is possible to repair the deterioration due to the loss of the depth image in a form that is difficult for humans to notice, but in the case of video, the frame Each time the defect and the repair result are different, the time discontinuity is conspicuous, resulting in an unnatural arbitrary viewpoint video. According to the configuration described above, it is possible to continuously synthesize an arbitrary viewpoint video by repair in consideration of temporal continuity. In addition, due to occlusion and lack of depth values, a region that is not projected from all observation viewpoints is generated, and pixels with unknown luminance values are generated on an arbitrary viewpoint image. A method for filling in the loss of an arbitrary viewpoint image using luminance information of neighboring pixels has been proposed, but because it depends only on the luminance information, it becomes an unnatural three-dimensional shape when changing the viewpoint, resulting in deterioration of the quality of the arbitrary viewpoint image. However, according to the configuration described above, such a problem can also be solved.

  According to the above-described image processing apparatus, it is possible to synthesize a natural arbitrary viewpoint image with less deterioration than normal rendering according to a depth image having a plurality of viewpoints missing. In particular, it is possible to reduce noise on an arbitrary viewpoint image caused by an encoding error that is likely to occur at a depth boundary, and to suppress multiple images caused by depth inconsistencies between viewpoints. In addition, it is possible to reduce flicker in the time direction due to variations in repair results on an arbitrary viewpoint video. By complementing the luminance and depth in the layer structure, it is possible to synthesize a natural arbitrary viewpoint image that is three-dimensionally matched even if there is a blockage or depth defect.

  You may make it implement | achieve the image processing apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  It can be applied to applications where it is essential to estimate the depth of a scene from multi-viewpoint images and depth information.

  DESCRIPTION OF SYMBOLS 1 ... Multi viewpoint image storage part, 2 ... Multi viewpoint depth image storage part, 3 ... Camera parameter storage part, 4 ... Arbitrary viewpoint parameter storage part, 5 ... Layer image group storage part, 6 ... Arbitrary viewpoint image storage unit, 7 ... Region division unit, 8 ... Pixel projection unit, 9 ... Depth connectivity measurement unit, 10 ... Luminance affinity measurement unit, 11 ... Reliability measurement unit, 12 ... suitability determination unit, 13 ... area correction unit, 14 ... layer correction unit, 15 ... image composition unit, 16 ... luminance continuity measurement unit

Claims (4)

Area dividing means for dividing each captured image at a plurality of viewpoints into small areas;
Projecting the pixels of the captured image onto an arbitrary viewpoint image plane according to the depth value of each depth image to generate a projection image, projecting the pixels of the depth image onto the arbitrary viewpoint image plane, and depth Pixel projection means for generating a projection image;
Reliability measurement means for calculating the suitability of luminance values of pixels of the arbitrary viewpoint image using the projection image or the depth projection image for each pixel on the arbitrary viewpoint image;
Determination means for determining the reliability of the pixel on the arbitrary viewpoint image using the compatibility calculated by the reliability measurement means;
For the pixels determined to be reliable by the determination means, the layer image in which the luminance value of the pixel of the arbitrary viewpoint image and the depth value of the pixel are determined from the luminance values of the pixels of the predetermined plurality of projection images Layer image group storage means for storing
For the pixels determined to be unreliable by the determination means, the depth value is corrected using the reliability and depth values of other pixels in the small area to which the pixel on the arbitrary viewpoint image belongs. And an area correcting unit that repeats the process by the pixel projecting unit, the reliability measuring unit, and the determining unit until all the regions of the captured image are sufficiently projected;
Layer pixels that have already been combined with respect to a hidden area due to a shielding area that may occur when the layer image group is projected on a plurality of arbitrary viewpoint image planes, or an area that has not been fully filled with reliability. Layer correction means for expanding the composite area on the layer image by performing the hole filling used;
Image synthesizing means for outputting the arbitrary viewpoint image by projecting each layer image onto one arbitrary viewpoint image plane, performing a matching process on a portion where the depth is discontinuous, and synthesizing the images; equipped with a,
When the captured image is a temporally continuous image, the reliability measuring unit includes a luminance value of a pixel on the arbitrary viewpoint image and a pixel on the arbitrary viewpoint image between the projected images compared to the immediately preceding frame. An image processing apparatus that calculates the suitability based on whether a luminance value of a pixel on the arbitrary viewpoint image is not significantly different from a neighboring pixel .   The reliability measuring unit calculates depth connectivity indicating whether the projected images are not significantly different in depth from the neighboring pixels in the pixels on the arbitrary viewpoint image and the known pixels on the arbitrary viewpoint image. The image processing apparatus according to claim 1, wherein the suitability is calculated by:   The reliability measurement means calculates a luminance affinity indicating whether the projected images are not significantly different from the neighboring pixels in the pixels on the arbitrary viewpoint image and the known pixels on the arbitrary viewpoint image. The image processing apparatus according to claim 1, wherein the compatibility is calculated by: The image processing program for a computer to function as an image processing apparatus according to any one of claims 1 to 3.