KR101241654B1 - Apparatus and method for editing disparity graph of stereoscopic image, image processing apparatus, and recording medium - Google Patents

Abstract

The present invention relates to an apparatus and method for adjusting the disparity graph compression of an stereoscopic image, an image processing apparatus, and a recording medium,
The apparatus for adjusting the disparity graph compression of the stereoscopic image may include: an initial disparity graph generator configured to generate an initial disparity graph of a stereoscopic image; An entire compression unit compressing the initial disparity graph using a Poisson equation; And a scene connection unit connecting the disparity graph compressed by the entire compression unit in a one-dimensional motion warping manner.

Description

Apparatus and method for editing disparity graph of stereoscopic image, image processing apparatus, and recording medium}

The present invention relates to a three-dimensional image editing technology, an apparatus and method for adjusting the disparity graph compression of the stereoscopic image to ensure the fixed point of the operator and the visual safety area of the viewer while having the initial disparity graph features, image processing apparatus , And a recording medium.

In general, a stereo camera such as a stereo camera or a multi-view camera uses two or more cameras, such as a human eye, to obtain images of left and right, and the viewers are displayed by the distance between the two images, that is, disparity. It makes the image feel three-dimensional. Specifically, the disparity between two images captured by the stereoscopic camera causes the operator to experience parallax between the two eyes, and such binocular parallax makes the stereoscopic sense.

As described above, disparity values representing depth in a stereoscopic image are data having a great influence on securing visual safety of an image viewer.

The disparity value range is limited for the visual safety of the viewer. Workers can't control the disparity, which is determined by various factors between the camera and the object, so they applied disparity to their work by creating a disparity graph. But simply creating a disparity graph has many limitations in creating the operator's desired scene and requires a skilled stereographer.

To make it easier for the operator to calculate a visually safe disparity range, a stereoscopic calculator has been proposed in the past. In addition, Visumotion's Maya plug-in from Autodestk provides a guide by showing the visual safety zone in the form of a box to the operator. However, this method has difficulty in controlling camera movement and disparity value.

There have been several proposals on how to adjust the disparity value. For example, when the initial disparity graph deviates from a preset condition, a method of collectively changing to a preset value (first method), or a method of linearly compressing a disparity graph collectively (second method) may be used. have.

However, the first method has a disadvantage in that the disparity design of the initial disparity graph does not save the disparity design previously made by the operator, and the second method does not effectively utilize the visual safety area. .

Accordingly, an object of the present invention is to provide an apparatus and method for compressing and adjusting a disparity graph of an stereoscopic image, an image processing apparatus, and a recording medium for effectively utilizing the visual safety region while maintaining the characteristics of the initial disparity graph of the stereoscopic image.

In addition, the present invention is to provide a disparity graph compression control apparatus and method, an image processing apparatus, and a recording medium of a three-dimensional image to support the partial correction of the disparity graph, to enable a more detailed editing operation.

In addition, the present invention provides an apparatus and method for compressing and adjusting a disparity graph of a stereoscopic image, an image processing apparatus, and a recording medium, by which the inter-scene connection of disparity values is naturally performed to ensure visual comfort of a visitor.

As a means for solving the above problems, in a first embodiment of the present invention, an initial disparity graph generator for generating a one-dimensional initial disparity graph for a stereoscopic image; An entire compression unit compressing the initial disparity graph using a Poisson equation; And a scene connection unit connecting the disparity graph compressed by the entire compression unit in a one-dimensional motion warping manner.

The compressed disparity graph has the same graph form as the initial disparity graph and is included in the visual safety area.

,

, s'= t, at start frame, |t| < safe zone (상기 D는 미분 매트릭스, 상기 s'는 압축전 디스패리티 벡터값, 상기 t 는 압축후 디스패리티 벡터값, 상기 start frame은 압축 시작점(또는 고정점), 상기 safe zone은 시각적 안전 영역)"의 식에 따라 상기 초기 디스패리티 그래프를 압축하는 것을 특징으로 한다. The full compression section "

,

, s' = t, at start frame, | t | <safe zone (where D is the differential matrix, s' is the disparity vector value before compression, t is the disparity vector value after compression, the start frame is the compression start point (or fixed point), and the safe zone is the visual safe zone). Compressing the initial disparity graph according to the equation

The apparatus for adjusting the disparity graph further includes a partial correcting unit provided to the scene connection unit after partially correcting the compressed disparity graph according to the partial correction range and the boundary condition information input by the operator. do.

(상기 D는 미분 매트릭스, 상기 s'는 압축전 디스패리티 벡터, 상기 t 는 압축후 디스패리티 벡터값, 상기 i 은 부분 수정될 프레임들(frames of local constraints), 상기 j은 부분 수정 범위의 시작 및 끝 프레임(start/end frames of local area), 상기 t_i,v_i은 부분 수정된 프레임들에서의 디스패리티 값(disparity values in locally constrained frames), 상기 t_j, s'_j 은 부분 수정 범위의 시작 및 끝 프레임에서의 디스패리티 값들임)"의 식에 따라, 상기 압축된 디스패리티 그래프를 부분 수정하는 것을 특징으로 한다. The partial modification is "

(Where D is a differential matrix, s' is a pre-compression disparity vector, t is a post-compression disparity vector value, i is frames of local constraints, j is the start of a partial modification range) And start / end frames of local area, t _i and v _i are disparity values in locally constrained frames, and t _j and s' _j are partial modification ranges. The disparity values in the start and end frames of &lt; RTI ID = 0.0 &gt;).&Lt; / RTI &gt;

The initial disparity graph generator calculates disparity for each object by comparing and analyzing the left and right images of the stereoscopic image, and converts the disparity for each object into a one-dimensional graph to generate the initial disparity graph. The initial disparity graph may be a one-dimensional graph.

As a means for solving the said subject, 2nd Embodiment of this invention provides the image editing apparatus containing the disparity graph adjustment apparatus which concerns on 1st Embodiment of this invention.

As a means for solving the above problems, in a third embodiment of the present invention, generating a one-dimensional initial disparity graph for a stereoscopic image; Compressing the initial disparity graph entirely through a Poisson equation; Partially modifying the compressed disparity graph according to information about a partial correction range and a boundary condition input by an operator; And connecting the partially corrected disparity graph by a one-dimensional motion warping method.

The compressed disparity graph is included in the visual safety region and has the same graph form as the initial disparity graph.

,

,, s'= t, at start frame, |t| < safe zone (상기 D는 미분 매트릭스, 상기 s'는 압축전 디스패리티 벡터값, 상기 t 는 압축후 디스패리티 벡터값, 상기 start frame은 압축 시작점(또는 고정점), 상기 safe zone은 시각적 안전 영역)"의 식에 따라 상기 초기 디스패리티 그래프를 압축하는 것을 특징으로 한다. Compressing the initial disparity graph entirely may include:

,

,, s' = t, at start frame, | t | <safe zone (where D is the differential matrix, s' is the disparity vector value before compression, t is the disparity vector value after compression, the start frame is the compression start point (or fixed point), and the safe zone is the visual safe zone). Compressing the initial disparity graph according to the equation

(상기 D는 미분 매트릭스, 상기 s'는 압축전 디스패리티 벡터, 상기 t 는 압축후 디스패리티 벡터값, 상기 i 은 부분 수정될 프레임들(frames of local constraints), 상기 j은 부분 수정 범위의 시작 및 끝 프레임(start/end frames of local area), 상기 t_i,v_i은 부분 수정된 프레임들에서의 디스패리티 값(disparity values in locally constrained frames), 상기 t_j, s'_j 은 부분 수정 범위의 시작 및 끝 프레임에서의 디스패리티 값들임)"의 식에 따라, 상기 압축된 디스패리티 그래프를 부분 수정하는 것을 특징으로 한다. The partial modification step is "

(Where D is a differential matrix, s' is a pre-compression disparity vector, t is a post-compression disparity vector value, i is frames of local constraints, j is the start of a partial modification range) And start / end frames of local area, t _i and v _i are disparity values in locally constrained frames, and t _j and s' _j are partial modification ranges. The disparity values in the start and end frames of &lt; RTI ID = 0.0 &gt;).&Lt; / RTI &gt;

In the generating of the initial disparity graph, the disparity for each object is calculated by comparing and analyzing the left and right images of the stereoscopic image, and converting the disparity for each object into a one-dimensional graph form to form the initial disparity graph. And the initial disparity graph is a one-dimensional graph.

As a means for solving the above problems, in the fourth embodiment of the present invention, a recording medium on which a program for performing the disparity graph adjusting method according to the third embodiment of the present invention is provided.

The present invention provides an environment in which the operator can freely disparity design according to storytelling without considering visual comfort. That is, the compression operation is performed while maintaining the characteristics of the initial disparity graph designed by the operator, and the compressed disparity graph is stably included in the visual safety region.

In addition, by allowing the initial disparity graph to have a one-dimensional graph form, the partial correction of the graph and the connection between the scenes are made naturally. In addition, by connecting object-specific disparity values through one-dimensional motion warping technology, viewers can feel visual comfort in the scene transition scene.

1 is a diagram illustrating an apparatus for adjusting a disparity graph of a stereoscopic image according to an exemplary embodiment of the present invention.
FIG. 2 is a diagram for describing a correlation between disparity and depth that vary depending on a distance between a camera and an object.
3 illustrates an example of a disparity graph compressed according to an embodiment of the present invention.
4 is a diagram illustrating an example of a disparity graph in which partial modification is performed according to an embodiment of the present invention.
5A is a diagram illustrating an example of changing a disparity graph before and after a scene connection according to an embodiment of the present invention.
5B is a diagram illustrating an example of an input image and an output image of a scene connection unit according to an embodiment of the present invention.
6A illustrates a rendering result before disparity graph compression according to an embodiment of the present invention.
6B is a diagram illustrating a rendering result after disparity graph compression according to an embodiment of the present invention.
7 is a view for explaining a disparity graph adjustment method according to an embodiment of the present invention.
8 is a view for explaining the effect of the disparity graph adjustment method according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

In order to clearly illustrate the present invention, parts not related to the description are omitted, and like parts are denoted by similar reference numerals throughout the specification.

In addition, when a part is said to "include" a certain component, this means that it may further include other components, except to exclude other components unless otherwise stated.

1 is a diagram illustrating an apparatus for adjusting a disparity graph of a stereoscopic image according to an exemplary embodiment of the present invention.

Referring to FIG. 1, an apparatus for adjusting a disparity graph of a stereoscopic image according to an embodiment of the present invention may include an input unit 100, a left image buffer 111, a right image buffer 112, and an initial disparity graph generator 120. ), An initial disparity graph editing unit 130, a scene connection unit 140, a rendering unit 150, and the like, and the initial disparity graph editing unit 130 includes a total compression unit 131, and a partial correction unit 132. And the like.

The input unit 100 receives a stereoscopic image provided from a stereoscopic image acquisition means such as a stereoscopic camera, and separates the stereoscopic image into a left image and a right image.

The left image buffer 111 temporarily stores the left image separated by the input unit 100, and the right image buffer 112 temporarily stores the right image separated by the input unit 100.

The initial disparity graph generator 120 compares and analyzes the left image and the right image to calculate horizontal distance deviations (ie, disparity for each object) for each object simultaneously present in the left image and the right image, and for each object. The initial disparity graph is generated by converting disparity into a one-dimensional graph. In this case, the initial disparity graph has a one-dimensional graph form in order to allow a worker to later perform partial modification and scene connection of the disparity graph more easily.

If the object is close to the camera, the disparity has a large value. On the contrary, as the object moves away from the camera, the disparity has a small value. Accordingly, in the present invention, the disparity value of the scene may be obtained from the disparity for each object, and an initial disparity graph reflecting the scene may be generated.

FIG. 2 is a diagram illustrating a correlation between a disparity and a depth depending on a distance between a camera and an object. Referring to FIG. 2, a correlation between a disparity and a depth has an inverse relationship, and the disparity is represented by the following equation. It can be seen that it is expressed as in 1.

d = xl-xr

Where Z is the focal length of the camera, B is the distance between cameras, d is the disparity, xl is the object position in the left image, xr is the object position in the right image, and Z is the depth.

The entire compression unit 131 compresses the initial disparity graph as a whole using the Poisson equation, so that the compressed disparity graph has an initial disparity graph shape and is included in the visual safe zone. .

Since the Poisson equation defines the derivative of the disparity graph over time, the visual derivative of the disparity graph can be calculated, and the visual derivative at this time represents the difference value for each frame. Accordingly, the present invention calculates the visual derivative of the initial disparity graph using the Poisson equation, derives a disparity graph having the visual derivative with the smallest deviation (most preferably "0"), and compresses it. By obtaining the disparity graph, the compressed disparity graph has the same graph form as the initial disparity graph and is stably included in the visual safety region as shown in FIG. 3.

In this case, the visual safety area may be actively set or changed in consideration of the '3D safety guidelines' issued by the Korea Information and Communication Technology Association, the system application environment, etc., but most preferably, the angle formed by the camera and the object is 1.5 degrees. Make it to a smaller range.

Such a compression process of the entire compression unit 131 may be represented by Equation 2.

 s' = t, at start frame,

| t | <safe zone

In this case, D is a differential matrix, s' is a disparity vector value before compression, t is a disparity vector value after compression, start frame is a compression start point (or fixed point), safe zone is a visual safety zone.

The partial correction 132 is selectively activated when a partial correction operation is requested by an operator. If the worker requests a partial correction operation and inputs information on the partial correction range and the boundary condition, the partial correction unit 132 performs partial modification of the compressed disparity graph in response. That is, in the present invention, by allowing only a specific region of the compressed disparity graph to be selectively modified, the operator guarantees finer modification than the disparity graph.

The partial correction process of the partial correction unit 132 may be expressed by Equation 3 below.

Where i is frames of local constraints, j is start / end frames of local area, t _i , v _i are discs in partially modified frames. Parity values in locally constrained frames, t _j , s' _j, represent the disparity values in the start and end frames of the partial modification range, respectively.

4 illustrates an example of a disparity graph in which partial correction is performed by the partial correction unit 132, and the partial correction unit 132 falls within the partial correction range selected by the operator as shown in FIG. It can be seen that the disparity value is modified to a boundary condition, that is, worker fixed points.

The scene connection unit 140 performs scene connection based on the disparity graph extraction, and uses a one-dimensional motion warping technique for a natural connection of the disparity graph.

5A illustrates an example of changing a disparity graph before and after a scene connection according to an embodiment of the present invention, and FIG. 5B illustrates an example of an input image and an output image of the scene connection unit 140.

That is, even if there is a large difference between the disparity in the scene 1 (# 1) and the disparity of the object in the scene 2 (# 2) before the scene connection, the scene connection unit 140 connects them through the one-dimensional motion warping technique. This allows the viewer to feel visual comfort after a scene connection.

The rendering unit 150 renders an image in which the scene connection is completed and outputs the final result.

6A and 6B illustrate a rendering result before and after disparity graph compression according to an exemplary embodiment of the present invention.

6A is a result of rendering before adjusting the disparity graph, and FIG. 6B is a result of rendering after adjusting the disparity graph. Comparing these figures, the disparity of each object is drastically reduced after the disparity graph is adjusted as compared to before the disparity graph is adjusted. It can be seen that. That is, after adjusting the disparity graph according to the present invention, the disparity of the object is included in the visual safety region, so that the viewer can visually comfortably watch the corresponding image.

7 is a view for explaining a disparity graph adjustment method according to an embodiment of the present invention.

First, when a stereoscopic image is received, it is divided into a left image and a right image. The left and right images are compared and analyzed to calculate horizontal distance deviations (i.e., disparity) for each object, and to generate a one-dimensional initial disparity graph by converting all of the calculated disparities for each object into disparity values. (S1).

The initial disparity graph is compressed using the Poisson equation to maintain the shape of the initial disparity graph and to include all object-specific disparities in the visual safety region (S2).

In this state, if the worker requests partial modification of the compressed disparity graph (S3), the operator receives information on the partial modification range and boundary condition (S4).

The disparity graph is partially modified based on the partial correction range and boundary condition input by the operator. That is, the disparity value of the region belonging to the partial correction range selected by the operator among the entire regions of the disparity graph is adjusted to satisfy the boundary condition input by the operator (S5).

The disparity value for each object is connected through one-dimensional motion warping technology (S6), and the final image is output through the rendering operation (S6).

8 is a view for explaining the effect of the disparity graph adjustment method according to an embodiment of the present invention.

In FIG. 8, (a) shows a disparity graph before compression, and (b) shows a method of changing the disparity graph to a predetermined value when it is out of a predetermined condition according to the conventional art. As a result of using the method of linearly compressing the disparity graph according to the present invention, (d) shows the results of using the method of compressing the disparity graph according to the disparity graph adjusting method of the present invention.

That is, conventionally, as shown in (b), there is a problem in that the disparity in the form of the initial disparity graph is not generated so as not to save the disparity design previously made by the operator, or as shown in (c). As can be seen, there is a problem that does not effectively utilize the visual safety area.

However, the present invention compresses the initial disparity graph as a whole using the Poisson equation as described above, so that the compressed disparity graph has the same graph form as the initial disparity graph and is also stably included in the visual safety region.

In addition, the present invention is not only the above method of adjusting the disparity graph, but also executed through a computer, and the initial disparity editing to maintain the initial disparity graph characteristics while ensuring the operator's fixed point and the viewer's visual safety area. A method of controlling a disparity graph of a stereoscopic image that provides a function and a scene connection function to ensure a viewer's visual comfort, and also includes a recording medium on which a computer executable program is recorded.

However, since the method for adjusting the disparity graph of the stereoscopic image included in the program is the same as described above, a detailed description thereof will be omitted.

Although described with reference to the embodiments above, those skilled in the art will understand that the present invention can be variously modified and changed without departing from the spirit and scope of the invention as set forth in the claims below. Could be.

100: input unit 111: left image buffer
112: right image buffer 120: initial disparity graph generator
130: initial disparity graph editing unit 131: whole compression unit
132: partial correction 140: thin connection
150: rendering unit

Claims (13)

An initial disparity graph generator for generating a one-dimensional initial disparity graph for the stereoscopic image;
An entire compression unit compressing the initial disparity graph using a Poisson equation; And
And a scene connection unit connecting the disparity graph compressed by the entire compression unit using a one-dimensional motion warping method. The method of claim 1, wherein the compressed disparity graph
Disparity graph control device of the stereoscopic image having the same graph form as the initial disparity graph, included in the visual safety area. The method of claim 1, wherein the entire compression unit
Disparity graph adjusting apparatus, characterized in that for compressing the initial disparity graph according to the following equation,

,

s' = t, at start frame,
| t | <safe zone
Where D is a differential matrix, s' is a pre-compression disparity vector value, t is a post-compression disparity vector value, the start frame is a compression start point (or fixed point), and the safe zone is a visual safe zone. The method of claim 1,
Partial correction of the compressed disparity graph according to the information on the partial correction range and the boundary condition input by the operator, and further comprising a partial correction unit provided to the scene connection unit. Regulating device. The method of claim 4, wherein the partial correction portion
Disparity graph adjusting device, characterized in that for partially modifying the compressed disparity graph according to the following equation,

D is the differential matrix, s' is the precompression disparity vector, t is the postcompression disparity vector value, i is frames of local constraints, j is the beginning of the partial modification range and Start / end frames of local area, t _i and v _i are disparity values in locally constrained frames, and t _j and s' _j are partial modification ranges. Disparity values in the start and end frames. The apparatus of claim 1, wherein the initial disparity graph generator
Comparing and analyzing the left image and the right image of the stereoscopic image to calculate the disparity for each object, and converts the disparity for each object into a one-dimensional graph form to generate the initial disparity graph of the stereoscopic image Disparity Graph Control. An image editing apparatus comprising a disparity graph adjusting apparatus for a stereoscopic image according to any one of claims 1 to 6. Generating a one-dimensional initial disparity graph for the stereoscopic image;
Compressing the initial disparity graph entirely through a Poisson equation;
Partially modifying the compressed disparity graph according to information about a partial correction range and a boundary condition input by an operator; And
Connecting the partially corrected disparity graph by a one-dimensional motion warping method. The method of claim 8, wherein the compressed disparity graph is
The disparity graph adjustment method of the stereoscopic image, which is included in the visual safety region and has the same graph form as the initial disparity graph. 9. The method of claim 8, wherein compressing the initial disparity graph entirely
Disparity graph adjustment method, characterized in that for compressing the initial disparity graph according to the following equation,

,

s' = t, at start frame,
| t | <safe zone
Where D is a differential matrix, s' is a pre-compression disparity vector value, t is a post-compression disparity vector value, the start frame is a compression start point (or fixed point), and the safe zone is a visual safe zone. The method of claim 8, wherein the step of modifying
Disparity graph adjustment method, characterized in that for partially modifying the compressed disparity graph according to the following equation,

D is the differential matrix, s' is the precompression disparity vector, t is the postcompression disparity vector value, i is frames of local constraints, j is the beginning of the partial modification range and Start / end frames of local area, t _i and v _i are disparity values in locally constrained frames, and t _j and s' _j are partial modification ranges. Disparity values in the start and end frames. The method of claim 8, wherein generating the initial disparity graph
Comparing and analyzing the left image and the right image of the stereoscopic image to calculate the disparity for each object, and converts the disparity for each object into a one-dimensional graph form to generate the initial disparity graph of the stereoscopic image How to adjust the disparity graph. A recording medium having recorded thereon a program for performing the method for adjusting the disparity graph of the stereoscopic image according to any one of claims 8 to 12.

Images