KR102093622B1 - Method and apparatus for real-time correction of projector image using depth camera - Google Patents

Abstract

The present invention relates to a method and apparatus for real-time projector image correction. In the present invention, in a real-time projector image correction method, a three-dimensional position of a depth camera coordinate system using image coordinates of a specific point included in an image projected through a projector and three-dimensional coordinates of the specific point obtained using a depth camera Calculating a calibration transform coefficient indicating a correlation between information and 3D position information of the projector coordinate system, and converting the 3D coordinates of the depth image obtained from the depth camera into normalized image coordinates using the calibration transform coefficient , If the normalized image coordinates are included in the image size range, setting the coordinates as a projection prediction area, detecting an outer pixel of the projection prediction area, and calculating the depth camera coordinate system 3D coordinates of the outer pixel Step, the calibration conversion factor Converting the 3D coordinates of the depth camera coordinate system of the outer pixel into the 3D coordinates of the projector coordinate system using the depth-projector coordinate system transformation matrix derived by using the 3D coordinate information of the projector coordinate system based on the user's position Converting the projector coordinate system 3D coordinates of the outer pixel into a user coordinate system 3D coordinate using a projector-user coordinate system transformation matrix that converts the 3D position information into 3D coordinates of the user pixel, and converting the user coordinate system 3D coordinates of the outer pixel according to the user position. And projecting to detect a two-dimensional user viewpoint correction area, and correcting a pixel belonging to the user viewpoint correction area in a projection image projected by the projector. According to the present invention, the projection image can be corrected in real time without distortion even when the projection surface is continuously changing.

Description

METHOD AND APPARATUS FOR REAL-TIME CORRECTION OF PROJECTOR IMAGE USING DEPTH CAMERA}

The present invention relates to a real-time projector image correction method and apparatus, and more specifically, to predict a shape to be distorted when projecting a projector image using a depth camera, and to correct a projected image in real time based on the predicted distortion and the user's viewpoint. It relates to a method and apparatus that can be output.

The distribution and use of projectors are increasing due to the demand for a large screen for the construction of a home theater and a presentation environment, and the falling price of the projector. The projector is evolving from a traditional three-tube beam projector to a liquid crystal display (LCD) and digital light processing (DLP) method with good color reproducibility. As a result of the development of technology to improve the performance of the projector, bright and clear ANSI lumens, high contrast, and high resolution projectors have been developed. Recently, a short-focus projector that saves space and a portable battery built-in battery Small projectors have also been developed and are actively spreading.

The projector has the advantage of being cheaper than the screen size compared to a video device such as a television. However, if the installed projector is not perpendicular to the screen, there is a problem in that distortion occurs on the screen according to the non-planar state of the screen.

In the past, it was common for expensive projectors to be used as separate flat screens, and because the position of the projectors was fixed, there was little distortion caused by non-planar screens. Therefore, the user could use a distortion-free screen by adjusting the keystoning only during installation.

However, in recent years, a similar plane such as a wall surface is often used as a screen without a separate flat screen, and as the projector is miniaturized and portability is improved, a non-planar screen is often used as a screen. Since non-planar screen distortion occurring in such an environment is very difficult to correct using a conventional manual keystoning correction method, many studies have been conducted to correct the projection image distortion according to the screen.

As a typical method of correcting a projected image distortion according to a non-flat screen, there is a method using a correction pattern. This method projects a specific pattern for correction (sprite pattern, droplet pattern, etc.) using a projector and analyzes the distorted pattern by receiving the actual distorted image from the camera. Then, the distortion of the screen is determined according to the analyzed pattern, and a correction image according to the analyzed pattern is generated to perform correction. This method has an advantage in that calibration can be performed using a relatively inexpensive general camera and projector image.

However, the method using the correction pattern and the camera has a problem that the projector cannot be used when performing the correction, and therefore the projector image correction cannot be performed in real time in a situation where the projection surface is constantly changing. For example, a situation where the projection surface is not securely fixed and is easily shaken or distorted, a situation where the projection surface and the person frequently overlap due to a lot of people passing through, and the shape of the projecting projection screen is constantly changing (recall reality using the projector) It is difficult to use the method using the existing correction pattern. Therefore, a new method for viewing a projected image without distortion is required even when the shape of the screen is constantly changing due to the constantly changing projection surface.

An object of the present invention is to provide a real-time projector image correction method capable of correcting a projected image in real time without distortion even when the projection surface is continuously changing, in order to solve the aforementioned problems.

Another object of the present invention is to significantly reduce processing time by rapidly acquiring three-dimensional coordinates of a projected image using a recursive Gaussian weight interpolation method.

In addition, another object of the present invention is to quickly generate a correction image by detecting a correction area at a user's point of view and correcting only the corresponding area.

In addition, another object of the present invention is to increase the accuracy and application speed of the correction of the projector image using a depth camera.

In the present invention, in a real-time projector image correction method, a three-dimensional position of a depth camera coordinate system using image coordinates of a specific point included in an image projected through a projector and three-dimensional coordinates of the specific point obtained using a depth camera Calculating a calibration transform coefficient indicating a correlation between information and 3D position information of the projector coordinate system, and converting the 3D coordinates of the depth image obtained from the depth camera into normalized image coordinates using the calibration transform coefficient , If the normalized image coordinates are included in the image size range, setting the coordinates as a projection prediction area, detecting an outer pixel of the projection prediction area, and calculating the depth camera coordinate system 3D coordinates of the outer pixel Step, the calibration conversion factor Converting the 3D coordinates of the depth camera coordinate system of the outer pixel into the 3D coordinates of the projector coordinate system by using the depth-projector coordinate system transformation matrix, and the 3D position information of the projector coordinate system based on the user's position Converting the projector coordinate system 3D coordinates of the outer pixel into a user coordinate system 3D coordinate using a projector-user coordinate system transformation matrix that converts the 3D position information into 3D coordinates, and converting the user coordinate system 3D coordinates of the outer pixel according to a user position. And projecting to detect a two-dimensional user viewpoint correction area, and correcting a pixel belonging to the user viewpoint correction area in a projection image projected by the projector.

In addition, in the real-time projector image correction apparatus, the present invention provides a three-dimensional depth camera coordinate system using the coordinates of a specific point included in the projected image projected through the projector and the three-dimensional coordinates of the specific point obtained using a depth camera. A calibration unit that calculates a calibration conversion coefficient indicating a correlation between the position information and the 3D position information of the projector coordinate system, and converts the 3D coordinates of the depth image obtained from the depth camera into normalized image coordinates using the calibration conversion coefficient And, when the normalized image coordinates are included in the projection image size range, a projection area prediction unit that sets the coordinates as a projection prediction area, and predicts a correction area of a user's viewpoint based on the user's position using the projection prediction area Correction area prediction unit, the projector An image correcting unit for correcting a pixel belonging to the user's viewpoint correction region in a projected projected image, the correction region predicting unit detecting an outer pixel of the projection predicting region, an outer pixel detecting unit, a depth of the outer pixel camera coordinate system 3D A first coordinate system conversion unit that calculates coordinates and converts a 3D coordinate of a camera coordinate system to a 3D coordinate of a projector coordinate system using a depth-projector coordinate system transformation matrix derived using the calibration transform coefficient, a projector coordinate system A 3D coordinate system that converts 3D coordinates of the projector coordinate system of the outer pixel into 3D coordinates of the user coordinate system by using a projector-user coordinate system transformation matrix that converts 3D location information of the user coordinate system to 3D location information based on a user location. 2 coordinate system conversion unit, the outer pick Another feature is to include a correction area detection unit that detects a 2D user viewpoint correction area by projecting a 3D coordinate of a user coordinate system of a cell according to a user position.

According to the present invention as described above, the projection image can be corrected in real time without distortion even when the projection surface continuously changes.

In addition, according to the present invention, by using the recursive Gaussian weight interpolation method, the 3D coordinates of the projected image are rapidly obtained, thereby significantly reducing the processing time compared to the conventional method.

In addition, according to the present invention, a correction region at a user's viewpoint is detected and corrected only with the corresponding region, so that a correction image can be quickly generated.

In addition, according to the present invention, it is possible to increase the accuracy and application speed of the correction of the projector image by using the depth camera.

1 is a view showing a real-time projector image correction system according to an embodiment of the present invention,
Figure 2 is a block diagram for explaining the configuration of a real-time projector image correction apparatus according to an embodiment of the present invention,
3 is a block diagram for explaining a configuration of a correction region prediction unit according to an embodiment of the present invention;
4 is a view for explaining a depth camera coordinate system and a projector coordinate system according to an embodiment of the present invention;
5 is a view for explaining the operation of the calibration unit according to an embodiment of the present invention,
6 is a view for explaining the operation of the projection area prediction unit according to an embodiment of the present invention,
7 is a view showing the overall process of a real-time projection correction method at the user's point of view according to an embodiment of the present invention,
8 is a diagram illustrating an example of generating a rare projection depth image according to an embodiment of the present invention;
9 is a diagram illustrating a method for generating an interpolation projection depth image according to an embodiment of the present invention,
10 is a flowchart illustrating a real-time projector image correction method according to an embodiment of the present invention,
11 is a flow chart for explaining a correction area correction method according to an embodiment of the present invention,
12 is a flowchart illustrating a method for generating an interpolation projection depth image according to an embodiment of the present invention,
13 is a flowchart illustrating a pixel correction method according to an embodiment of the present invention.

The above-described objects, features, and advantages will be described in detail below with reference to the accompanying drawings, and accordingly, a person skilled in the art to which the present invention pertains can easily implement the technical spirit of the present invention. In the description of the present invention, when it is determined that detailed descriptions of known technologies related to the present invention may unnecessarily obscure the subject matter of the present invention, detailed descriptions will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same reference numbers in the drawings are used to indicate the same or similar elements, and all combinations described in the specification and claims can be combined in any way. And it should be understood that unless otherwise specified, a reference to a singular may include one or more, and a reference to a singular expression may also include a plural expression.

1 is a block diagram illustrating a real-time projector image correction system 10 (hereinafter referred to as 'image correction system 10') according to an embodiment of the present invention. The image correction system 10 may include a depth camera 50, a real-time projector image correction apparatus 100 (hereinafter referred to as 'image correction apparatus 100'), a projector 200, and an image correction apparatus ( 100) may be physically included in the projector 200 to perform a function, and may operate in connection with the depth camera 50 and the projector 200 as a separate device from the projector 200.

The image correcting apparatus 100 may transmit a control signal for obtaining a depth image to the depth camera 50, and may acquire a depth image from the depth camera 50. In addition, the image correction device 100 may correct the projected image projected through the projector in real time, and provide the corrected projected image to the projector 200 so that the projector 200 projects the corrected projected image. .

2 is a block diagram illustrating a configuration of a real-time projector image correction apparatus 100 according to an embodiment of the present invention. Referring to FIG. 2, the image correction device 100 according to an embodiment of the present invention includes a calibration unit 130, a projection area prediction unit 150, a correction area prediction unit 170, and an image correction unit 190. It can contain.

The calibration unit 130 uses 3D coordinates of the depth camera coordinate system and 3D of the projector coordinate system using the 3D coordinates of the specific point obtained using the coordinates and the depth camera of a specific point included in the projected image projected through the projector. A calibration transform coefficient indicating a correlation between dimensional position information can be calculated. In addition, a depth-projector coordinate system transformation matrix can be derived using a calibration transform coefficient. The depth-projector coordinate system transformation matrix means a transformation matrix that converts a depth camera coordinate system to a projector coordinate system.

This is to increase the prediction accuracy of the irregular projection area made by the projection area prediction unit 150, and the calibration unit 130 performs calibration by mathematically calculating a relative positional relationship between the depth camera 50 and the projector 200.

To this end, the calibration unit 130 needs to know exactly where the depth image obtained using the depth camera exists in the actual 3D space, and what point of the projected image projected through the projector is in the actual 3D space. You need to know if you are in a location.

In order to perform the calibration, the calibration unit 130 defines a depth camera coordinate system with the z-axis as the origin of the depth camera, the x-axis as the right side of the measurement direction, and the y-axis as the top as the origin of the depth camera, as shown in FIG. 4. You can define the projector coordinate system with the measurement position as the origin and the measurement direction as the x-axis, the right side of the measurement as the x-axis, and the top as the y-axis.

, 프로젝터 좌표계의 한 점을

로 설정하고, 이를 바탕으로 캘리브레이션을 수행할 수 있다. In one embodiment of the invention, the calibration unit 130 is one point of the depth camera coordinate system

, A point in the projector coordinate system

Set to, and based on this, calibration can be performed.

Since the depth camera provides an image having a depth value, 3D position information in a depth camera coordinate system can be obtained from the measured depth image.

값을 이용하여 특정 지점(

)의 깊이 카메라 좌표계에서의 3차원 좌표(

)를 계산할 수 있다. 이는 좀 더 정확한 깊이 카메라 좌표계에서의 3차원 위치 정보를 계산하기 위한 것으로, 캘리브레이션부(130)는 깊이 카메라의 렌즈 왜곡을 고려하여 렌즈의 반경 및 접선 계수를 이용한 [수학식 1]을 이용하여 깊이 영상을 보정한 후 3차원 변환을 수행할 수 있다. The calibration unit 130 is a pre-operation for the calibration, and the camera-specific parameters of the depth camera

Use a value to point

3D coordinates in the depth camera coordinate system (

). This is for calculating 3D position information in a more accurate depth camera coordinate system, and the calibration unit 130 uses depth [Equation 1] using the radius and tangential coefficient of the lens in consideration of the lens distortion of the depth camera. After the image is corrected, 3D transformation may be performed.

는 왜곡된 영상의 좌표 값이고

는 왜곡되지 않은 영상의 좌표 값,

은 n차 반경 왜곡 계수,

은 n차 접선 왜곡 계수이다. 본 발명의 일 실시 예에 의한 캘리브레이션부(130)는 깊이 카메라(50)에서 제공하는 보정 파라미터를 사용하거나 깊이 영상과 특정 패턴(체스 보드, 물방울 패턴 등)을 이용한 왜곡 계수 계산 방법을 사용할 수 있다. In the above equation

Is the coordinate value of the distorted image

Is the coordinate value of the undistorted image,

Is the nth-order radius distortion factor,

Is the nth order tangential distortion coefficient. The calibration unit 130 according to an embodiment of the present invention may use a correction parameter provided by the depth camera 50 or a method of calculating a distortion coefficient using a depth image and a specific pattern (chess board, droplet pattern, etc.). .

The calibration unit 130 may perform calibration to calculate the relative positions of the depth camera and the projector. At this time, the relative positional relationship can be expressed by a homogeneous transform matrix, which can be expressed as follows. In this specification, a matrix representing a relative positional relationship between a depth camera and a projector is referred to as a depth-projector coordinate system transformation matrix.

는 깊이 카메라 좌표계의 한 점을 나타내고

는 프로젝터 좌표계의 한 점,

은 회전 변환,

는 위치 변환을 나타낸다. 깊이-프로젝터 좌표계 변환 행렬은 깊이 영상으로부터 구해진 깊이 카메라 좌표계의 3차원 위치 정보를 프로젝터 좌표계 3차원 위치 정보로 변환할 수 있다. In [Equation 2]

Denotes a point in the depth camera coordinate system

Is a point in the projector coordinate system,

Silver rotation transformation,

Indicates position conversion. The depth-projector coordinate system transformation matrix may convert 3D position information of the depth camera coordinate system obtained from the depth image into 3D position information of the projector coordinate system.

Since there is no method for separately measuring the 3D position information of the projector coordinate system, the calibration unit 130 may use the coordinates of the projected image (image) projected by the projector to obtain a transformation matrix value indicating a relative positional relationship.

)와 이미지의 크기

를 이용하여 원점이 중심이고 y축 방향이 반대인 정규화된 이미지 좌표값 (

)을 다음과 같이 계산할 수 있다. Referring to FIG. 5, the calibration unit 130 includes image coordinates of a specific point included in an image to be projected by the projector (

) And the size of the image

Normalized image coordinate values with the origin at the center and opposite the y-axis (

) Can be calculated as:

라고 할 때, 캘리브레이션부(130)는 [수학식 4]를 이용하여 프로젝터의 초점 길이(Focal length)인

를 계산할 수 있다.Next, the projector's Field of View

When it is said, the calibration unit 130 is the focal length (Focal length) of the projector using [Equation 4]

Can be calculated.

와

를 이용하여 프로젝터 좌표계의 3차원 좌표를 정규화된 이미지 좌표값 (

)로 변환하는 식은 다음과 같다. Calculated from the above formula

Wow

Using 3D coordinates of the projector coordinate system, normalized image coordinate values (

The formula to convert) is as follows.

Calculating by substituting the normalized image coordinate calculated in [Equation 5] into the depth-projector coordinate system transformation matrix is as follows.

으로 나눠주고 나눠준 값을

로 치환 하면 다음과 같은 식을 얻을 수 있다.In Equation 6, both the right numerator and the denominator are

Divide by and divide the value

Substituting with gives the following equation.

또는

중 임의의 값으로 나누어도 식이 성립되나, 깊이 카메라와 프로젝터 간 위치 및 회전 변화가 크지 않을 때, 0이 생성될 수 있는 값으로 식을 나누면 캘리브레이션 오류가 크게 발생할 수 있다. 따라서, 본 명세서에서는 위치 변화가 없어도 1의 값을 갖는 안전한 값을 선택하였으며 이를 이용하여 [수학식 7]을 나누는 경우의 일 실시 예를 설명하였으나, 본 발명은 이에 한정되는 것은 아니다. [Equation 7]

or

Even if divided by any of the values, the expression is established, but when the position and rotation change between the depth camera and the projector is not large, dividing the expression by a value that can generate 0 can cause a large calibration error. Therefore, in this specification, a safe value having a value of 1 was selected even if there was no position change, and an example of dividing [Equation 7] using this was described, but the present invention is not limited thereto.

성분이 들어가 있지 않은 회전 변환 값은

이다. 따라서 이 중

으로 [수학식 7]을 나누면, 변환 계수 중

은 다음과 같이 계산된다. In the rotation conversion equation of Equation 8, the value of 0 is calculated when there is no rotation change.

The rotation conversion value without the component

to be. Therefore, of these

Dividing [Equation 7] by,

Is calculated as follows.

성분이 들어가 있지 않은 회전 변환 값으로 변환식을 나누면 후에 변환식을 복원하기 수월하다는 장점이 있는 바, 본 명세서에서는 캘리브레이션 식을

으로 나누어 깊이 카메라 좌표계의 3차원 좌표를 정규화된 프로젝터 투사 영상 좌표로 변환하는 식을 정리하였다. 여기서 계산의 편의를 위하여 일부 값을 [수학식 10]과 같이 치환할 수 있다. In other words,

Dividing the conversion equation by the rotation conversion value that does not contain components has the advantage of making it easier to restore the conversion equation afterwards.

The equation for converting the three-dimensional coordinates of the depth camera coordinate system to the normalized projection image coordinates of the camera is summarized. Here, for convenience of calculation, some values may be substituted as shown in [Equation 10].

When the position and rotation change equation between the depth camera and the projector is calculated using the substituted above value, it is as shown in Equation 11.

와 깊이 카메라 좌표계의 3차원 좌표 값

을 이용하여 연산 가능하다. 이 식은 한 점에 대한 식이므로 여러 점에 대한 선형식을

형태로 만들면 다음과 같이 나타낼 수 있다. [Equation 11] corresponds to the normalized projection image coordinates

And depth 3D coordinate values in the camera coordinate system

It can be calculated using. Since this is an expression for one point, the linear expression for multiple points

If we make it into shape, we can express it as follows.

In order to calculate the calibration transform coefficient, it is necessary to know the 3D coordinates of the depth camera corresponding to the image coordinates of a specific point, and the calibration unit 130 obtains the 3D coordinates of the depth camera corresponding to the image coordinates in the following way. can do. Here, 'image coordinates' refers to coordinates of a specific point in an image to be projected through a projector. For example, when projecting an image of 1920ⅹ1080 resolution to a projector, the coordinate value of one point in the image above is image coordinates. Can be understood.

As an example, the calibration unit 130 projects the chess board image to the projection surface through the projector as shown in FIG. 5, and depth of the black-white intersection of the chess board through infrared or RGB input values built into the depth camera. It can be detected in the image. The calibration unit 130 may obtain 3D coordinates of the depth camera coordinate system using the detected depth value of the intersection point.

형태로 계산하여 변환 계수

을 계산하면 다음과 같다. 여기서, 변환 계수

는 최초의 변환식에서의 변환 계수인

과

값을 편의상 치환한 것이므로, 다음과 같은 방법으로 캘리브레이션 변환 계수

과

를 산출할 수 있다. Since the value of the crossing point (image coordinates of a specific point) of the chess board image is a preset value, when the 3D coordinates of the depth camera coordinate system corresponding to the specific point are obtained, a calibration transform coefficient can be calculated. Assign each coordinate to a depth-projector coordinate system transformation matrix,

Transform coefficients

Is calculated as follows. Where the conversion factor

Is the transform coefficient in the first transform expression

and

Since the value is substituted for convenience, the calibration conversion factor is as follows.

and

Can be calculated.

를 계산하고, 다음 식을 이용하여

를 계산할 수 있다. For example, the calibration unit 130 uses [Equation 9]

Calculate and use the following equation

Can be calculated.

이므로

값을 복원 할 수 있으며, 따라서

에

값을 곱해주면 깊이-프로젝터 좌표계 변환 행렬을 도출할 수 있다. 캘리브레이션부(130)는 이와 같은 수학적 방법을 통해 깊이 카메라와 프로젝터 간의 상대적인 위치 관계를 정확하게 계산하는 캘리브레이션을 수행할 수 있다. here,

Because of

The value can be restored and thus

on

By multiplying the values, we can derive the depth-projector coordinate system transformation matrix. The calibration unit 130 may perform calibration that accurately calculates the relative positional relationship between the depth camera and the projector through this mathematical method.

The projection area prediction unit 150 converts the three-dimensional coordinates of the depth image obtained from the depth camera into normalized image coordinates using a calibration transform coefficient, and if the normalized image coordinates are included in the projected image size range, the corresponding coordinates Can be set as a projection prediction area.

를 이용하여 깊이 카메라 좌표계의 3차원 좌표로부터 정규화된 이미지 좌표를 계산할 수 있다. 깊이 카메라 좌표계의 3차원 좌표는 깊이 카메라가 획득한 깊이 영상을 이용해 계산할 수 있다. In more detail, the projection area prediction unit 150 transform coefficients obtained through calibration

Using can calculate normalized image coordinates from the three-dimensional coordinates of the depth camera coordinate system. The 3D coordinates of the depth camera coordinate system can be calculated using the depth image acquired by the depth camera.

The projection area prediction unit 150 may calculate (convert) the normalized image coordinates corresponding to each of the 3D coordinates of the depth image (3D coordinates of the depth camera coordinate system) using the following equation.

Next, the projection area prediction unit 150 may check whether the normalized image coordinates calculated using the above formula are included in the image size range using the following conditions.

When the normalized image coordinates satisfy both of the above conditions, the projection area prediction unit 150 may set the coordinates as the projection prediction area. The result of predicting the actual projection area using the depth camera and the projector in this way is as shown in FIG. 6.

Referring to FIG. 6, in (a), a projection area when an image is projected through a projector can be confirmed. The projection area in (a) can be confirmed with an RGB camera. In order to confirm the accuracy of the projection area prediction, the projection prediction area predicted through the depth image (b) in (c) is displayed in red color and added to the RGB image. Through (c), it can be confirmed that the projection area prediction unit 150 according to an embodiment of the present invention correctly sets the projection area (projection prediction area).

The correction area prediction unit 170 may predict the correction area of the user's viewpoint based on the user's position using the projection prediction area. The projection prediction area predicted using the 3D coordinates of the depth camera coordinate system is a projection surface area viewed by the depth camera. If the correction is performed using this, the shape may be different from the projection surface viewed by the user, so that the correction may not be properly performed.

Therefore, the correction area prediction unit 170 defines a user coordinate system based on the user's position-the user's position as the origin and the direction facing the projection plane as the z-axis, the right as the x-axis, and the upper as the y-axis. , A coordinate system of a projection prediction region may be transformed using a homogeneous transformation matrix (hereinafter referred to as a 'projector-user coordinate system transformation matrix') that converts a projector coordinate system and a user coordinate system.

)은 다음과 같이 나타낼 수 있다. Projector-user coordinate system transformation matrix (

) Can be represented as

라고 하면, 사용자 위치에서 바라보는 3차원 좌표

는 다음과 같이 정의될 수 있다. Depth-projector coordinate system transformation matrix

Speaking of, 3D coordinate viewed from the user's position

Can be defined as

Using the above formula, it is possible to convert the 3D coordinates of the projector coordinate system to the 3D coordinates of the user coordinate system. The correction area prediction unit 170 uses only the outer pixels of the projection prediction area of the depth image to predict the projection area and / or the correction area of the user's viewpoint before performing the correction, and / or the correction area of the user's viewpoint Can predict.

The method for predicting the correction region will be described in more detail as follows. The correction area prediction unit 170 may include an outer pixel detection unit 173, a first coordinate system conversion unit 175, a second coordinate system conversion unit 177, and a correction area detection unit 179 as illustrated in FIG. 3. have.

The outer pixel detector 173 may detect outer pixels of the projection prediction area of the depth image.

The first coordinate system converting unit 175 may calculate the depth camera coordinate system 3D coordinates of the outer pixel, and convert the depth camera coordinate system 3D coordinates of the outer pixel into the projector coordinate system 3D coordinate using the depth-projector coordinate system transformation matrix. have.

The second coordinate system conversion unit 177 may convert the projector coordinate system three-dimensional coordinates of the outer pixel into the user coordinate system three-dimensional coordinates using the projector-user coordinate system transformation matrix. As described above, the projector-user coordinate system transformation matrix means a matrix that converts 3D location information of the projector coordinate system to 3D location information of the user coordinate system based on the user location.

The correction area detector 179 may detect a 2D user viewpoint correction area by projecting 3D coordinates of a user coordinate system of an outer pixel according to a user viewpoint.

For reference, since the rectangular correction area of the user's viewpoint can be quickly found on the projection surface of the user's viewpoint expressed as a two-dimensional image, the correction area detection unit 179 assumes that the projection surface of the user's viewpoint is a two-dimensional image. The method for detecting the maximum rectangular area disclosed in the prior art ("The maximal rectangle problem", Vandevoorde et al, 1998) may be used to find a user viewpoint correction area on a two-dimensional image in consideration of a user position, but the present invention is not limited thereto. .

Since the detected user viewpoint correction area is an area on a 2D image, it is necessary to calculate what position the user viewpoint correction area corresponds to in the user coordinate system. However, since the present invention uses only outer pixels for fast calculation, information inside the region is unknown. Even if the information in the inner region exists, there is a limit to defining each of the projected points.

In order to solve this problem, the present invention redefines the maximum square area using the viewing angle and the transformed value, and through this, it is possible to calculate the 3D coordinates of the user coordinate system corresponding to the correction area by simply comparing the size. The viewing angle and the conversion value for redefining the correction area rectangle can be calculated using the following equation.

는 2차원 이미지 상의 보정 영역 직사각형의 시작점 및 크기를,

와

는 사용자 시점 보정 영역을 재정의한 시야각(FOV)값과 변환값(Transformation)을,

은 2차원 이미지를 생성할 때 사용된 투사면의 Z값을,

는 투사면의 크기,

은 이미지로 래스터화(Rasterization)된 크기,

,

를 나타낸다. In the above equation

Is the starting point and size of the correction area rectangle on the 2D image,

Wow

Is the field of view (FOV) and transformation values that redefine the user's viewpoint correction area,

Is the Z value of the projection plane used to create the 2D image,

Is the size of the projection surface,

The size of the image is rasterized,

,

Indicates.

는 각각 1이 될 것이다. 첫번째 투사 단계에서는 3차원 다각형의 모양(투사면의 모양)이 이미지로 표현되지 않고 꼭지점의 위치만 나타나므로, 이를 실제 다각형 이미지로 표현하기 위해서는 래스터화 단계가 수행되어야 한다. 즉,

은 래스터화가 수행되는 이미지의 크기를 의미하는 것으로 이해될 수 있다. To output a polygon or a point expressed in a 3D coordinate system to a monitor screen, there are usually two steps, firstly projecting the vertices of the polygon into a two-dimensional image, and secondly connecting the vertices projected into the two-dimensional image. This is a rasterization step of creating an image to fill the color and output it to the actual screen. For example, if you project a 3D polygon onto a 2D image with a width and height of 1ⅹ1,

Will be 1 each. In the first projection step, since the shape of the 3D polygon (the shape of the projection surface) is not expressed as an image, only the position of the vertex appears, a rasterization step must be performed to express it as an actual polygon image. In other words,

Can be understood to mean the size of the image on which rasterization is performed.

The obtained viewing angle value and the converted value of the correction region redefinition may be used when calculating the 3D coordinates and image coordinates belonging to the correction region when the image correction unit 179 performs correction.

The image compensator 190 may correct pixels belonging to a user viewpoint correction region in the projected image. In more detail, the image correction unit 190 calculates the depth value of each pixel constituting the projected image, and uses the projector's viewing angle to calculate the three-dimensional coordinates of the projector's coordinate system of the interpolated projector depth image (Interpolated Projector Depth Imgae). Can be calculated. Next, the image correcting unit 190 converts the user coordinate system to the projector coordinate system 3D coordinates of the interpolation projection depth image, determines whether the user coordinate system 3D coordinates belong to the user's viewpoint correction area, and the user of the interpolation projection depth image. If the coordinate system 3D coordinates belong to a user viewpoint correction area, the corresponding pixel may be corrected.

In general, the projection image resolution of the projector is much higher than the depth image resolution of the depth camera. Therefore, it is possible to convert the three-dimensional coordinates of the depth image obtained from the depth camera to the projection image coordinates on the projector coordinate system using the calibration, but it is not possible to convert the coordinates of the projected image to the depth camera coordinate system or the depth image coordinates. Do.

However, in order to create a warping table for correction, it is necessary to know all three-dimensional coordinates of the projected image projected by the projector. In order to know the 3D coordinates of the projected image, it is necessary to know the depth value of each pixel constituting the projected image. As described above, there is a problem that the coordinates of the depth image corresponding to the coordinate values of all pixels of the projected image are unknown.

To solve this problem, the image correction unit 190 according to an embodiment of the present invention can quickly calculate all three-dimensional coordinates of a projected image using a recursive Gaussian weight interpolation method. In addition, the image correction unit may quickly generate a warping table using a viewing angle value and a conversion value defined as a user position and a user viewpoint correction area.

The overall process of the real-time projection correction method of the user's viewpoint performed by the image correction unit 190 of the present invention is as shown in FIG. 7.

On the other hand, as described above, according to the calibration performed by the calibration unit 130 of the present invention, it is possible to predict the projection area from the depth image, but calculating all three-dimensional coordinates of the projection image projected by the projector from the depth image impossible. Instead, since the resolution of the projected image is much higher than that of the depth image, a 3D coordinate value of pixels whose depth value is unknown among 3D coordinates of the projected image can be calculated using the projected image.

When the projection image and the depth image are matched based on the coordinate system of the projected image, a part of the depth information is unknown due to a difference in resolution, and thus, in this specification, a depth image in which all of the depth information of the entire pixel is not filled. Is referred to as a sparse projector depth image. 8 shows an example of generating a rare projection depth image.

(a) shows the projection area detected by the low resolution 512x424. In (a), since only a part of the low resolution depth image is detected as the projection surface, all depth information of the projection image cannot be known.

(b) is a diagram showing how much information is emptied when pixels of the projection area detected in the depth image are matched to the actual projection image.

(c) is a diagram when depth information of the projection area detected from the depth image is inserted into the projection image. Black indicates no depth information, and gray indicates depth information obtained from a depth image.

Conventionally, a high-dimensional surface fitting method was used to calculate depth information of such a rare projection depth image. In the case of the conventional surface fitting method, since it is premised that the projection surface does not change, the warping table is generated once in a manner that requires a very large amount of computation, and correction is performed only once for the first time.

However, the present invention aims to provide a correction method capable of responding to a continuously changing projection surface, and does not use a surface fitting method having a slow processing speed to respond to a projection surface change, and can significantly reduce processing time. By using the recursive Gaussian Weight Interpolation, the depth value of each pixel constituting the projected image can be calculated to obtain 3D coordinates corresponding to each pixel.

When the image corrector 190 acquires three-dimensional coordinates of the projected image using the recursive Gaussian weight interpolation method, the depth value of the pixel having no depth value in the rare projected depth image corresponding to the projected image is calculated and calculated. A method of generating an interpolated projection depth image by reflecting the depth value is as shown in FIG. 9.

Referring to FIG. 9, the image correction unit 190 may apply a Gaussian mask (kernel) as illustrated in FIG. 9A to each pixel of the rare projection depth image. Before applying the Gaussian mask to each pixel, the image correction unit 190 may determine whether the first pixel to which the mask is to be applied is an effective pixel. Here, the effective pixel means a pixel having an effective depth value.

When the first pixel is not an effective pixel-the depth value of the first pixel does not exist-and an effective pixel exists within a gaussian mask application range for the first pixel, the image correction unit 190 may have a Gaussian mask in the first pixel The depth value of the first pixel may be calculated by applying.

The image correction unit 190 calculates a weighted average using a depth value of the first effective pixel and a weight of a Gaussian mask corresponding to the first effective pixel for all the first effective pixels existing in the gaussian mask application range, and a weighted average A gaussian mask may be applied to the first pixel by setting the depth value of the first pixel.

로 설정될 수 있다. For example, consider an embodiment shown in FIG. 9. In FIG. 9, the image correction unit 190 intends to apply a 5ⅹ5 Gaussian mask 5 to the first hatched pixel 1 in the center in (b). There are two effective pixels 3 within the gaussian mask coverage, and the depth values of each effective pixel are 100 and 108, respectively. The image compensator 190 may calculate a weighted average using a depth value of an effective pixel and a weight of a Gaussian mask corresponding to the effective pixel for all effective pixels, and set the weighted average as the depth value of the first pixel have. Therefore, in the example of FIG. 9, the depth value of the first pixel is

Can be set to

The image corrector 190 does not apply a Gaussian mask to the first pixel if the first pixel is an effective pixel or there is no effective pixel within the gaussian mask application range for the first pixel, and repeats the above process for other pixels. can do.

The interpolation projection depth image generation unit 193 of the image correction unit 190 projects the image until the depth values of all pixels of the rare projection depth image are calculated, that is, until all the pixels of the rare projection depth image become effective pixels. By repeating the above process for each pixel of the depth image, an interpolated projection depth image can be generated. Interpolated Projector Depth Image is a depth image filled with depth values, and means a depth image that can be converted into a projector coordinate system.

Using the projector's viewing angle, the projector coordinate system 3D coordinates of the interpolated projection depth image can be calculated, and the calculated projector coordinate system 3D coordinates can be used to generate a warping table for correction. In order to generate the warping table, since correction must be performed in the projection area at the user's point of view, the correction unit 195 of the image correction unit 190 converts the interpolated projection depth image into three-dimensional coordinates of the projector coordinate system, and again Can be converted to user coordinate system. Since such a process is performed for each pixel unit of an interpolation projection depth image, it is possible to determine whether each pixel belongs to a user viewpoint correction area by calculating coordinate values of a user coordinate system for each pixel.

,

,

로 변환할 수 있다. The determination as to whether each pixel belongs to the user viewpoint correction area may be performed as follows. For example, the correction unit 195 converts the pixel corresponding to the (x, y) position of the interpolation projection depth image into three-dimensional coordinates of the projector coordinate system, and again, the three-dimensional coordinates of the user coordinate system.

,

,

Can be converted to

와

를 이용하여 사용자 좌표계의 3차원 좌표가 사용자 시점 보정 영역에 해당되는지를 계산할 수 있는데 이때 다음과 같은 식을 이용할 수 있다. The correction unit 195 defines a user viewpoint correction area

Wow

It is possible to calculate whether the 3D coordinates of the user coordinate system correspond to the user's viewpoint correction area using the following equation.

은 2차원의 사용자 시점 보정 영역을 검출할 때 사용된 프로젝터 평면(투사면)의 z값이며,

,

는 보간 투사 깊이 영상의 x, y 위치에서의 비교 시야각이다. In Equation 19

Is the z value of the projector plane (projection plane) used to detect the 2D user viewpoint correction area,

,

Is a comparative viewing angle at the x and y positions of the interpolated projection depth image.

,

으로, 각각 픽셀의 x축 방향의 투사 각도와 픽셀의 y축 방향의 투사 각도를 의미하는 것으로 이해될 수 있다. For example, projection is made based on the user's viewpoint to make the distorted image of the projector plane appear to be square from the user's position. )to be. At this time, the value used to calculate whether each pixel deviates from the viewing angle or not is included.

,

As a result, it can be understood that each means a projection angle in the x-axis direction of the pixel and a projection angle in the y-axis direction of the pixel.

,

를 계산할 수 있다. If the interpolation projection depth image of each pixel position x, y and the depth value of the corresponding position are known, it is possible to check whether the pixel is out of the viewing angle, and the image correction unit 190 checks this.

,

Can be calculated.

,

을 사용자 시점 투사 보정 영역의 시야각

과 비교하여 다음의 조건을 만족하는 지 확인할 수 있다. Compensation unit 195 is a calculated comparison viewing angle value

,

The viewing angle of the user's perspective projection correction area

It can be confirmed that the following conditions are satisfied by comparing with.

If the above expression is satisfied, the (x, y) position of the interpolated projection depth image corresponds to the position shown as the correction area from the user's point of view. That is, the above pixel can be regarded as belonging to the user perspective projection correction area.

The equation for calculating which pixel should be replaced with the pixel corresponding to the user perspective projection correction area is as follows.

는 보정을 수행할 프로젝터 투사 영상의 크기이며,

는 보정 영역 (x, y)에 보여질 투사 영상의 픽셀의 좌표를 나타낸다. 영상 보정부(190)는 수학식 22에 의해 생성된 맵핑 테이블을 기준으로 보정 영역의 픽셀을 다른 픽셀로 대체함으로써 보정을 수행할 수 있다. In Equation 21

Is the size of the projector projection image to be corrected,

Denotes the coordinates of pixels of the projected image to be displayed in the correction area (x, y). The image correction unit 190 may perform correction by replacing a pixel in the correction region with another pixel based on the mapping table generated by Equation (22).

는 워핑된 영상(대체 픽셀을 맵핑한 영상)을,

는 프로젝터의 투사 원본 영상,

는

에 해당되는

값을 맵핑 해주는 함수다. 이와 같은 과정을 통해 영상을 보정하면, 역방향 맵핑으로 영상을 워핑할 수 있기 때문에 빠르게 보정 영상을 만들 수 있다.From here

Is a warped image (an image mapped with a replacement pixel),

Projection original image of the projector,

The

Applicable to

It is a function that maps values. When the image is corrected through the above-described process, since the image can be warped by reverse mapping, a corrected image can be quickly generated.

Hereinafter, a real-time projector image correction method of an electronic device according to an embodiment of the present invention will be described with reference to FIGS. 10 to 13. An electronic device for performing the real-time projector image correction method of the present invention is hardware including a computer processor, and is embedded in a projector or a depth camera to perform the method, and the method can be performed as a device separate from the projector or depth camera. Can be done.

Referring to FIG. 10, an electronic device according to an embodiment of the present invention uses a 3D coordinate of a specific point obtained using an image coordinate and a depth camera of a specific point included in an image projected through a projector. A calibration transform coefficient indicating a correlation between the 3D position information of the camera coordinate system and the 3D position information of the projector coordinate system may be calculated (S100).

If the electronic device calculates a calibration transform coefficient, the electronic device may convert the three-dimensional coordinates of the depth image obtained from the depth camera into normalized image coordinates using the calibration transform coefficient (S200). Then, when the normalized image coordinates are included in the image size range, the coordinates may be set as a projection prediction area (S300).

When the projection prediction region is set, the electronic device may detect an outer pixel of the projection prediction region (S400), and calculate a depth camera coordinate system 3D coordinate of the outer pixel. In addition, the depth camera coordinate system 3D coordinates of the outer pixel may be converted into a projector coordinate system 3D coordinate using a depth-projector coordinate system transformation matrix derived using a calibration transform coefficient (S500).

Next, the electronic device converts the 3D coordinates of the projector coordinate system of the outer pixel to the user coordinate system by using a projector-user coordinate system transformation matrix that converts the 3D location information of the projector coordinate system into 3D location information of the user coordinate system based on the user's location. It can be converted into three-dimensional coordinates (S600), and the three-dimensional coordinates of the user's viewpoint can be detected by projecting the three-dimensional coordinates of the user coordinate system of the outer pixel according to the user's location (S700).

The electronic device detecting the correction area of the user's viewpoint may correct the projector image in real time by correcting pixels belonging to the user viewpoint correction area in the projected image projected by the projector (S800).

Referring to FIG. 11, in more detail, step 800 of real-time correcting a projector image, the electronic device calculates a depth value of a pixel in which a depth value does not exist in the rare projection depth image corresponding to the projected image, and calculates the calculated value. An interpolation projection depth image is generated by reflecting the depth value (S810), and the projector coordinate system 3D coordinates of the interpolation projection depth image may be calculated using a field of view of the projector (S820).

Referring to FIG. 12 showing an example of step 810, the electronic device may first determine whether one pixel of the rare projection depth image corresponding to the projected image is an effective pixel (S811). If the result of the determination in step 811 is that the first pixel is not an effective pixel, the electronic device may additionally determine whether an effective pixel exists within a gaussian mask application range for the first pixel (S813). If an effective pixel exists within a gaussian mask application range, the electronic device may calculate a depth value of the first pixel by applying a gaussian mask to the first pixel. At this time, steps 811 and 813 are not dependent on the order shown in FIG. 12.

If, as a result of the determination in step 811, the target pixel to which the Gaussian mask is to be applied is an effective pixel or the target pixel is not an effective pixel, if there is no effective pixel within the gaussian mask application range for the target pixel, the electronic device generates a rare projection depth image Steps 811 to 812 may be performed with respect to other pixels in step S814.

In this way, the electronic device repeatedly performs steps 811 to 815 until all of the pixels of the rare projection depth image become effective pixels, thereby calculating depth values of all pixels that do not have a depth value and depth values for each pixel. You can fill in

The method of applying the Gaussian mask has been described above with reference to FIG. 9, and thus redundant description is omitted.

The depth image in which all the depth values are filled is called an interpolation projection depth image, and the electronic device converts the projector coordinate system 3D coordinates of the interpolation projection depth image into a user coordinate system 3D coordinates (S830), and the user coordinate system 3 of the interpolation projection depth image. If the dimensional coordinates belong to the user viewpoint correction area, the corresponding pixel may be corrected (S840).

A method of correcting a pixel belonging to a user viewpoint correction area is illustrated in more detail in FIG. 13. The electronic device determines whether the comparison viewing angle at the corresponding position is less than or equal to the viewing angle of the user's viewpoint projection correction area for the pixel located at (x, y) of the interpolation projection depth image (S841), and the condition in step 841 If is satisfied, the (x, y) position can be designated as the correction area (S843).

The electronic device may warp (correct) pixels of the correction region by mapping pixels (x _c , y _c ) of the projection image to (x, y) positions of the correction region, and pixel positions of the projection image mapped to the correction region May be calculated by Equation 21 described above.

Some embodiments omitted in the present specification are equally applicable to the same subject. In addition, the above-described present invention is to those skilled in the art to which the present invention pertains, various substitutions, modifications and changes are possible within the scope not departing from the technical spirit of the present invention. It is not limited by the drawings.

100: real-time projector image correction device
130: calibration unit
150: projection area prediction unit
170: correction area prediction unit
190: image correction unit

Claims (9)

Correlation between 3D position information in the depth camera coordinate system and 3D position information in the projector coordinate system using 3D coordinates of the specific point obtained using a depth camera and image coordinates of a specific point included in an image projected through a projector. Calculating a calibration transform coefficient representing the relationship;
Converting three-dimensional coordinates of the depth image obtained from the depth camera into normalized image coordinates using the calibration transform coefficients;
If the normalized image coordinates are included in the image size range, setting the coordinates as a projection prediction area;
Detecting an outer pixel of the projection prediction area;
Calculating three-dimensional coordinates of the depth camera coordinate system of the outer pixel;
Converting a depth camera coordinate system 3D coordinate of the outer pixel into a projector coordinate system 3D coordinate using a depth-projector coordinate system transformation matrix derived using the calibration transform coefficients;
Convert the 3D coordinates of the projector coordinate system of the outer pixel to the 3D coordinates of the user coordinate system by using the projector-user coordinate system transformation matrix that converts the 3D location information of the projector coordinate system into 3D location information of the user coordinate system based on the user's location. To do;
Projecting a 3D coordinate of the user coordinate system of the outer pixel according to a user position to detect a 2D user viewpoint correction area;
The interpolation projection depth image generated according to the depth value of the rare projection depth image corresponding to the projection image and the user's viewpoint correction region are used to designate a correction region, and then the pixels of the projection image are mapped to the location of the correction region to And correcting a pixel belonging to a user viewpoint correction area.
According to claim 1,
The step of correcting a pixel belonging to the user viewpoint correction area is
Calculating a depth value of a pixel having no depth value in the rare projection depth image corresponding to the projected image, and generating an interpolated projection depth image by reflecting the calculated depth value;
Calculating a 3D coordinate of a projector coordinate system of an interpolated projection depth image using a field of view of the projector;
Converting a 3D coordinate of a projector coordinate system of the interpolation projection depth image into a 3D coordinate of a user coordinate system;
And if a user coordinate system 3D coordinate of the interpolation projection depth image belongs to the user viewpoint correction area, correcting a corresponding pixel.
According to claim 2,
The step of generating the interpolation projection depth image is
A step of determining whether a first pixel of the sparse projection depth image is an effective pixel;
When the first pixel is not an effective pixel and an effective pixel exists within a gaussian mask application range for the first pixel, step b is calculated by applying a preset Gaussian mask to the first pixel to calculate the depth value of the first pixel ;
A step c of not applying the Gaussian mask to the first pixel if the first pixel is an effective pixel or there is no effective pixel within a gaussian mask application range for the first pixel;
And repeating steps a to c for each pixel of the sparse projection depth image until all of the pixels of the sparse projection depth image become effective pixels,
Applying a Gaussian mask to the first pixel
Calculating a weighted average using a depth value of the first effective pixel and a weight of a Gaussian mask corresponding to the first effective pixel for all the first effective pixels existing in the gaussian mask application range;
And setting the weighted average to a depth value of the first pixel.

The method of claim 2
If the user coordinate system 3D coordinates of the interpolation projection depth image belong to the user viewpoint correction area, correcting the corresponding pixel
If the comparison viewing angle at the (x, y) position of the interpolation projection depth image is less than or equal to the viewing angle of the user viewpoint correction area, designating the (x, y) position as a compensation area;
And mapping pixels (x _c , y _c ) of the projected image to (x, y) positions of the correction area.
According to claim 4,
The pixel (x _c , y _c ) of the projected image is

(

= Height of projected image,

= X-axis comparison viewing angle at (x, y) position of interpolation projection depth image,

Y-axis comparison viewing angle at (x, y) position of interpolation projection depth image,

X-axis viewing angle of the user's viewpoint correction area,

Y-axis viewing angle of the user's viewpoint correction area)
Real-time projector image correction method, characterized in that calculated by.
Correlation between 3D position information of the depth camera coordinate system and 3D position information of the projector coordinate system using 3D coordinates of the specific point obtained using the depth camera and coordinates of a specific point included in the projected image projected through the projector. A calibration unit for calculating a calibration transform coefficient indicating the relationship;
The 3D coordinates of the depth image obtained from the depth camera are converted into normalized image coordinates using the calibration transform coefficients, and when the normalized image coordinates are included in the projected image size range, the corresponding coordinates are set as a projection prediction area. Projection area prediction unit;
A correction region predicting unit predicting a correction region of a user's viewpoint based on a user's position using the projection prediction region;
The interpolation projection depth image generated according to the depth value of the rare projection depth image corresponding to the projection image and the user's viewpoint correction region are used to designate a correction region, and then the pixels of the projection image are mapped to the location of the correction region to It includes an image correction unit for correcting the pixels belonging to the user viewpoint correction area,
The correction region prediction unit
An outer pixel detecting unit detecting an outer pixel of the projection prediction area;
Calculate the depth camera coordinate system 3D coordinates of the outer pixel, and convert the depth camera coordinate system 3D coordinates of the outer pixel to the projector coordinate system 3D coordinates using the depth-projector coordinate system transformation matrix derived using the calibration transform coefficients. A first coordinate system conversion unit;
Convert the 3D coordinates of the projector coordinate system of the outer pixel to the 3D coordinates of the user coordinate system by using the projector-user coordinate system conversion matrix that converts the 3D position information of the projector coordinate system into 3D position information of the user coordinate system based on the user's location. A second coordinate system conversion unit;
A real-time projector image correction device comprising a correction area detection unit that detects a 2D user viewpoint correction area by projecting a 3D coordinate of a user coordinate system of the outer pixel according to a user position.
The method of claim 6,
The image correction unit
An interpolation projection depth image generation unit for calculating a depth value of a pixel having no depth value in the rare projection depth image corresponding to the projection image and generating an interpolation projection depth image by reflecting the calculated depth value;
The projector coordinate system 3D coordinates of the interpolation projection depth image are calculated using the field of view of the projector, the projector coordinate system 3D coordinates of the interpolation projection depth image are converted to the user coordinate system 3D coordinates, and the interpolation projection is performed. A real-time projector image correction apparatus including a correction unit that corrects a corresponding pixel when a user coordinate system 3D coordinate of a depth image belongs to the user viewpoint correction region.
The method of claim 7,
The image correction unit
When the user coordinate system 3D coordinate of the interpolation projection depth image belongs to the user perspective projection correction area, in correcting a corresponding pixel,
If the comparison viewing angle at the (x, y) position of the interpolation projection depth image is less than or equal to the viewing angle of the user's viewpoint projection correction area, the (x, y) position is designated as a correction area, and (x , y) Mapping a pixel (x _c , y _c ) of the projected image to a location.
The method of claim 8,
The pixel (x _c , y _c ) of the projected image is

(

= Height of projected image,

= X-axis comparison viewing angle at (x, y) position of interpolation projection depth image,

Y-axis comparison viewing angle at (x, y) position of interpolation projection depth image,

X-axis viewing angle of the user's point of view projection correction area,

Y-axis viewing angle of the user perspective projection correction area)
Real-time projector image correction device, characterized in that calculated by.

Images