KR102460659B1 - Method and Apparatus FOR obtaining DEPTH IMAGE USING TOf(Time-of-flight) sensor - Google Patents

Abstract

A method and apparatus for generating a depth image using a Time-of-Flight (ToF) sensor may generate a depth image using a plurality of images in which a motion blur region due to a movement of a subject is corrected. In this case, since the motion blur region is corrected after compensating for the initial phase difference of the irradiated light, the accuracy of the depth image may be improved.

Description

Method and apparatus for obtaining a depth image using a ToF sensor {Method and Apparatus FOR obtaining DEPTH IMAGE USING TOf (Time-of-flight) sensor}

The disclosed embodiments relate to a method and apparatus for acquiring a depth image using a ToF sensor.

Research on a 3D camera, a motion sensor, and a laser radar (LADAR) capable of acquiring distance information from an object is on the rise in recent years. In particular, the importance of 3D content is being emphasized with the development and increase in demand for 3D display devices capable of displaying images with a sense of depth. Accordingly, various depth image acquisition devices that allow general users to directly produce 3D content are being studied.

The depth information on the distance between the surfaces of the subject and the 3D image acquisition device is obtained by using the stereo vision method using two cameras or the triangulation method using structured light and a camera. can be obtained by However, in this method, it may be difficult to obtain precise depth information because the accuracy of the depth information rapidly decreases as the distance of the subject increases, and it is dependent on the surface state of the subject.

Meanwhile, a time-of-flight (ToF) method has been recently introduced in a depth image acquisition device. ToF technology is a method of measuring the light flight time until the light reflected from the subject is received by the light receiving unit after irradiating the illumination light to the subject. According to the ToF technology, light of a specific wavelength (for example, near infrared rays of 850 nm) is projected onto a subject using an illumination optical system including a light emitting diode (LED) or a laser diode (LD), and the light of the same wavelength reflected from the subject is emitted. After light is received by the light receiving unit, a series of processing steps are performed to extract depth information, such as modulating the light received by the optical shutter. Various ToF technologies are being developed according to this series of light processing processes.

An object of the present invention is to provide a method and an apparatus for acquiring a depth image in which motion blur caused by movement of a subject is corrected by using a ToF sensor. The technical problems to be achieved by the present embodiment are not limited to the technical problems as described above, and other technical problems may be inferred from the following embodiments.

According to one aspect, a method of generating a depth image using a Time-of-Flight (ToF) sensor is a method of generating a motion blur region generated by a movement of a subject in a plurality of original images obtained through irradiation lights having different phases. detecting; compensating for a difference in intensity according to initial phases of reflected lights reflected from the subject with respect to the original images; correcting the motion blur region in the compensation images generated by the compensation; and generating a depth image based on the corrected images generated by the correction.

The compensating for the intensity difference may include: selecting a reference image from among the plurality of original images; and adjusting the cumulative distribution function regarding the intensity of the reflected light corresponding to the remaining original images to approximate the cumulative distribution function with respect to the intensity of the reflected light corresponding to the reference image.

Also, in the step of correcting the motion blur region, the motion blur region may be corrected by using a patchMatch technique in the compensated images.

Also, the irradiated lights having different phases may be sequentially irradiated to the subject, and the plurality of original images may be obtained by modulating the reflected lights through an optical shutter.

In addition, initial phases of the irradiated lights are 0 degrees, 180 degrees, 90 degrees and 270 degrees, and the plurality of original images include a first original image corresponding to 0 degrees and a second original image corresponding to 180 degrees. , a third original image corresponding to the 90 degrees and a fourth original image corresponding to the 270 degrees.

In addition, the generating of the depth image may include: based on the phase information and modulation frequency obtained by using the reflected light corresponding to the first original image and the reflected light corresponding to the second corrected image to the fourth corrected image, the depth It may include; calculating the information.

In addition, the detecting of the motion blur region includes: a first group indicating a sum of intensities of the reflected lights corresponding to the first original image and the second original image; and the third original image and the fourth original image. When the difference between the second group, which means the sum of the intensity of the reflected lights corresponding to the image, is equal to or greater than a threshold value, the motion blur region may be detected from the plurality of original images.

In addition, the detecting of the motion blur region includes selecting two original images from among the plurality of original images, and the intensity of the reflected light corresponding to the first region included in the two original images is When it is determined that the difference is due to the movement of the subject, the first area may be detected as the motion blur area.

Also, the motion blur region may be determined to be the same region in the plurality of original images.

According to another aspect, a recording medium recording a program to be executed in a computer records a method for generating a depth image using a Time-of-Flight (ToF) sensor as a program for executing in the computer.

According to another aspect, an apparatus for generating a depth image using a ToF sensor includes a sensing unit configured to detect a motion blur region generated by a movement of a subject in a plurality of original images obtained through irradiation lights having different phases. ; Compensating unit for compensating for a difference in intensity of the reflected lights according to an initial phase difference of the reflected lights reflected from the subject with respect to the original images, and correcting the motion blur region in the plurality of compensation images generated by the compensation ; and a depth image processor configured to generate a depth image based on the corrected images generated by the correction.

In addition, the compensator selects a reference image from among the plurality of original images, and has a cumulative distribution function with respect to the intensity of reflected light corresponding to the remaining original images to approximate the cumulative distribution function for the intensity of reflected light corresponding to the reference image. can be adjusted.

Also, the compensator may correct the motion blur region in the compensation images by using a PatchMatch technique.

In addition, an apparatus for generating a depth image using a ToF sensor may include: a light source that generates irradiated lights having different phases and sequentially irradiates them toward the subject; an optical shutter modulating a waveform of the reflected lights by changing the transmittance of the reflected lights reflected from the subject; and an imaging unit generating a plurality of original images using the reflected light modulated by the optical shutter.

In addition, the apparatus for generating a depth image using the ToF sensor includes: a driver applying driving voltages to the light source and the optical shutter; and a control unit for controlling operations of the driving unit and the imaging unit.

In addition, the controller controls the driving unit to sequentially irradiate the irradiated lights having the initial phase of 0 degrees, 180 degrees, 90 degrees, and 270 degrees, respectively, and the plurality of original images are first images corresponding to 0 degrees. It may include an original image, a second original image corresponding to the 180 degrees, a third original image corresponding to the 90 degrees, and a fourth original image corresponding to the 270 degrees.

In addition, the depth image processing unit may calculate depth information based on the phase information and modulation frequency obtained by using the reflected light corresponding to the first original image and the reflected light corresponding to the second corrected image to the fourth corrected image. have.

In addition, the sensing unit includes a first group indicating the sum of intensities of the reflected lights corresponding to the first original image and the second original image, and the third original image and the reflected lights corresponding to the fourth original image. When the difference between the second group, which means the sum of intensities, is equal to or greater than a threshold, the motion blur region may be detected from the plurality of original images.

In addition, the sensing unit selects two original images from among the plurality of original images, and it is determined that the difference in intensity of the reflected light corresponding to the first region included in the two original images is due to the movement of the subject. In this case, the first area may be detected as the motion blur area.

Also, the motion blur region may be determined to be the same region in the plurality of original images.

As described above, since a depth image may be generated using a plurality of images in which a motion blur region due to a movement of a subject is corrected, an accurate depth image may be obtained.

1 is a diagram schematically illustrating an apparatus for acquiring a depth image according to an exemplary embodiment.
2 is a diagram for explaining an operation method of the apparatus for acquiring a depth image according to an exemplary embodiment.
3 is a diagram illustrating a plurality of images obtained according to an exemplary embodiment.
4 is a diagram for explaining a method of correcting a motion blur region, according to an embodiment.
5 is a diagram illustrating a motion blur area detected in a plurality of images, according to an exemplary embodiment.
6A is a histogram comparing intensities of a CIS output signal corresponding to a reference image and a CIS output signal corresponding to other images, according to an exemplary embodiment.
6B is a graph illustrating a cumulative distribution function with respect to intensities of a CIS output signal corresponding to a reference image and a CIS output signal corresponding to other images, according to an exemplary embodiment.
7 is a diagram illustrating an image in which a motion blur region is corrected based on a patch match technique, according to an exemplary embodiment.
8 is a diagram illustrating a depth image obtained by using an original depth image and an image in which a motion blur region is corrected, according to an exemplary embodiment.
9 is a diagram illustrating an algorithm for generating a depth image after removing motion blur, according to an exemplary embodiment.
10 is a block diagram illustrating an apparatus for acquiring a depth image, according to an embodiment.
11 is a block diagram illustrating an apparatus for acquiring a depth image, according to another exemplary embodiment.
12 is a flowchart illustrating a method of acquiring a depth image using a ToF sensor, according to an exemplary embodiment.
13A to 13B are detailed flowcharts specifically illustrating a method of acquiring a depth image using a ToF sensor, according to an exemplary embodiment.

The terms used in the present embodiments are selected as currently widely used general terms as possible while considering the functions in the present embodiments, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. can In addition, there are also arbitrarily selected terms in a specific case, and in this case, the meaning will be described in detail in the description of the embodiment. Therefore, the terms used in the present embodiments should be defined based on the meaning of the term and the overall contents of the present embodiments, rather than the simple name of the term.

In the descriptions of the embodiments, when it is said that a part is connected to another part, this includes not only a case in which it is directly connected, but also a case in which it is electrically connected with another component interposed therebetween. . Also, when it is said that a part includes a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated. In addition, the terms "... unit" and "... module" described in the embodiments mean a unit that processes at least one function or operation, which is implemented as hardware or software, or is a combination of hardware and software. can be implemented.

Terms such as “consisting of” or “comprising” used in the present embodiments should not be construed as necessarily including all of the various components or various steps described in the specification, and some components or It should be construed that some steps may not be included, or may further include additional components or steps.

Also, terms including an ordinal number such as “first” or “second” used in the present embodiments may be used to describe various objects, but the objects should not be limited by the terms. These terms are used only for the purpose of distinguishing one object from another.

The description of the following embodiments should not be construed as limiting the scope of rights, and what can be easily inferred by those skilled in the art should be construed as belonging to the scope of the embodiments. Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings.

1 is a diagram schematically illustrating an apparatus for obtaining a depth image according to an exemplary embodiment.

Referring to FIG. 1 , the depth image obtaining apparatus 100 includes a light source 120 irradiating light to a subject 10 and a reflected light provided in a path through which reflected light reflected from the subject 10 travels by changing the transmittance of the reflected light. an optical shutter 130 that modulates a waveform of can In addition, the depth image acquisition apparatus 100 receives the reflected light passing through the optical shutter 130 , extracts a phase difference between the light irradiated to the subject 10 from the light source 120 and the reflected light, and based on the phase difference to obtain the depth information of the subject 10 may include an imaging unit 160 .

The light source 120 may irradiate light toward the subject 10 . The light source 120 may irradiate light having a wavelength in a near-infrared (NIR) region of 800 nm to 1000 nm, but the type of light that the light source 120 may radiate is not limited thereto. In addition, by irradiating light of a wavelength, the light irradiated from the light source 120 may not be visible to the human eye. The light source 120 may include a light emitting diode (LED) or a laser diode (LD). The light source 120 may be connected to the driving unit 140 .

The driving unit 140 may apply the driving voltage V _d1 to the light source 120 to operate the light source 120 . The intensity and wavelength of the light irradiated from the light source 120 may vary according to the magnitude of the driving voltage V _d1 applied by the driving unit 140 . For example, when the magnitude of the driving voltage V _d1 applied to the light source 120 increases, the wavelength and output of the light irradiated from the light source 120 may increase.

In addition, the light irradiated from the light source 120 may be a pulse wave having a predetermined shape. For example, the light irradiated from the light source 120 may have a waveform such as a sine wave, a ramp wave, or a square wave. In addition, the light irradiated from the light source 120 may be reflected from the subject 10 . Therefore, depending on the distance between the subject 10 and the light source 120 , the phase difference may vary between the phase of the light irradiated from the light source 120 and the light reflected from the subject 10 and incident on the optical shutter 130 . .

The optical shutter 130 may modulate a waveform of the reflected light by varying the degree of transmission of the reflected light reflected from the subject 10 . In this case, the shape in which the reflected light is modulated by the optical shutter 130 may vary depending on the phase of the reflected light incident on the optical shutter 130 . That is, the phase of the reflected light incident on the optical shutter 130 varies according to the distance between the subject 10 and the light source 120 , and thus the waveform of the reflected light modulated by the optical shutter 130 may vary. The transmittance of the optical shutter 130 may vary depending on the driving voltage V _d2 applied by the driving unit 140 to the optical shutter 130 .

The imaging unit 160 may acquire depth information of the subject 10 by receiving the reflected light passing through the optical shutter 130 . The imaging unit 160 may include an imaging device 162 and a calculation module 164 for calculating depth information. The imaging device 162 may detect the reflected light modulated by the optical shutter 130 . If it is desired to measure only the distance to any one point of the subject 10 , the imaging device 162 may include, for example, a photodiode or a single optical sensor. However, when it is desired to simultaneously measure the distances to a plurality of points P1, P2, P3, P4, and P5 on the subject 10, the imaging device 162 may be a two-dimensional or one-dimensional method of a plurality of photodiodes or a plurality of photosensors. It may also have a dimensional array. For example, the imaging device 162 may be a CCD (Charge Coupled Device) image sensor or a CIS (CMOS Image Sensor) sensor having a two-dimensional array, but is not limited thereto. The imaging device 162 may measure the intensity of the reflected light passing through the optical shutter 130 . The intensity of light measured by the imaging device 162 may depend on a waveform in which the light is modulated.

The calculation module 164 may calculate depth information of the subject 10 from image information measured by the imaging device 162 . The calculation module 164 may calculate a phase difference between the reflected light and the light emitted from the light source 120 by using the intensity of the reflected light measured by the imaging device 162 . Also, the calculation module 164 may obtain depth information of the subject 10 by calculating the distance between the light source 120 and the subject 10 using the phase difference.

The controller 110 may control the driving unit 140 to change the transmittance of the optical shutter 130 . Specifically, the driving unit 140 may control the driving voltage V _d1 applied to the light source 120 and the driving voltage V _d2 applied to the optical shutter 130 . For example, when the distance between the light source 120 and the subject 10 is long, the light source 120 may emit light with high output. However, when the distance between the light source 120 and the subject 10 is close, even if the light source 120 irradiates light with a low output, depth information of the subject 10 may be obtained. Therefore, when the distance between the light source 120 and the subject 10 is close, reducing the light output amount of the light source 120 may be good in terms of power consumption efficiency. In addition, by adjusting the light output amount of the light source 120 , it is possible to prevent light saturation from occurring in the imaging device 162 of the imaging unit 160 .

2 is a diagram for explaining an operation method of the apparatus for acquiring a depth image according to an exemplary embodiment.

In FIG. 2 (a), the graph is a graph showing the change in intensity according to time of the light irradiated from the light source 120 . In addition, the graph (b) of FIG. 2 is a graph showing the change in intensity over time of the reflected light incident on the optical shutter 130 . In addition, the graph (c) of FIG. 2 is a graph showing the degree to which the transmittance of the optical shutter 130 changes with time.

Referring to FIG. 2 , the light source 120 may sequentially project a plurality of irradiation lights 210 onto the subject 10 . The plurality of irradiation lights 210 may be irradiated to the subject 10 with a predetermined idle time. In addition, the plurality of irradiation lights 210 may have different phases from each other and may be irradiated from the light source 120 . For example, when the light source 120 irradiates N irradiated lights to the subject 10 , the phase difference between irradiated lights at an adjacent time period among the irradiated lights may be a value obtained by dividing 360 degrees into N equal parts. That is, when N is 4, the phase of the irradiated lights may be 0 degrees, 90 degrees, 180 degrees, or 270 degrees, but is not limited thereto.

When the light source 120 projects the plurality of irradiated lights 210 onto the subject 10 with an idle time, the reflected light 220 reflected by the subject 10 passes through the optical shutter 130 independently of each other and passes through the imaging unit. (160) may be incident. The transmittance of the optical shutter 130 may change with time as shown in FIG. 2C . Accordingly, when the reflected light 220 passes through the optical shutter 130 , the waveform of the reflected light 220 may be modulated. A waveform of the modulated reflected lights 220 may depend on a phase of the reflected lights 220 and a change in transmittance of the optical shutter 130 over time. The imaging unit 160 may extract a phase difference between the reflected lights 220 and the irradiated lights 210 using the reflected lights 220 modulated by the optical shutter 130 .

3 is a diagram illustrating a plurality of images obtained according to an exemplary embodiment.

As described above, the depth image acquiring apparatus 100 may generate a single depth image by irradiating a plurality of irradiated lights having different phases, and then sequentially acquiring a plurality of images through the acquired reflected lights. In the following drawings, it is assumed that the plurality of images are IR images, but it can be understood by those of ordinary skill in the art related to the present embodiment that other types of images can be acquired.

Referring to FIG. 3 , the depth image acquisition apparatus 100 may sequentially acquire four IR images corresponding to a plurality of irradiation lights having initial phases of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. In this case, when the four IR images have different shooting times, motion blur may occur. For example, while sequentially photographing four IR images, a subject moves or the depth image acquisition apparatus 100 recognizes different objects as the same subject and captures the same. Such a motion blur phenomenon frequently occurs at the boundary of a moving object, and also occurs frequently when the object moves quickly or when the image is taken slowly.

,

,

,

를 통해 생성된 복수의 영상(310 내지 340)에서 피사체의 움직임에 의한 모션 블러 현상이 발생되었다. 구체적으로, 사람이 손을 흔드는 동안, 영상 310 내지 340이 촬영 되었으므로, 모션 블러 현상이 발생되었다. Referring to FIG. 3 , the CIS output signal acquired by the imaging unit 160 .

,

,

,

In the plurality of images 310 to 340 generated through , motion blur caused by the movement of the subject occurred. Specifically, since images 310 to 340 were captured while a person waved a hand, motion blur occurred.

In this case, the motion blur phenomenon may have a greater effect on the depth image than on a normal image. For example, when a motion blur phenomenon occurs when capturing a normal image, the boundary surface of a moving object may be determined based on an average image with other adjacent objects, for example, average brightness or color. However, in the case of a depth image, completely different results may be derived due to motion blur in the process of calculating the depth image. Therefore, it is very important to remove the motion blur region when generating a depth image.

Hereinafter, in order to effectively remove the motion blur region generated by the movement of the subject, four images (eg, a first image corresponding to 0 degrees, a second image corresponding to 180 degrees, and a second image corresponding to 90 degrees) A method of detecting a motion blur region in the third image and the fourth image corresponding to 270 degrees) and a method of correcting an image or pixel in which motion blur is generated when motion blur is generated will be described in detail. Meanwhile, in the following drawings, a signal obtained by the imaging unit 160 to generate a plurality of images is displayed as a CIS output signal, but the imaging device 164 is not limited to CIS.

4 is a diagram for explaining a method of correcting a motion blur region, according to an embodiment.

First, referring to FIG. 4 , the apparatus for acquiring a depth image may acquire a plurality of images. Specifically, the depth imaging apparatus may sequentially obtain four images 410 through the obtained reflected light after irradiating the subject with irradiation light having an initial phase of 0 degrees, 90 degrees, 180 degrees, and 270 degrees.

If the motion blur region is included in the four images 410 , the depth image acquisition apparatus may detect 420 the motion blur region included in the four images 410 .

For example, the apparatus for acquiring a depth image may detect a motion blur region using the following equation.

In this case, B ₁ and B ₂ may be obtained through a plurality of CIS output signals, respectively, and are specifically defined by Equations 2 to 3. ε is a predetermined threshold and may be obtained through analysis.

는 초기 위상이 0도인 조사광에 대응하는 CIS 출력 신호를 의미하고,

는 초기 위상이 180도인 조사광에 대응하는 CIS 출력 신호를 의미한다. Referring to Equation 2,

denotes a CIS output signal corresponding to the irradiation light having an initial phase of 0 degrees,

denotes a CIS output signal corresponding to the irradiated light having an initial phase of 180 degrees.

는 초기 위상이 90도인 조사광에 대응하는 CIS 출력 신호를 의미하고,

는 초기 위상이 270도인 조사광에 대응하는 CIS 출력 신호를 의미한다. CIS 출력 신호의 단위는 [v]이지만, 다른 유형의 신호일 수 있음은 본 실시예와 관련된 기술분야에서 통상의 지식을 가진 자라면 이해할 수 있다.Referring to Equation 3,

denotes a CIS output signal corresponding to the irradiation light having an initial phase of 90 degrees,

denotes a CIS output signal corresponding to the irradiated light having an initial phase of 270 degrees. Although the unit of the CIS output signal is [v], it can be understood by those of ordinary skill in the art related to the present embodiment that it may be another type of signal.

Referring to Equations 1 to 3, if a CIS output signal corresponding to a region included in an image is used, it is possible to detect whether the corresponding region is a motion blur region. Also, the motion blur region may be adjusted according to the ε value of Equation 1.

Meanwhile, the apparatus for obtaining a depth image may detect a motion blur region according to another exemplary embodiment. Specifically, the difference in the CIS output signal between the plurality of images may be caused by an initial phase of the irradiated light or a motion blur phenomenon. Accordingly, by compensating for the difference in signal strength according to the initial phase difference of the irradiated light in the difference of the CIS output signal between the plurality of images, the CIS output signal according to the motion blur phenomenon can be determined, so that the motion blur region can be detected. .

For example, the apparatus for acquiring a depth image may arbitrarily select two images from among a plurality of images and then analyze a difference in intensity of a CIS output signal corresponding to the two images. After that, it is possible to determine the region where the difference in the intensity of the CIS output signal is generated due to the motion blur phenomenon by compensating for the difference according to the initial phase of the irradiated light among the causes of the difference in the intensity of the CIS output signal corresponding to the two images. In this case, there may be a method of using a cumulative distribution function with respect to the intensity of the CIS output signal to compensate for the difference according to the initial phase of the irradiated light. If the motion blur region is detected after compensating for the difference according to the initial phase of the irradiated light, the process of detecting the motion blur region can be performed after the process of analyzing the cumulative distribution function is a technical field related to the present embodiment. Anyone with ordinary knowledge can understand.

Meanwhile, a method of compensating for a difference according to the initial phase of the irradiated light using the cumulative distribution function regarding the intensity of the CIS output signal will be described in detail with reference to the accompanying drawings.

Meanwhile, after compensating for the difference according to the initial phase of the irradiation light, the apparatus for obtaining a depth image may correct the motion blur region ( 440 ). Specifically, a region of the reference image (eg, the first image) corresponding to the detected motion blur region is selected from the images (eg, the second image to the fourth image) other than the reference image. area can be replaced. When the motion blur region is corrected, since the subjects photographed differently in the plurality of images are corrected as if they are fixed, depth information with higher accuracy than when depth information is calculated from the original plurality of images can be calculated.

After correcting the motion blur region, the apparatus for obtaining a depth image may calculate depth information using the plurality of corrected images 450 in which the motion blur region is corrected and the reference image ( 460 ). A method of calculating depth information will be described in detail with reference to the drawings below.

Finally, the apparatus for obtaining a depth image may generate ( 470 ) a depth image based on the calculated depth information.

5 is a diagram illustrating a motion blur area detected in a plurality of images, according to an exemplary embodiment.

Referring to FIG. 5 , a motion blur region included in four images 510 to 540 may be detected. In this case, the detected motion blur region 550 may be the same region in the plurality of images.

After detecting the motion blur region, the apparatus for obtaining a depth image may compensate for a difference in intensity of the reflected light according to the initial phase difference with respect to the reflected light used as a basis for generating a plurality of images. In this case, the apparatus for obtaining a depth image may compensate for an intensity difference according to an initial phase by using a cumulative distribution function with respect to the intensity of the reflected light. A method of compensating for an intensity difference according to an initial phase will be described in detail with reference to FIGS. 6A to 6B .

6A is a histogram comparing intensities of a CIS output signal corresponding to a reference image and a CIS output signal corresponding to other images, according to an exemplary embodiment.

Referring to FIG. 6A , the reference signal in the first histogram 610 to the third histogram 630 is a CIS output signal corresponding to the irradiated light having an initial phase of 0 degrees. In this embodiment, the CIS output signal corresponding to the irradiated light having an initial phase of 0 degrees is determined as the reference signal. If so, you can understand

Specifically, the first histogram 610 shows a histogram according to the intensity of the CIS output signal corresponding to the irradiation light having an initial phase of 90 degrees and the intensity of the reference signal. The second histogram 620 shows a histogram according to the intensity of the CIS output signal corresponding to the irradiated light having an initial phase of 180 degrees and the intensity of the reference signal. Finally, the third histogram 630 shows a histogram according to the intensity of the CIS output signal corresponding to the irradiation light having an initial phase of 270 degrees and the intensity of the reference signal.

In this case, the X-axis of the histogram may be in the same unit as the intensity of the CIS output signal, and the Y-axis means the number of pixels having the intensity of the corresponding CIS output signal. Also, if the CIS output signal is a continuous function, a histogram can be obtained by quantizing the CIS output signal.

Referring to FIG. 6A , each CIS output signal shown in each histogram 610 to 630 has a different histogram from the reference signal. Among the causes of each CIS output signal having a different histogram from the reference signal, there may be an initial phase difference and motion blur. Accordingly, if the difference in the CIS output signal according to the initial phase difference is compensated, the depth image acquisition apparatus may more accurately detect the motion blur region.

6B is a graph illustrating a cumulative distribution function with respect to intensities of a CIS output signal corresponding to a reference image and a CIS output signal corresponding to other images, according to an exemplary embodiment.

Specifically, the reference signal in the first graph 640 to the third graph 660 is a CIS output signal corresponding to the irradiation light having an initial phase of 0 degrees. In this embodiment, the CIS output signal corresponding to the irradiated light having an initial phase of 0 degrees is determined as the reference signal. If so, you can understand The X-axis of each graph may have the same unit as the intensity of the CIS output signal, and the Y-axis means the number of accumulated pixels of the corresponding CIS output signal intensity.

In detail, the first graph 640 shows the cumulative distribution function according to the intensity of the CIS output signal corresponding to the irradiation light having an initial phase of 90 degrees and the intensity of the reference signal. The second graph 650 shows the cumulative distribution function according to the intensity of the CIS output signal corresponding to the irradiation light having an initial phase of 180 degrees and the intensity of the reference signal. Finally, the third graph 660 shows a cumulative distribution function according to the intensity of the CIS output signal corresponding to the irradiated light having an initial phase of 270 degrees and the intensity of the reference signal.

Referring to 6b, each cumulative distribution function is different from the cumulative distribution function with respect to the reference signal. If the cumulative distribution function corresponding to each CIS output signal is adjusted to approximate the cumulative distribution function corresponding to the reference signal, an image in which the difference in intensity of the CIS output signal due to the initial phase difference is compensated may be obtained. That is, after compensating for the initial phase difference based on the cumulative distribution function, the apparatus for obtaining a depth image may correct the motion blur region.

7 is a diagram illustrating an image in which a motion blur region is corrected based on a patch match technique, according to an exemplary embodiment.

According to an embodiment, as a method of correcting the motion blur region, there may be a correction method based on a patchMatch technique. The patch-match technique, which is one of image editing techniques, refers to a technique for editing an image by replacing a selected region with a plurality of small regions (patches). For example, if the patch match technique is used, a predetermined subject included in the image may be replaced with a background screen, and an image in which the predetermined subject has disappeared may be edited. Accordingly, the depth image acquisition apparatus replaces the motion blur region included in the remaining images except for the reference image with a region corresponding to the motion blur region included in the reference image through the patch-matching technique to obtain the motion blur included in the plurality of images. area can be corrected. In the present embodiment, the motion blur region is corrected through the patch match technique, but it can be understood by those skilled in the art related to the present embodiment that the motion blur region can be corrected using other image editing techniques.

Referring to FIG. 7 , the apparatus for obtaining a depth image may correct motion blur regions of the remaining images 720 to 740 based on the reference image 710 . Specifically, the human arm part 550 detected as the motion blur region in FIG. 5 is corrected to be similar to the human arm part included in the reference image 710 . Accordingly, compared with FIG. 3 , it can be seen that the motion blur region is corrected in the plurality of images 710 to 740 .

8 is a diagram illustrating a depth image obtained by using an original depth image and an image in which a motion blur region is corrected, according to an exemplary embodiment.

Referring to FIG. 8 , in a depth image 810 generated using a plurality of images for which the motion blur region is not corrected, the depth level of the human arm is incorrectly displayed. However, according to an embodiment, the depth image 820 generated using the corrected images in which the motion blur region is corrected indicates that the depth of the human arm is more accurately displayed than the depth image 810 in which the motion blur region is not corrected. Able to know.

9 is a diagram illustrating an algorithm for generating a depth image after removing motion blur, according to an exemplary embodiment.

The algorithm shown in FIG. 9 was written in OpenCV (Open Computer Vision) language. OpenCV (Open Computer Vision) is an open source computer vision C library that focuses on real-time image processing. However, it can be understood by those of ordinary skill in the art related to the present embodiment that the algorithm for generating a depth image after removing the motion blur according to an embodiment of the present disclosure may be written in another language.

Referring to FIG. 9 , an algorithm according to an exemplary embodiment includes four images (Im0, Im90, Im180, Im270) obtained through irradiation light having an initial phase of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, and a patch size. (patchSize) and the maximum voltage value (maxV) and minimum voltage value (minV), which are the basis of the normalization process, are received, and three converted images (Im90_new, Im180_new, Im270_new) and a depth image (I_depth) can be output. .

First, according to the algorithm disclosed according to an embodiment, a motion blur region (blurMask) may be detected using four input images. After detecting the motion blur region, the reference image may be normalized to an image between the maximum voltage value and the minimum voltage value. Thereafter, a histogram of a portion corresponding to the motion blur region in the normalized reference image may be created (imageHistgram).

Also, after a histogram corresponding to the reference image is created, a normalization process and a process of creating a histogram for the remaining images may be performed. In this case, the cumulative distribution function of the plurality of images other than the reference image may be performed transform to approximate the cumulative distribution function of the reference image. After adjusting the cumulative distribution function, the motion blur area is corrected by patchMatching the rest of the plurality of images except the motion reference image and the area corresponding to the motion blur area in the reference image in consideration of the patch size using the patch matching technique. A reconstructed image may be created (reconstrctImage).

After the process of generating an image in which the motion blur region is corrected from the plurality of images other than the reference image is repeatedly performed, the motion blur region is removed through the images (Im90_new, Im180_new, Im270_new) and the reference image Im0. Depth information (I_depth) may be calculated.

10 is a block diagram illustrating an apparatus for acquiring a depth image, according to an embodiment.

The depth image acquisition apparatus 1000 may include a sensing unit 1010 , a correcting unit 1020 , and a depth image processing unit 1030 . Only components related to the embodiments are shown in the apparatus 1000 for obtaining a depth image shown in FIG. 10 . Accordingly, it can be understood by those skilled in the art that other general-purpose components may be further included in addition to the components shown in FIG. 10 .

The sensing unit 1010 may detect a motion blur region generated by a movement of a subject based on a plurality of original images obtained through irradiation lights having different phases. In this case, the plurality of original images have a first original image corresponding to 0 degrees, a second original image corresponding to 180 degrees, a third original image corresponding to 90 degrees, and a fourth original image corresponding to 270 degrees, respectively, in which the initial phase of the irradiated light corresponds to 0 degrees. It can be defined as an image.

According to an embodiment, the sensing unit 1010 may include a first group indicating the sum of intensities of reflected lights corresponding to the first original image and the second original image, the third original image, and the fourth based on Equation (1). When the difference between the second group, which means the sum of intensities of reflected lights corresponding to the original image, is equal to or greater than a predetermined threshold, a motion blur region may be detected from the plurality of original images.

According to another exemplary embodiment, the sensing unit 1010 selects two original images from among a plurality of original images, and the difference between the intensity of reflected light corresponding to the first region included in the two original images affects the movement of the subject. If it is determined that the result is caused by the motion blur, the first area may be detected as a motion blur area.

In this case, the motion blur region may mean the same region in the plurality of original images.

The compensator 1020 may compensate for an intensity difference according to an initial phase difference of reflected lights reflected from the subject with respect to the original images, and may correct a motion blur region in the compensated images generated by the compensation. Also, the compensator 1020 selects a reference image from among the plurality of original images, and a cumulative distribution regarding the intensity of reflected light corresponding to the remaining original images to approximate the cumulative distribution function with respect to the intensity of reflected light corresponding to the reference image. You can tweak the function. In addition, the compensator 1020 may generate images in which the intensity difference is compensated. Meanwhile, the compensator 1020 may correct the motion blur region in the compensation images by using a PatchMatch technique.

The depth image processing unit 1030 may generate a depth image based on the corrected images generated by the correction. If the depth image generating apparatus 1000 receives four original images (first to fourth original images) and corrected images ( When generating the second corrected image to the fourth corrected image), the depth image processing unit 1030 generates a phase obtained by using reflected light corresponding to the first original image and reflected light corresponding to the second to fourth corrected image. Depth information may be calculated based on the information and the modulation frequency. Specifically, the depth information may be calculated through the following equation.

는 초기 위상이 θ도인 CIS 출력 신호를 의미한다. 본 개시의 일 실시예에 따르면,

는 모션 블러 영역을 보정하여 생성된 보정 영상에 대응하는 신호일 수 있다. In this case, c is the speed of light (3*10 ⁸ m/s), f is the frequency of the irradiated light,

denotes a CIS output signal having an initial phase of θ degrees. According to an embodiment of the present disclosure,

may be a signal corresponding to a corrected image generated by correcting the motion blur region.

11 is a block diagram illustrating an apparatus for acquiring a depth image, according to another exemplary embodiment. According to the disclosed embodiment, the depth image acquisition apparatus 1100 includes a detector 1110 , a corrector 1120 , and a depth image processing unit 1130 , as well as a light source 1140 , an optical shutter 1150 , and an imaging unit ( 1160 ), a driving unit 1170 , and a control unit 1180 may be included. In the depth image acquisition apparatus 1100 illustrated in FIG. 11 , only components related to the embodiments are illustrated. Accordingly, it can be understood by those skilled in the art that other general-purpose components may be further included in addition to the components shown in FIG. 11 .

The sensing unit 1110 , the correcting unit 1120 , and the depth image processing unit 1130 correspond to the sensing unit 1010 , the correcting unit 1020 , and the depth image processing unit 1030 of FIG. 10 , and thus a detailed description thereof will be omitted. do.

The light source 1140, the optical shutter 1150, the imaging unit 1160, and the driving unit 1170 correspond to the light source 120, the optical shutter 130, the imaging unit 160, and the driving unit 140 of FIG. A detailed description will be omitted.

The controller 1180 may control the operation of the driving unit 1170 and the imaging unit 1160 . Also, the controller 1180 may control the driving unit 1170 to sequentially irradiate the irradiated lights having initial phases of 0 degrees, 180 degrees, 90 degrees, and 270 degrees, respectively. In this case, the plurality of original images may be a first original image corresponding to 0 degrees, a second original image corresponding to 180 degrees, a third original image corresponding to 90 degrees, and a fourth original image corresponding to 270 degrees. .

12 is a flowchart illustrating a method of acquiring a depth image using a ToF sensor, according to an exemplary embodiment.

In operation 1210, the apparatus 1000 for obtaining a depth image may detect a motion blur region generated by a movement of a subject in a plurality of original images acquired through irradiation lights having different phases. In this case, the irradiated lights having different phases are sequentially irradiated onto the subject, and the apparatus for obtaining a depth image may use a plurality of original images obtained by modulating reflected lights through an optical shutter. In this case, the initial phases of the irradiated lights are 0 degrees, 180 degrees, 90 degrees, and 270 degrees, and the plurality of original images are at a first original image corresponding to 0 degrees, a second original image corresponding to 180 degrees, and 90 degrees. It may include a corresponding third original image and a fourth original image corresponding to 270 degrees.

According to an embodiment, the apparatus 1000 for obtaining a depth image corresponds to a first group indicating a sum of intensities of reflected lights corresponding to the first original image and the second original image, and the third original image and the fourth original image. When the difference between the second group, which means the sum of the intensity of reflected lights, is greater than or equal to a predetermined threshold, a motion blur region may be detected from the plurality of original images.

According to another exemplary embodiment, the depth image acquisition apparatus 1000 selects any two original images from among the plurality of original images, and determines the difference between the intensity of the reflected light corresponding to the first region included in the two original images. When it is determined that the movement of the subject is caused, the first area may be detected as a motion blur area.

In this case, the motion blur region may mean the same region in the plurality of original images.

In operation 1220 , the depth image acquiring apparatus 1000 may compensate for a difference in intensity of reflected lights according to an initial phase difference between reflected lights reflected from a subject with respect to the original images. For example, after the depth image acquisition apparatus selects a reference image from among a plurality of original images, the cumulative distribution function regarding the intensity of reflected light with respect to the remaining original images is close to the cumulative distribution function with respect to the intensity of reflected light with respect to the reference image. can be adjusted to compensate for the difference in intensity of reflected light.

In operation 1230, the apparatus 1000 for obtaining a depth image may correct a motion blur region in the compensation images generated by compensation. In order to correct the motion blur region in the compensation images, a patchmatch technique may be used, but is not limited thereto.

In operation 1240 , the depth image acquisition apparatus 1000 may generate a depth image based on corrected images generated by correction of the motion blur region. Specifically, the depth image acquisition apparatus 1000 may generate a depth image based on Equation (4).

13A to 13B are detailed flowcharts specifically illustrating a method of acquiring a depth image using a ToF sensor, according to an exemplary embodiment.

In operation 1310, the light source 1140 may sequentially irradiate the subject with irradiation lights having initial phases of 0 degrees, 180 degrees, 90 degrees, and 270 degrees, respectively.

In operation 1320 , the depth image obtaining apparatus 1100 may modulate the waveform of the reflected light by changing the transmittance of the reflected light reflected on the subject using the optical shutter 1150 . Also, the imaging unit 1160 may generate first to fourth original images having phases of 0 degrees, 180 degrees, 90 degrees, and 270 degrees, respectively, by using the modulated reflected light.

In step 1330 , the sensing unit 1110 of the depth image acquisition apparatus 1100 performs a first group B ₁ that is a sum of CIS output signals corresponding to the first original image and the second original image to detect the motion blur region, and The second group B ₂ that is the sum of the CIS output signals corresponding to the third original image and the fourth original image may be calculated. Specifically, the first group (B ₁ ) and the second group (B ₂ ) may be calculated based on Equations 2 to 3 .

In operation 1340 , the detector 1110 of the apparatus 1100 for obtaining a depth image may compare the absolute value of the difference between the first group and the second group with a predetermined threshold value. If the absolute value is greater than the threshold value, it may be determined that the region corresponding to the CIS output signal included in the first group and the second group is the motion blur region. However, if the absolute value is smaller than the threshold value, the apparatus 1100 for obtaining a depth image may determine that there is no motion blur region.

In operation 1350, the correction unit 1120 of the depth image acquisition apparatus 1100 calculates the cumulative distribution function regarding the intensity of the CIS output signal corresponding to the second original image to the fourth original image of the CIS output signal corresponding to the first image. It can be adjusted to approximate the cumulative distribution function with respect to intensity. In this case, the corrector 1120 of the apparatus 1100 for obtaining a depth image may generate images in which the difference in intensity is compensated.

In operation 1360 , the corrector 1120 of the depth image acquisition apparatus 1100 may correct the motion blur region in the compensation images using a patchMatch technique. Specifically, the compensator 1120 patch-matches the motion blur region included in the remaining compensation images (the second compensation image to the fourth compensation image) based on the motion blur region of the reference image (the first compensation image). , the motion blur area can be corrected. As a method of correcting the motion blur region, various image editing techniques may be used in addition to the patch matching technique, but the present invention is not limited thereto.

In operation 1370 , the corrector 1120 of the apparatus 1100 for obtaining a depth image may generate three corrected images in which the motion blur region is corrected.

In operation 1380 , the depth image processing unit 1130 of the apparatus 1100 for obtaining a depth image may generate a depth image using three corrected images and a reference image. Since the method of generating the depth image corresponds to step 1240 of FIG. 12 , a detailed description thereof will be omitted.

The present embodiments may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, or other data in modulated data signals, such as program modules, or other transport mechanisms, and includes any information delivery media.

The scope of the present embodiment is indicated by the claims to be described later rather than the above detailed description, and it should be construed as including all changes or modifications derived from the meaning and scope of the claims and their equivalents.

Claims (20)

A method of generating a depth image using a Time-of-Flight (ToF) sensor, the method comprising:
sequentially irradiating light having a plurality of phases toward a subject;
generating a plurality of original images corresponding to the plurality of phases based on the light sequentially reflected from the subject;
setting one of the plurality of original images as a reference image and setting other original images as the plurality of remaining images;
Compensating for a difference in intensity of reflected light between the reference image and the plurality of remaining images with respect to the plurality of remaining images based on a difference between the phase corresponding to the reference image and phases corresponding to the plurality of remaining images correcting the motion blur region in the plurality of remaining images by doing so; and
generating a depth image based on the reference image and the plurality of compensated remaining images including the corrected motion blur region;
How to include. The method of claim 1,
Compensating for the intensity difference comprises:
adjusting the cumulative distribution function regarding the intensity of the reflected light corresponding to the remaining images to approximate the cumulative distribution function with respect to the intensity of the reflected light corresponding to the reference image;
A method comprising The method of claim 1,
The step of correcting the motion blur region includes:
Compensating for the motion blur region using a patch match (PatchMatch) technique in the compensation images. delete The method of claim 1,
The initial phases of the irradiation lights are 0 degrees, 180 degrees, 90 degrees and 270 degrees,
The plurality of original images may include a first original image corresponding to 0 degrees, a second original image corresponding to 180 degrees, a third original image corresponding to 90 degrees, and a fourth original image corresponding to 270 degrees. A method comprising 6. The method of claim 5,
The step of generating the depth image comprises:
calculating depth information based on phase information and modulation frequencies obtained using the reflected light corresponding to the first original image and the reflected light corresponding to the second to fourth corrected images;
A method comprising 6. The method of claim 5,
The step of detecting the motion blur region comprises:
The first group means the sum of the intensity of the reflected lights corresponding to the first original image and the second original image, and the sum of the intensity of the reflected lights corresponding to the third original image and the fourth original image The method of detecting the motion blur region from the plurality of original images when the difference between the second group is greater than or equal to a threshold value. The method of claim 1,
selecting two original images from among the plurality of original images;
further comprising,
When it is determined that the difference in intensity of the reflected light corresponding to the first area included in the two original images is due to the movement of the subject, detecting the first area as the motion blur area. The method of claim 1,
The method, wherein the motion blur region is determined to be the same region in the plurality of original images. A computer-readable recording medium recording a method for executing the method of any one of claims 1 to 3 and 5 to 9 on a computer. In the apparatus for generating a depth image using a ToF sensor,
a light source configured to sequentially emit light having a plurality of phases to a subject;
a sensing unit for detecting the light sequentially reflected from the subject; and
A plurality of original images corresponding to the plurality of phases are generated based on the light sequentially reflected from the subject, one of the plurality of original images is set as a reference image, and the other original images are set as a plurality of remaining images. , and based on a difference between a phase corresponding to the reference image and phases corresponding to the plurality of remaining images, the amount of reflected light between the reference image and the plurality of remaining images for the plurality of remaining images a processing unit for correcting a motion blur region in the plurality of remaining images by compensating for an intensity difference, and generating a depth image based on the reference image and the plurality of compensated remaining images including the corrected motion blur region;
A device comprising a. 12. The method of claim 11,
The processing unit,
and adjusting the cumulative distribution function regarding the intensity of the reflected light corresponding to the plurality of remaining images to approximate the cumulative distribution function with respect to the intensity of the reflected light corresponding to the reference image. 12. The method of claim 11,
The processing unit,
Compensating for the motion blur region using a patch match (PatchMatch) technique in the compensation images. 12. The method of claim 11,
an optical shutter modulating a waveform of the reflected lights by changing the transmittance of the reflected lights reflected from the subject; and
an imaging unit generating a plurality of original images with the reflected light modulated by the optical shutter;
Further comprising, the device. 15. The method of claim 14,
a driver applying driving voltages to the light source and the optical shutter; and
a control unit controlling operations of the driving unit and the imaging unit;
Further comprising, the device. 16. The method of claim 15,
The control unit is
controlling the driving unit so that the irradiated lights having the initial phase of 0 degrees, 180 degrees, 90 degrees, and 270 degrees are sequentially irradiated,
The plurality of original images may include a first original image corresponding to 0 degrees, a second original image corresponding to 180 degrees, a third original image corresponding to 90 degrees, and a fourth original image corresponding to 270 degrees. A device comprising a. 17. The method of claim 16,
The processing unit,
and calculating depth information based on phase information and modulation frequencies obtained using reflected light corresponding to the first original image and reflected light corresponding to second to fourth corrected images. 17. The method of claim 16,
The sensing unit,
The first group means the sum of the intensity of the reflected lights corresponding to the first original image and the second original image, and the sum of the intensity of the reflected lights corresponding to the third original image and the fourth original image When the difference between the second group is greater than or equal to a threshold, detecting the motion blur region from the plurality of original images. 12. The method of claim 11,
The sensing unit,
selecting two original images from among the plurality of original images,
When it is determined that the difference in intensity of the reflected light corresponding to the first area included in the two original images is due to the movement of the subject, detecting the first area as the motion blur area. 12. The method of claim 11,
The motion blur region is determined to be the same region in the plurality of original images.

Images