TWI631849B - Apparatus for generating depth image - Google Patents

Abstract

The present invention provides an apparatus for generating a depth image. The apparatus according to an exemplary embodiment of the present invention is configured to perform accurate stereo matching by obtaining RGB images and/or IR images even in the case of low brightness; and obtaining depth using the obtained RGB image and/or IR image Capture of images.

Description

Device for generating depth images

The present invention claims priority to the Korean Patent Publication No. 10-2012-0097101, filed on Sep. 3, 2012, which is hereby incorporated by reference.

The present invention relates to an apparatus for generating a depth image.

As is well known, human vision is a type of function that is used to obtain information about the surrounding environment, and the position, and the distance of an object can be recognized through both eyes. That is to say, the phantom information input through both eyes is synthesized into a distance information to allow humans to freely move. In the past, a stereo camera system was used to implement a phantom structure in a machine.

The stereo camera system performs a stereo matching on an image obtained by two cameras, wherein the stereo camera system obtains a depth image in a stereo matching program using binocular disparity of the two cameras. Depth image data is used for identification purposes.

In addition, at low brightness, it is difficult for a stereo camera to recognize an image, which in turn causes problems in obtaining a depth image.

The embodiments of the present invention are to address at least one of the above problems and/or disadvantages, and to provide at least one of the advantages. Accordingly, the present invention provides for generating a depth An image device that produces a depth image by obtaining an infrared (IR) image at low brightness.

In an embodiment of the invention, an apparatus for generating a depth image is provided, the apparatus comprising: an imaging unit configured to output left and right RGB (red-green-blue) images and left and right IR images; The first generating unit is configured to generate a depth image from the IR images, wherein the IR images are images received by the camera unit.

In an embodiment of the invention, the camera unit may include: a first sensing unit configured to obtain the left RGB image and the left IR image, and a second sensing unit configured to obtain the right RGB image and the right IR The image, and the at least one firing unit are configured to emit an infrared ray.

In an embodiment of the invention, the transmitting unit may comprise a single IR device or a plurality of IR devices.

In an embodiment of the invention, the IR device may comprise any of an LD (Laser Diode) and an LED (Light Emitting Diode).

In an embodiment of the invention, the image capturing unit may further include a beam splitter arranged at a front end of the emitting unit to disperse the infrared rays emitted from the emitting unit in a predetermined pattern.

In an embodiment of the invention, the first generating unit includes: a synchronization unit configured to synchronize the left and right IR images in the outputs of all the first and second sensing units, and a second generating unit Configuring to generate the depth image from the left and right IR images, wherein the left and right IR images are received by the synchronization unit.

In an embodiment of the invention, an apparatus for generating a depth image is provided, the apparatus comprising: an imaging unit configured to obtain left and right RGB images, left and right RGB and IR images (RGB images/IR images), And any of the left and right IR images; and a first production order Meta-configure: control the camera unit to obtain left and right RGB images, left and right RGB/IR images, and left and right IR images, and generate from the left and right RGB images or the left and right IR images A depth image, wherein the left and right RGB images or the left and right IR images are received by the camera unit.

In an embodiment of the invention, the camera unit may include: a first sensing unit configured to obtain any of a left RGB image, a left RGB/IR image, and a left IR image; and a second sensing unit configured to obtain Any one of a right RGB image, a right RGB/IR image, and a right IR image; a light receiving unit configured to detect an illumination intensity, and at least one firing unit configured to emit an infrared (IR) by turning it on or off Responding to the control of the first generating unit.

In an embodiment of the invention, the transmitting unit may comprise a single IR device or a plurality of IR devices.

In an embodiment of the invention, the IR device can include any of an LD and an LED.

In an embodiment of the invention, the image capturing unit may further include a beam splitter arranged at a front end of the emitting unit to disperse the infrared rays emitted from the emitting unit in a predetermined pattern.

In an embodiment of the invention, the first generating unit may include: a controller configured to control ON (ON) and OFF (OFF) of the transmitting unit in response to a light intensity detected by the light receiving unit, a synchronization unit configured to synchronize left and right IR images output by the first and second sensing units, and a second generation unit configured to generate the left and right IR images received from the synchronization unit The depth image.

In an embodiment of the invention, the first generating unit may further include: The off unit is configured to turn the transmit unit on or off in response to control of the controller.

In an embodiment of the invention, the controller is configurable to change the firing unit to an ON state when the light receiving unit detects that the light intensity is below a predetermined illumination intensity.

In another embodiment of the present invention, an apparatus for generating a depth image is provided, the apparatus comprising: an imaging unit configured to obtain left and right RGB images, or left and right RGB and IR images (RGB/IR images) And a first generating unit configured to control the camera unit to obtain an RGB image or an RGB/IR image, and generate a depth image from the left/right RGB image or the IR image received by the camera unit.

In another embodiment of the present invention, the camera unit may include: a first sensing unit configured to obtain the left RGB image or the left RGB/IR image, and a second sensing unit configured to obtain the right RGB image or The right RGB/IR image, and the at least one firing unit configuration, emits an infrared (IR) by turning on or off in response to control of the first generating unit.

In an embodiment of the invention, the transmitting unit may comprise a single IR device or a plurality of IR devices.

In an embodiment of the invention, the IR device can comprise one of an LD and an LED.

In an embodiment of the invention, the image capturing unit may further include: a beam splitter arranged at a front end of the emitting unit to disperse the infrared rays emitted from the emitting unit in a predetermined pattern.

In an embodiment of the invention, the first generating unit may include: a synchronization unit configured to synchronize the outputs of the first and second sensing units, and a second generating unit configured to receive from the synchronization unit Left and right IR images produce depth images and a controller configuration The transmitting unit is turned on or off in response to the average brightness of the left and right RGB image pixels received from the synchronization unit.

In an embodiment of the invention, the first generating unit may further include: a switch unit configured to turn the transmitting unit on or off in response to control of the controller.

In an embodiment of the invention, the controller is configurable to turn on the transmitting unit when the average pixel of the RGB image is less than a threshold.

An apparatus for generating a depth image according to an exemplary embodiment of the present invention has a superior advantage in that it can be simultaneously obtained by transmitting infrared rays (IR) onto a target even at a low light level. An RGB image and an IR image produce an accurate depth image.

Another advantage of the apparatus for generating a depth image according to an exemplary embodiment of the present invention is that the apparatus detects a low brightness while performing normal operations under normal illuminance to obtain an RGB image and an IR image. Or only the IR image, whereby the IR image is used to produce a precise depth image.

4‧‧‧ camera unit

5‧‧‧Deep image generation unit

6‧‧‧Encoder unit

14‧‧‧First sensing unit

15‧‧‧Infrared emitting unit

16‧‧‧Second sensing unit

17‧‧‧Light receiving unit

24‧‧‧Synchronization unit

25‧‧‧ alignment unit

26‧‧‧Generating unit

27‧‧‧ Controller

28‧‧‧Switch unit

29‧‧‧Photometric detector

30‧‧‧Switch unit

1 is a block diagram of an apparatus for generating a depth image in accordance with a first exemplary embodiment of the present invention.

2 is a block diagram of an apparatus for generating a depth image in accordance with a second exemplary embodiment of the present invention.

3 is a block diagram of an apparatus for generating a depth image in accordance with a third exemplary embodiment of the present invention.

The features and advantages of the present invention will become more apparent from the detailed description of the embodiments. As such, the invention is not limited to the exemplary embodiments that will be described hereinafter, and may be embodied in other forms. Therefore, the described considerations are intended to cover all modifications, adaptations and variations, which are within the scope of the invention.

Exemplary embodiments of the present invention will now be described in detail with reference to the drawings.

1 is a block diagram of an apparatus for generating a depth image in accordance with a first exemplary embodiment of the present invention.

Referring to FIG. 1 , a depth image according to a first exemplary embodiment of the present invention may include an image capturing unit 1 and a depth image generating unit 2, wherein a depth image generated by the depth image generating unit 2 may be provided to an encoder. Unit 3. The encoder unit 3 is used to encode a stereoscopic image received by, for example, H.264, and the description thereof will be omitted hereinafter because the image encoding technique is a conventional technique, and it should also be applied to An exemplary embodiment of the invention.

Although the exemplary embodiment of the present invention has been explained and illustrated that the imaging unit 1, the depth image generating unit 2, and the encoder unit 3 are formed as separate hardware configurations, the present invention is not limited thereto. Therefore, the imaging unit 1 and the depth image generating unit are 2 and the encoder unit 3 may be formed as a separate wafer in the same hard body (for example, a video camera or the like) or may be formed as a wafer.

The imaging unit 1 according to an exemplary embodiment of the present invention includes a first sensing unit 11, an infrared (IR) emitting unit 12, and a second sensing unit 13, and obtains RGB (red-green-blue) images and relative images. IR image of the target. It should be obvious that it is exemplary according to the present invention. The elements of the image pickup unit 1 of the embodiment are not limited to the arrangement in the drawings.

The first and second sensing units 11, 13 may be, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor, and obtain left and right RGB images and IR images, respectively. However, it should be apparent to those skilled in the art that the first and second sensing units 11, 13 are not limited to CCD sensors or CMOS sensors, and any device capable of performing the same function can also be used as First and second sensing units 11, 13.

In the exemplary embodiment of the present invention, for convenience of explanation, it will be explained later that when the image obtained by the first sensing unit 11 is the left image and the image obtained by the second sensing unit 13 is the right image, However, the present invention is not limited thereto.

The IR transmitting unit 12 can include a single IR device or a plurality of IR devices and emit an infrared (IR) to a target. Although in the exemplary embodiment of the present invention, the IR emitting unit 12 includes an IR device, such as an LD (Laser Diode) or an LED (Light Emitting Diode), the IR device is not limited thereto and should be obvious. The application of an IR device capable of emitting infrared light to a target should not be excluded.

Although the exemplary embodiment of the present invention has been explained and illustrated only by an IR transmitting unit 12, the present invention is not limited thereto. That is to say, when the number of IR transmitting units 12 is increased, the output is also increased at the same time, which indicates that the increase in the number of IR transmitting units 12 can increase the IR identification range at low luminance and for recognizing the long distance, and thus the IR transmitting unit The number of 12 can be judged based on the appropriate identification distance.

Meanwhile, the IR transmitting unit 12 includes an LD having a straightness. The depth image according to the first exemplary embodiment of the present invention may further include: a beam splitter arranged at the front end of the IR transmitting unit 12 to be the self-IR emitting unit 12 The infrared rays (IR) emitted are scattered at a predetermined time In the pattern. According to the first exemplary embodiment of the present invention, the first and second sensing units 11, 13 can simultaneously obtain RGB images and IR images by infrared rays (IR) emitted from the IR emitting unit 12.

The depth image generating unit 2 in FIG. 1 may include a synchronization unit 21, a pair of bit units 22, and a generating unit 23.

The synchronization unit 21 can receive left and right IR images from the first and second sensing units 11, 13. The synchronization unit 21 can synchronize the received left and right IR images. The term "synchronization" as used in the exemplary embodiments of the present invention represents an equal alignment of time. That is, the synchronization unit 21 performs alignment at substantially the same time for the received left and right images.

The alignment unit 22 is corrected so that the input left and right IR images are aligned. In the process of the alignment performed by the alignment unit 22, the encoder unit 3 can receive a correction data from the first and second sensing units 11, 13 of the imaging unit 1, convert the correction data into an arrangement parameter, and Transfer to the registration unit 22. The registration unit 22 can correct the image that has been synchronized by the synchronization unit 21 using the arrangement parameters received from the encoder unit 3.

The generating unit 23 can generate a depth image from the aligned left and right images. The method for generating the depth image by the generating unit 23 may include a stereo matching method, but the present invention is not limited thereto, and other different depth image generating methods may be used. The operations of the registration unit 22 and the generation unit 23 are well known to those skilled in the art, and thus the description thereof will be omitted.

Now, the operation of the apparatus for generating a depth image according to the first exemplary embodiment of the present invention will be described in detail.

Apparatus for generating a depth image according to the first exemplary embodiment of the present invention The IR transmitting unit 12 is in an ON state, and emits an infrared ray to a target, while the first and second sensing units 11, 13 respectively obtain RGB images and IR images.

Thereafter, the synchronization unit 21 can synchronize the left and right IR images and transmit the synchronized left and right IR images to the registration unit 22. The alignment unit 22 can arrange the left and right images, and the generating unit 23 can generate a depth image according to the left and right images.

Therefore, according to the first exemplary embodiment of the present invention, the IR emitting unit 12 can emit infrared rays, and the first and second sensing units 11, 13 can obtain the RGB image and the IR image, and generate the RGB image and the A depth image of an IR image of an IR image and an accurate depth image extracted at low light levels.

2 is a block diagram of an apparatus for generating a depth image in accordance with a second exemplary embodiment of the present invention.

Referring to FIG. 2, a device depth image for generating a depth image according to a second exemplary embodiment of the present invention may include an image capturing unit 1 and a depth image generating unit 2, wherein the depth image generated by the depth image generating unit 2 may be It is supplied to an encoder unit 3.

The camera unit 1 according to the second exemplary embodiment of the present invention may include a first sensing unit 11, an IR transmitting unit 12, a light receiving unit 14, and a second sensing unit 13, wherein the depth image generating unit 2 may include A synchronization unit 21, a pair of bit units 22, a generating unit 23, a controller 24, and a switching unit 25.

The second exemplary embodiment of the present invention is structurally different from the exemplary first embodiment of the present invention in that the second embodiment includes the light receiving unit 14 of the image capturing unit 1, and the controller 24 and the switching unit of the depth image generating unit 2 25, therefore, the description of the differences from the first embodiment will be emphasized.

The first and second sensing units 11, 13 of the camera unit 1 respectively obtain left and right RGB images or RGB images and IR images. The IR transmitting unit 12 is in an OFF state at normal illumination intensity, but when it is detected that the illumination intensity of the light receiving unit 14 has a luminosity lower than a predetermined illumination intensity, it is changed by the control of the controller 24 to The ON state will be described in detail later.

The light receiving unit 14 can receive the light in which the area of the image pickup unit 1 is arranged, and transmits a light intensity (illuminance) to the controller 24 of the depth image generating unit 2. The light receiving unit 14 may include a light sensor or an illuminance sensor, but the invention is not limited thereto.

The controller 24 of the depth image generating unit 2 can receive the light intensity from the light receiving unit 14 and control the switching unit 22 to turn on the IR transmitting unit 12 when the received light intensity is less than a predetermined illuminance and turn off the IR transmitting unit 12 at the received When the light intensity is greater than the predetermined luminosity. The switching unit 22 can switch the ON (ON) and the OFF (OFF) of the IR transmitting unit 12 in response to the control of the controller 24.

The sync unit 21 according to the second exemplary embodiment of the present invention can receive RGB images from the first and second sensing units 11, 13 at normal light intensity, and receive RGB images and IR images or less when less than a predetermined illuminance Receive IR images.

Now, the operation of the apparatus for generating a depth image according to the second exemplary embodiment of the present invention will be described.

At normal light intensity, for example, when the light intensity detected by the light receiving unit 14 is within normal luminosity, the apparatus for generating a depth image according to the second exemplary embodiment of the present invention will operate in such a manner: Synchronizing the RGB images obtained by the first and second sensing units 11, 13 by the synchronizing unit 21, arranging the left and right images of the aligning unit 22, and generating a single Element 23 produces a depth image based on the arranged left and right images.

However, in the case where the light intensity detected by the light receiving unit 14 is less than the predetermined illuminance, the controller 24 controls the switching unit 25 to change the IR transmitting unit 12 to the ON state.

In the case where the IR emitting unit 12 is changed to the ON state to emit an infrared ray to an object, the first and second sensing units 11, 13 can respectively obtain RGB image and IR image, or only IR image. The images supplied to the sync unit 21 are IR images in the obtained RGB video and IR video or IR only video.

Thereafter, the synchronization unit 21 can synchronize the input left and right IR images, and the alignment unit 22 can arrange the left and right images, whereby the generating unit 23 can generate a depth image.

According to the second exemplary embodiment of the present invention, the device can detect low brightness while performing normal operations under normal light, obtain RGB image and IR image, or only IR image, thereby using the IR image to generate accurate depth image. .

3 is a block diagram of an apparatus for generating a depth image in accordance with a third exemplary embodiment of the present invention.

Referring to FIG. 3, an apparatus for generating a depth image according to a third exemplary embodiment of the present invention may include an image capturing unit 1 and a depth image generating unit 2, wherein a depth image generated by the depth image generating unit may be provided to An encoder unit 3.

The camera unit 1 according to the third exemplary embodiment of the present invention may include a first sensing unit 11, an IR transmitting unit 12, and a second sensing unit 13, wherein the depth image generating unit 2 may include a synchronization unit 21, A pair of bit units 22, a generating unit 23, a photometric detector 26, and a switching unit 27.

The third exemplary embodiment of the present invention is structurally different from the exemplary first embodiment of the present invention in that the third embodiment includes the photometric detector 26 and the switching unit 27 of the depth image generating unit 2, and therefore, Description of differences from the first embodiment.

The first and second sensing units 11, 13 of the camera unit 1 respectively obtain left/right RGB images and/or IR images. The IR transmitting unit 12 is in an OFF state under normal illumination intensity, and is changed by switching of the switching unit 27 in the case where it is detected that the illumination intensity of the photometric detector 26 has a luminosity lower than a predetermined illumination intensity. In the ON state.

The photometric detector 24 of the depth image generating unit 2 can receive the RGB image synchronized by the synchronization unit 21 to monitor the average of the pixel values of the pixels, and when the average of the pixels is less than a threshold. It is judged that the light is at a low illuminance, wherein the switching unit 27 can change the IR emitting unit 12 to an ON state.

The switching unit 27 can switch the ON (ON) and the OFF (OFF) of the IR transmitting unit 12 in response to the control of the photodetector 26.

The sync unit 21 according to the third exemplary embodiment of the present invention can receive RGB images from the first and second sensing units 11, 13 at normal light intensity, and receive RGB images and IR images below a predetermined illuminance. In the third exemplary embodiment of the present invention, the photometric detector 26 uses the pixels of the RGB images to detect the intensity of luminance, whereby the RGB images can be received without considering brightness. .

Now, an apparatus for generating a depth image according to a third exemplary embodiment of the present invention will be described in detail.

Under normal light intensity, for example, in the case where the photometric detector 26 judges the light system from the average value of the pixels of the RGB image at normal light intensity, according to the third exemplary embodiment of the present invention The apparatus for generating a depth image will operate in such a manner that the RGB images obtained by the first and second sensing units 11, 13 are synchronized by the synchronization unit 21, the left and right images are aligned by the alignment unit 22, and The generating unit 23 generates a depth image based on the arranged left and right images. At this time, the IR transmitting unit 12 is in an OFF state.

However, when it is judged that the average value of the pixels of the RBG image is smaller than the threshold value, the photometric detector 26 controls the switching unit 27 to change the IR transmitting unit 12 to the ON state.

When the IR emitting unit 12 becomes an ON state to emit infrared rays to a target, the first and second sensing units 11, 13 can respectively obtain RGB images and IR images.

Thereafter, the synchronization unit 21 can synchronize the left and right RGB and IR images, and the alignment unit 22 can arrange the IR images, whereby the generating unit 23 can generate a depth image.

According to the third exemplary embodiment of the present invention, the device can detect low brightness while performing normal operations to obtain RGB images and IR images under normal illuminance, so that the IR images can be used to generate accurate depth images.

While the invention has been described with respect to the embodiments of the invention In particular, various changes and modifications may be made to the inventions and/or combinations of the components and/or combinations thereof, which are within the scope of the disclosure, the drawings and the scope of the claims.

Claims (16)

A device for generating a depth image, the device comprising: an imaging unit configured to output a left RGB (red-green-blue) image and a right RGB (red-green-blue) image and a left IR (infrared) An image and a right IR (infrared) image; and a first generating unit configured to generate a depth image from the IR images, wherein the IR images are images received by the camera unit, wherein the camera unit comprises: At least one transmitting unit configured to emit an infrared ray (IR), and a beam splitter configured to disperse infrared rays emitted from the emitting unit in a predetermined pattern, wherein the camera unit further comprises: a first sensing unit Configuring to obtain the left RGB image and the left IR image, a second sensing unit configured to obtain the right RGB image and the right IR image, wherein the first generating unit comprises: a synchronization unit configured to be Synchronizing the left IR image and the right IR image in the outputs of all the first sensing unit and the second sensing unit; a second generating unit configured to generate the depth image from the left IR image and the right IR image, wherein the left IR image and the right IR image are received by the synchronization unit. The device of claim 1, wherein the transmitting unit comprises a single IR device or a plurality of IR devices. The device of claim 2, wherein the IR device comprises any one of an LD (Laser Diode) and an LED (Light Emitting Diode). The device of claim 1, wherein the beam splitter is disposed at a front end of the transmitting unit. A device for generating a depth image, the device comprising: a camera unit configured to obtain a left RGB image and a right RGB image, a left RGB and IR image (RGB/IR image), and a right RGB and IR image ( RGB/IR image), and any one of a left IR image and a right IR image; and a first generating unit configured to: control the camera unit to obtain the left and right RGB images, the left and the a right RGB/IR image, and the left and right IR images, and a depth image from the left and right RGB images or the left and right IR images, wherein the left and right RGB images or the left and the right The right IR image is received by the camera unit, wherein the camera unit comprises: At least one transmitting unit configured to emit an infrared ray (IR) by turning on or off in response to control of the first generating unit, and a beam splitter configured to disperse infrared rays emitted from the transmitting unit in a predetermined pattern The camera unit further includes: a first sensing unit configured to obtain any one of the left RGB image, the left RGB/IR image, and the left IR image, and a second sensing unit configured to Obtaining any one of the right RGB image, the right RGB/IR image, and the right IR image, and a light receiving unit configured to detect an illumination intensity, wherein the first generating unit comprises: a controller, configured To control the opening or closing of the transmitting unit in response to a light intensity detected by the light receiving unit, a synchronization unit configured to be output from the first sensing unit and the second sensing unit The left IR image and the right IR image are synchronized; and a second generating unit configured to generate the depth image from the left and right IR images, wherein the left and right IR images are received by the synchronization unit. The device of claim 5, wherein the transmitting unit comprises a single IR device or a plurality of IR devices. The device of claim 6, wherein the IR device comprises any one of an LD (Laser Diode) and an LED (Light Emitting Diode). The device of claim 5, wherein the beam splitter is disposed at a front end of the transmitting unit. The device of claim 5, wherein the first generating unit further comprises: a switching unit configured to turn the transmitting unit on or off in response to control of the controller. The device of claim 5, wherein the controller is configured to change the transmitting unit to an on state when the light intensity detected by the light receiving unit is lower than a predetermined illumination intensity. A device for generating a depth image, the device comprising: a camera unit configured to obtain a left RGB image and a right RGB image, or a left RGB and IR image (RGB/IR image) and a right RGB and IR image (RGB/IR image); and a first generating unit configured to: control the camera unit to obtain the RGB images or the RGB/IR images; Generating a depth image from the left/right RGB image or the IR images, wherein the left/right RGB image or the IR images are received by the camera unit, wherein the camera unit comprises: at least one transmitting unit, Configuring to emit an infrared ray (IR) by turning on or off in response to control of the first generating unit, and a beam splitter configured to disperse infrared rays emitted from the transmitting unit in a predetermined pattern, wherein the image capturing The unit further includes: a first sensing unit configured to obtain the left RGB image or the left RGB/IR image, and a second sensing unit configured to obtain the right RGB image or the right RGB/IR image, wherein The first generating unit includes: a synchronization unit configured to synchronize the outputs of the first sensing unit and the second sensing unit; and a second generating unit configured to receive the left IR image and the right IR image Generating the depth image, wherein the left IR image and the right IR image are received by the synchronization unit; A controller configured to turn the transmitting unit on or off in response to an average brightness of the left RGB image pixel and the right RGB image pixel received from the synchronization unit. The device of claim 11, wherein the transmitting unit comprises a single IR device or a plurality of IR devices. The device of claim 12, wherein the IR device comprises any one of an LD (Laser Diode) and an LED (Light Emitting Diode). The device of claim 11, wherein the beam splitter is disposed at a front end of the transmitting unit. The device of claim 11, wherein the first generating unit further comprises: a switching unit configured to turn the transmitting unit on or off in response to control of the controller. The device of claim 11, wherein the controller is configured to turn on the transmitting unit when an average of pixels of the RGB image is less than a threshold.