WO2020235458A1 - Image-processing device, method, and electronic apparatus - Google Patents

Abstract

An image-processing device (10) comprises: a difference calculation unit (11) that, on the basis of a light reception signal pertaining to reflection light of first distance-measurement light (L1) having a first wavelength (f1) and a light reception signal pertaining to reflection light of second distance-measurement light (L2) having a second wavelength (f2), calculates the difference (d) between the reflection intensity of the first distance-measurement light (L1) and the reflection intensity of the second distance-measurement light (L2) at the same distance-measurement position; and an output unit (12) that, on the basis of the distance (d), outputs a depth signal (SDP) based on same-phase component signals (I1, I2) and orthogonal component signals (Q1, Q2) that correspond to a light-reception signal (SL1) pertaining to the reflection light of the first distance-measurement light (L1) or to a light-reception signal (SL2) pertaining to the reflection light of the second distance-measurement light (L2) at each distance-measurement position.

Description

Image processing equipment, methods and electronic devices

This disclosure relates to image processing devices, methods and electronic devices.

In recent years, there has been an increase in the movement to equip smartphones with ToF (Time of Flight) sensors for distance measurement.
Then, autofocus control of the camera, photo processing for separating the subject and the background, gesture recognition for the user interface, etc. using the depth (distance) information acquired by the mounted ToF sensor have been proposed.
In addition, obstacle detection using a ToF sensor has been proposed for automatic driving and driving support.

Japanese Patent No. 5952111 Japanese Unexamined Patent Publication No. 2015-141394 Japanese Unexamined Patent Publication No. 2019-049480

By the way, the ToF sensor irradiates a predetermined distance measuring object with a predetermined ranging light and measures the distance to the measurement target position (each part of the measurement object) based on the difference between the phase of the irradiation light and the phase of the reflected light. doing.

Therefore, if the distance measuring light is disturbed by external light such as sunlight, or if the reflectance of the distance measuring light is different, there is a risk that correct distance measuring processing cannot be performed. As a result, there is a risk that correct obstacle detection and the like cannot be performed.

The present disclosure has been made in view of such a situation, and it is possible to reduce the influence of disturbance due to external light or the difference in reflectance in the measurement target, and to perform distance measurement processing at the measurement target position more accurately. It is intended to provide possible image processing devices, methods and electronic devices.

In order to achieve the above object, the image processing apparatus of the present disclosure receives a received signal of the reflected light of the first ranging light of the first wavelength and a received light of the reflected light of the second ranging light of the second wavelength. A difference calculation unit that calculates the difference between the reflection intensity of the first distance measurement light and the reflection intensity of the second distance measurement light at the same distance measurement position based on the signal, and the distance measurement position based on the difference. An output unit that outputs a depth signal based on the in-phase component signal and the orthogonal component signal corresponding to either the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light. , Equipped with.

It is a principle explanatory drawing of the image processing apparatus of embodiment. It is explanatory drawing of an example of the sunlight spectrum after passing through the atmosphere. It is a schematic block diagram of the image processing apparatus of 1st Embodiment. It is explanatory drawing of the 1st aspect of the light receiving part. It is explanatory drawing of the 2nd aspect of the light receiving part. It is explanatory drawing of pixel interpolation. It is explanatory drawing of an example of the functional block of the signal processing part of 1st Embodiment. It is a whole processing flowchart of an embodiment. It is a processing flowchart of the depth measurement processing of 1st Embodiment. It is explanatory drawing of an example of the functional block of the signal processing part of 2nd Embodiment. It is a processing flowchart of the depth measurement processing of 2nd Embodiment. It is a processing flowchart of a reliability calculation process. It is explanatory drawing of an example of the functional block of the signal processing part of 3rd Embodiment. It is explanatory drawing of 4th Embodiment.

Next, the embodiments of the present disclosure will be described in detail with reference to the drawings.
[1] Explanation of the Principle of the Embodiment First, prior to the detailed explanation of the embodiment, the principle of the embodiment will be described.

FIG. 1 is an explanatory diagram of the principle of the image processing apparatus of the embodiment.
In the following description, the sunlight spectrum after passing through the atmosphere means, for example, the sunlight spectrum under the conditions of Air Mass 1.5 defined in ASTM [American Society for Testing and Materials].
The image processing device 10 of the embodiment includes a difference calculation unit 11 and an output unit 12.
In the sunlight spectrum after passing through the atmosphere, the difference calculation unit 11 has relative light receiving signals SL1 and spectral radiant intensity of the reflected light of the first ranging light L1 having a first wavelength λ1 having a relatively high spectral radiant intensity. Based on the received signal SL2 of the reflected light of the second ranging light L2 having a second wavelength λ2 (> f1), which is relatively low, the reflected intensity of the first ranging light L1 and the second ranging light L2 The difference d between the reflection intensity and the reflection intensity is calculated.
The output unit 12 has the in-phase component signal I1 and the orthogonal component signal Q1 corresponding to the light-receiving signal SL1 or the in-phase component signal I2 and the orthogonal component signal Q2 corresponding to the light-receiving signal SL2 based on the difference d calculated by the difference calculation unit 11. The depth signal SDP corresponding to any of the above is output.

FIG. 2 is an explanatory diagram of an example of the sunlight spectrum after passing through the atmosphere.
In the above configuration, as shown in FIG. 3, the first ranging light L1 (for example, wavelength λ1 = 850 nm) has a relatively high spectral radiant intensity in the sunlight spectrum. That is, it is a component that is abundant in sunlight.
By the way, the first ranging light L1 has a shorter wavelength than the second ranging light L2 (for example, wavelength λ1 = 940 nm). Therefore, the first ranging light L1 has less scattering and is superior in terms of image quality.

However, the first ranging light L1 cannot be separated from sunlight. Therefore, the distance measurement using the first distance measurement light L1 may reduce the distance measurement accuracy under sunlight.
On the other hand, the second ranging light L2 has a relatively low spectral radiant intensity in the sunlight spectrum. That is, it is a component that is not contained much in sunlight after passing through the atmosphere.
Therefore, the second ranging light L2 can be separated from the sunlight under sunlight and is not easily affected by the sunlight.

However, since the wavelength of the second ranging light L2 is relatively long, blurring or scattering in the lens may become a problem.
Further, even if the wavelengths are different, there is no influence on the distance measurement (influence on the phase difference).

Therefore, for the portion where the influence of sunlight is small (the pixel where the influence of sunlight is small), the in-phase component signal I corresponding to the light receiving signal SL1 corresponding to the first ranging light L1 which has small scattering and is superior in image quality and The depth signal SDP corresponding to the orthogonal component signal Q is output.

Further, the portion affected by sunlight (pixels affected by sunlight) can be separated from sunlight, and the light receiving signal SL2 corresponding to the second ranging light L2, which is not easily affected by sunlight. The depth signal SDP corresponding to the in-phase component signal I and the orthogonal component signal Q corresponding to the above is output.

As a result, according to the present embodiment, under a situation where a portion affected by sunlight and a portion affected by sunlight coexist, for example, indoors by a window or outdoors in the shade ( Depth signal SDP can be stably obtained for the entire image by adaptively selecting each of the scenes in which the area exposed to sunlight and the area not exposed to sunlight are mixed in one image such as (shade of trees). Will be.

Hereinafter, a more specific description will be given.
The signal intensity of the received signal SL1 of the reflected light of the first ranging light L1 is a correct value indoors, which is not affected by sunlight.
Further, in outdoors affected by sunlight, offset due to sunlight is included, and the signal intensity of the received signal SL1 of the reflected light of the first ranging light L1 becomes a large value.

On the other hand, the signal intensity of the received signal SL2 of the reflected light of the second ranging light L2 is not affected by sunlight both indoors and outdoors, and is a correct value.
Here, the signal strength of the light receiving signal SL1 is AL1, the signal strength of the light receiving signal SL2 is AL2, and the signal strength difference ΔA expressed by the following equation is obtained.
ΔA = (AL1-AL2) / (AL1 + AL2)

Then, for a pixel (light receiving position: light receiving element position) in which the signal intensity difference ΔA is larger than a predetermined threshold value, the light receiving signal SL2 of the reflected light of the second ranging light L2 is regarded as having a large influence of sunlight. The depth signal SDP is calculated using this.

Further, for pixels whose signal intensity difference ΔA is smaller than a predetermined threshold value (light receiving position: light receiving element position), the influence of sunlight is considered to be small, and the light receiving signal SL1 of the reflected light of the first ranging light L1 is used. The depth signal SDP is calculated using this.

As a result, the correct depth signal can be calculated at any position where the influence of sunlight is large or the influence of sunlight is small. Therefore, for example, the control of the camera is stable and the image quality of photo processing is improved. It can be improved and obstacles can be detected robustly.

[2] First Embodiment First, the image processing apparatus of the first embodiment will be described.
FIG. 3 is a schematic block diagram of the image processing apparatus of the first embodiment.
The image processing device 20 measures the distance measuring light L including the light having a wavelength λ1 = 850 nm as the first distance measuring light L1 and the light having a wavelength λ2 = 940 nm as the second distance measuring light L2. It has an irradiation unit 21 that irradiates the OBJ and a plurality of light receiving sensors, receives the reflected light of the first ranging light L1 and the reflected light of the second ranging light L2, and receives the light receiving signal SL1 or the light receiving signal SL1 for each pixel. It includes a light receiving unit 22 that outputs a light receiving signal SL2, and a signal processing unit 23 that controls the irradiation unit 21 and generates a depth signal SDP based on the input light receiving signal SL1 or the light receiving signal SL2.

The irradiation unit 21 includes, for example, an infrared LED having a peak emission wavelength in the vicinity of a wavelength of λ1 = 850 nm and capable of pulse wave height, and an infrared LED having a peak emission wavelength in the vicinity of a wavelength of λ2 = 940 nm. You can do it like this. Alternatively, it is also possible to use an infrared light source capable of pulse emission including both the wavelength λ1 = 850 nm and the wavelength λ2 = 940 nm.

FIG. 4 is an explanatory diagram of the first aspect of the light receiving unit.
The light receiving unit 22 includes a light receiving lens 22A, a beam splitter (half mirror) 22B, a first TOF (Times of Flight) sensor 22D-1, and a second TOF sensor 22D-2.
The light receiving lens 22A collects the light received.
The beam splitter 22B divides the light received through the light receiving lens 22A into two systems.
The first TOF sensor 22D-1 is provided with a band-limited filter 22C-1 on the light receiving surface so as to transmit light having a wavelength of λ1 = 850 nm among one of the lights divided by the beam splitter 22B. Then, the first TOF sensor 22D-1 receives light through the filter 22C-1 and outputs a light receiving signal SL1.
The second TOF sensor 22D-2 is provided with a band-limited filter 22C-2 on the light receiving surface so as to transmit light having a wavelength of λ2 = 940 nm among the other light separated by the beam splitter 22B. Then, the second TOF sensor 22D-2 receives light through the filter 22C-2 and outputs a light receiving signal SL2.
Here, the first TOF sensor 22D-1 and the second TOF sensor 22D-2 include a two-dimensional image pickup element in which a light receiving element (pixel cell) is two-dimensionally arranged.
A plurality of light receiving signals SL1 and light receiving signals SL2 are output in pixel (pixel cell) units.

FIG. 5 is an explanatory view of a second aspect of the light receiving unit.
In FIG. 5, the case of 4 × 4 pixels is shown for easy understanding and simplification of the illustration.
The light receiving unit 22 includes a plurality of pixel cells C1 band-limited so as to receive light having a wavelength λ1 = 850 nm, and a plurality of pixel cells C2 band-limited so as to receive light having a wavelength λ2 = 940 nm. The TOF sensor 25 is provided with.

In the TOF sensor 25, the lens array unit 25A in which the lens LS is arranged corresponding to each pixel cell C1 and C2, the filter FL1 corresponding to the pixel cell C1, and the filter FL2 corresponding to the pixel cell C2 are alternately arranged. It includes a filter unit 25B and a light receiving unit 25C in which a light receiving cell PC is arranged so as to correspond to the filter FL1 and the filter FL2.

FIG. 6 is an explanatory diagram of pixel interpolation.
In this configuration, the TOF sensor 25 further includes a signal interpolation unit, and for the light receiving cell PC corresponding to the pixel cell C1, the light receiving cell PC corresponding to the pixel cell C1 located around the light receiving cell corresponding to the pixel cell C2. If the pixel cell C1 (virtual pixel cell C1) is located in the pixel cell C2 based on the output of, the output of the light receiving cell of the pixel cell C1 may be interpolated. .. The same applies to the interpolation of the pixel cell C2.

Specifically, for example, when interpolating the pixel cell C1, in FIG. 6, four pixel cells C1 are adjacent to each other around the light receiving cell PC corresponding to the cross-hatched pixel cell C2. Therefore, the output of the virtual pixel cell C1 corresponding to the arrangement position of the pixel cell C2 is equal to the average value of the outputs of the four pixel cells C1 located adjacent to the light receiving cell PC. Just do it.

When the virtual pixel cell C1 is located at the four corners of the TOF sensor 25, it may be the average value of the outputs of the two adjacent pixel cells C1, and the virtual pixel cell C1 is the TOF sensor 25. When it is located on the peripheral edge excluding the four corners, it may be the average value of the outputs of the three adjacent pixel cells C1.

The same applies when 2 to 4 pixel cells C2 are adjacent to each other around the light receiving cell corresponding to the pixel cell C2.

If the TOF sensor 25 does not have a signal interpolation unit, the signal processing unit 23 may perform the same processing.

FIG. 7 is an explanatory diagram of an example of a functional block of the signal processing unit of the first embodiment.
The signal processing unit 23 is roughly classified into a first RAW image storage unit 30-1, a second RAW image storage unit 30-2, a first reflection intensity calculation unit 31-1, a second reflection intensity calculation unit 31-2, and an intensity signal. It includes a difference calculation unit 32, a selection signal generation unit 33, a first I_Q signal calculation unit 34-1, a second I_Q signal calculation unit 34-2, a selection unit 35, and a depth conversion unit 36.

The first reflection intensity calculation unit 31-1 receives RAW image data RAW850 corresponding to light having a wavelength λ1 = 850 nm as the first ranging light L1 from the light receiving unit 22. Then, the first reflection intensity calculation unit 31-1 calculates the reflection intensity signal C850 for each pixel and outputs it to the intensity signal difference calculation unit 32.

The second reflection intensity calculation unit 31-2 receives RAW image data RAW940 corresponding to light having a wavelength of λ2 = 940 nm as the second ranging light L2 from the light receiving unit 22. Then, the second reflection intensity calculation unit 31-2 calculates the reflection intensity signal C940 for each pixel and outputs it to the intensity signal difference calculation unit 32.

The intensity signal difference calculation unit 32 calculates the difference P between the input reflection intensity signal C850 and the reflection intensity signal C940 by the following equation and outputs it as a selection signal sel to the selection unit 35.
P = (C850-C940) / (C850 + C940)

The first I_Q signal calculation unit 34-1 calculates the first orthogonal component signal Q850 corresponding to the first in-phase component signal I850 and the first in-phase component signal I850 for each pixel based on the RAW image data RAW850, and the selection unit 35. Output to.

The second I_Q signal calculation unit 34-2 calculates the second orthogonal component signal Q940 corresponding to the second in-phase component signal I940 and the second in-phase component signal I940 for each pixel based on the RAW image data RAW940, and the selection unit 35. Output to.

The selection signal generation unit 33 compares the difference P with the predetermined threshold value th, and determines the difference P.
P ≤ th
In the case of, the selection signal sel for selecting the first common mode component signal I850 and the first orthogonal component signal Q850 is output to the selection unit 35, assuming that the influence of sunlight is small.

Further, the selection signal generation unit 33 compares the difference P with the predetermined threshold value th, and determines the difference P.
P> th
In this case, the selection signal sel for selecting the second common mode component signal I940 and the second orthogonal component signal Q940, which are less affected by sunlight, is output to the selection unit 35, assuming that the influence of sunlight is large.

In this case, the predetermined threshold value th is a value sufficiently larger than the difference P that may occur when the influence of sunlight is small, and may occur when the influence of sunlight is large. It is set to a value sufficiently smaller than the difference P.

The selection unit 35 depths either the combination of the first in-phase component signal I850 and the first orthogonal component signal Q850 or the combination of the second in-phase component signal I940 and the second orthogonal component signal Q940 based on the selection signal sel. Output to the conversion unit 36.

As a result, the depth conversion unit 36 is based on either the combination of the first in-phase component signal I850 and the first orthogonal component signal Q850 or the combination of the second in-phase component signal I940 and the second orthogonal component signal Q940. The depth signal SDP corresponding to the distance to the object OBJ is calculated and output for each pixel.

Next, the processing operation of the first embodiment will be described.
FIG. 8 is an overall processing flowchart of the embodiment.
When the process is started, the object OBJ is imaged, and the process of measuring the depth (distance to the object OBJ) (step S11) and the visible image acquisition process (step S12) are performed in parallel.
Then, an image processing process (step S13) for generating a depth image by applying the measured depth to the visible image is performed, and the process is completed.

Next, the depth measurement process will be described in detail.
FIG. 9 is a processing flowchart of the depth measurement process of the first embodiment.
The irradiation unit 21 of the image processing apparatus 20 measures the depth of the distance measuring light L including the light having a wavelength λ1 = 850 nm as the first distance measuring light L1 and the light having a wavelength λ2 = 940 nm as the second distance measuring light L2. When the object OBJ of the target (distance measuring target) is irradiated, the light receiving unit 22 receives the reflected light of the first ranging light L1 and the reflected light of the second ranging light L2. Then, the light receiving unit 22 generates RAW image data RAW850 and RAW image data RAW940 and outputs them to the signal processing unit 23.

As a result, the signal processing unit 23 acquires the RAW image data RAW850 (step S21) and outputs it to the first reflection intensity calculation unit 31-1 and the first I_Q signal calculation unit 34-1.
Similarly, the signal processing unit 23 acquires the RAW image data RAW940 (step S22) and outputs it to the second reflection intensity calculation unit 31-2 and the second I_Q signal calculation unit 34-2.

In these cases, the signal processing unit 23 stores the acquired RAW image data RAW850 and RAW image data RAW940 in a work memory (not shown).

As a result, the first reflection intensity calculation unit 31-1 calculates the reflection intensity signal C850 for each pixel based on the acquired RAW image data RAW850 and outputs it to the intensity signal difference calculation unit 32 (step S23).

Further, the second reflection intensity calculation unit 31-2 calculates the reflection intensity signal C940 for each pixel based on the acquired RAW image data RAW940 and outputs it to the intensity signal difference calculation unit 32 (step S24).

As a result, the intensity signal difference calculation unit 32 calculates the difference P between the input reflection intensity signal C850 and the reflection intensity signal C940 by the following equation and outputs it to the selection signal generation unit 33 (step S25).
P = (C850-C940) / (C850 + C940)

On the other hand, the first I_Q signal calculation unit 34-1 calculates and selects the first orthogonal component signal Q850 corresponding to the first in-phase component signal I850 and the first in-phase component signal I850 for each pixel based on the RAW image data RAW850. Output to unit 35 (step S26).

Similarly, the second I_Q signal calculation unit 34-2 calculates and selects the second orthogonal component signal Q940 corresponding to the second in-phase component signal I940 and the second in-phase component signal I940 for each pixel based on the RAW image data RAW940. Output to unit 35 (step S27).

The selection signal generation unit 33 compares the difference P with the predetermined threshold value th, and determines the difference P.
P ≤ th
In the case of, the selection signal sel for selecting the first common mode component signal I850 and the first orthogonal component signal Q850 is output to the selection unit 35, assuming that the influence of sunlight is small.
P> th
In the case of, the selection signal sel for selecting the second common mode component signal I940 and the second orthogonal component signal Q940, which are less affected by sunlight, is output to the selection unit 35 (assuming that the influence of sunlight is large). Step S28).

The selection unit 35 depths either the combination of the first in-phase component signal I850 and the first orthogonal component signal Q850 or the combination of the second in-phase component signal I940 and the second orthogonal component signal Q940 based on the selection signal sel. Output to the conversion unit 36 (step S29).

As a result, the depth conversion unit 36 is based on either the combination of the first in-phase component signal I850 and the first orthogonal component signal Q850 or the combination of the second in-phase component signal I940 and the second orthogonal component signal Q940. The depth signal DP corresponding to the distance to the object OBJ is calculated and output for each pixel (step S30).

As described above, according to the first embodiment, it is determined whether or not each pixel is affected by sunlight, and when it is determined that the pixel is less affected by sunlight, it is scattered. The depth signal SDP is calculated based on the first in-phase component signal I850 and the first orthogonal component signal Q850 corresponding to the first ranging light L1 having a first wavelength λ1. Further, when it is determined that the pixel is greatly affected by sunlight, the second in-phase component signal I940 and the second in-phase component signal I940 corresponding to the second ranging light L2 having the second wavelength λ2, which is less affected by sunlight, are used. 2 The depth signal DP is calculated based on the orthogonal component signal Q940. Therefore, the obtained depth image is highly accurate.

[3] Second Embodiment In the above first embodiment, the first in-phase component signal I850 and the first orthogonal component signal Q850 corresponding to the first ranging light L1 of the first wavelength λ1 based on the intensity signal difference. Alternatively, the depth signal SDP was generated by using either the second in-phase component signal I940 or the second orthogonal component signal Q940 corresponding to the second ranging light L2 having the second wavelength λ2. On the other hand, in the second embodiment, the first in-phase component signal I850 and the first orthogonal component signal Q850 corresponding to the first ranging light L1 of the first wavelength λ1 and the second ranging of the second wavelength λ2 are used. In the embodiment in the case where the depth signal is calculated for each of the second in-phase component signal I940 and the second orthogonal component signal Q940 corresponding to the optical L2, and then any depth signal is selected as the depth signal SDP based on the region determination feature amount. is there.

FIG. 10 is an explanatory diagram of an example of a functional block of the signal processing unit of the second embodiment.
In this case, since the overall configuration is the same as that of the first embodiment, it will be described with reference to FIG.

The signal processing unit 23 in the second embodiment is roughly classified into the first RAW image storage unit 40-1, the second RAW image storage unit 40-2, the first I_Q signal calculation unit 41-1, and the second I_Q signal calculation unit 41-2. , 1st depth conversion unit 42-1, 2nd depth conversion unit 42-2, 1st area determination feature amount calculation unit 43-1, 2nd area determination feature amount calculation unit 43-2, comparison unit 44 and selection unit 45 It has.

The 1st I_Q signal calculation unit 41-1 receives RAW image data RAW850 corresponding to light having a wavelength λ1 = 850 nm as the first ranging light L1 from the light receiving unit 22, and is input to each pixel based on the RAW image data RAW850. The first orthogonal component signal Q850 corresponding to the first in-phase component signal I850 and the first in-phase component signal I850 is calculated and output to the first depth conversion unit 42-1.

The second I_Q signal calculation unit 41-2 receives RAW image data RAW940 corresponding to light having a wavelength λ2 = 940 nm as the second ranging light L2 from the light receiving unit 22, and is input to each pixel based on the RAW image data RAW940. The second orthogonal component signal Q940 corresponding to the second in-phase component signal I940 and the second in-phase component signal I940 is calculated and output to the second depth conversion unit 42-2.

The first depth conversion unit 42-1 calculates the first depth signal DP1 corresponding to the distance to the object OBJ for each pixel based on the combination of the first in-phase component signal I850 and the first orthogonal component signal Q850, and selects the unit. Output to 45.

The second depth conversion unit 42-2 calculates for each second depth signal DP2 pixel corresponding to the distance to the object OBJ based on the combination of the second in-phase component signal I940 and the second orthogonal component signal Q940, and selects the selection unit 45. Output to.

On the other hand, the first region determination feature amount calculation unit 43-1 is based on the first depth image corresponding to the first depth signal DP1, and the edge depiction degree ED1 (edge degree: signal is buried in noise) in the first depth image. The SN ratio σ1 of the flat portion of the first depth image is calculated and output to the comparison unit 44.

Similarly, the second region determination feature amount calculation unit 43-2 is based on the second depth image corresponding to the second depth signal DP2, and the edge depiction degree ED2 (edge degree: signal is buried in noise) in the second depth image. The SN ratio σ2 of the flat portion of the second depth image is calculated and output to the comparison unit 44.

The comparison unit 44 determines the reliability of the edge degree in the first depth image, the SN ratio σ of the flat portion of the first depth image GDP1, the edge degree in the second depth image GDP2, and the SN ratio σ of the flat portion of the second depth image, respectively. As a result, a selection signal sel for selecting a more reliable depth image from the first depth image GDP1 and the second depth image GDP2 is output to the selection unit 45.

As a result of these, the selection unit 45 outputs either the first depth signal DP1 or the second depth signal DP2 as the depth signal DP for each pixel based on the selection signal sel.

Next, the processing operation of the second embodiment will be described.
Also in the second embodiment, the overall flow of processing is the same as that in the first embodiment, and therefore the detailed description thereof will be incorporated.

Next, the depth measurement process of the second embodiment will be described in detail.
FIG. 11 is a processing flowchart of the depth measurement process of the second embodiment.
The irradiation unit 21 of the image processing apparatus 20 measures the depth of the distance measuring light L including the light having a wavelength λ1 = 850 nm as the first distance measuring light L1 and the light having a wavelength λ2 = 940 nm as the second distance measuring light L2. When the object OBJ of the target (distance measuring target) is irradiated, the light receiving unit 22 receives the reflected light of the first ranging light L1 and the reflected light of the second ranging light L2, and the RAW image data RAW850 The RAW image data RAW940 is generated, the RAW image data RAW850 is output to the first I_Q signal calculation unit 41-1, and the RAW image data RAW940 is output to the second I_Q signal calculation unit 41-2.

As a result, the first RAW image storage unit 40-1 of the signal processing unit 23 acquires the RAW image data RAW850 (step S41).
The first I_Q signal calculation unit 41-1 has a first orthogonality corresponding to the first in-phase component signal I850 and the first in-phase component signal I850 for each pixel based on the RAW image data RAW850 read from the first RAW image storage unit 40-1. The component signal Q850 is calculated and output to the first depth conversion unit 42-1 (step S42).

The first depth conversion unit 42-1 calculates the first depth signal SDP1 corresponding to the distance to the object OBJ based on the combination of the first in-phase component signal I850 and the first orthogonal component signal Q850 for each pixel. It is output to the area determination feature amount calculation unit 43-1 and the selection unit 45 (step S43).

Subsequently, the first region determination feature amount calculation unit 43-1 calculates the reliability (step S44).
FIG. 12 is a processing flowchart of the reliability calculation process.
The first region determination feature amount calculation unit 43-1 is based on the first depth image corresponding to the first depth signal DP1, and the edge depiction degree (edge extraction degree: the signal is buried in noise) in the first depth image. The degree) is calculated (step S51), and the reliability E with respect to the edge extraction degree is calculated (step S52).

In parallel with this, the first region determination feature amount calculation unit 43-1 calculates the variance σ of the SN ratio of the flat portion of the first depth image (step S53), and calculates the reliability F with respect to the variance σ (step). S54).

Subsequently, the first region determination feature amount calculation unit 43-1 integrates both reliability E and F and outputs the integrated reliability RL1 to the comparison unit 44 (step S55).

Similarly, the second RAW image storage unit 40-2 acquires the RAW image data RAW940 (step S45).
As a result, the second I_Q signal calculation unit 41-2 calculates the second orthogonal component signal Q940 corresponding to the second in-phase component signal I940 and the second in-phase component signal I940 for each pixel based on the RAW image data RAW940. Output to the 2-depth conversion unit 42-2 (step S46).

The second depth conversion unit 42-2 calculates the second depth signal SDP2 corresponding to the distance to the object OBJ for each pixel based on the combination of the second in-phase component signal I940 and the second orthogonal component signal Q940, and the second It is output to the area determination feature amount calculation unit 43-2 and the selection unit 45 (step S47).
As a result, the second region determination feature amount calculation unit 43-2 calculates the reliability (step S48).

Specifically, the second region determination feature amount calculation unit 43-2 uses the second depth image corresponding to the second depth signal SDP2 to draw an edge in the second depth image (edge extraction degree: the signal is The degree of being buried in noise) is calculated (step S51), and the reliability E with respect to the edge extraction degree is calculated (step S52).

In parallel with this, the variance σ of the SN ratio of the flat portion of the second depth image is calculated (step S53), and the reliability F with respect to the variance σ is calculated (step S54).
Subsequently, both reliability E and F are integrated and the integrated reliability RL2 is output to the comparison unit 44 (step S55).

As a result, the comparison unit 44 is based on the integrated reliability RL1 output by the first area determination feature amount calculation unit 43-1 and the integrated reliability RL2 output by the second area determination feature amount calculation unit 43-2. It is determined which of the integrated reliability RL1 and the integrated reliability RL2 has the higher integrated reliability (step S49).
Then, as a result of the determination, the comparison unit 44 outputs the depth signal SDP1 and the depth signal SDP2 to the selection unit 45 as a selection signal sel for selecting a more reliable depth signal.

As a result, the selection unit 45 outputs either the first depth signal SDP1 or the second depth signal SDP2 as the depth signal SDP for each pixel based on the selection signal sel (step S50).

As described above, according to the second embodiment, it is determined whether or not each pixel is affected by sunlight based on the reliability of the depth image. Then, when it is determined that the pixel is less affected by sunlight, the first in-phase component signal I850 and the first orthogonal component signal Q850 corresponding to the first ranging light L1 having the first wavelength λ1 with less scattering are used. Based on this, the depth signal SDP is calculated. Further, when it is determined that the pixel is greatly affected by sunlight, the second in-phase component signal I940 and the second in-phase component signal I940 corresponding to the second ranging light L2 of the second frequency f2, which is less affected by sunlight, are used. 2 A more reliable depth signal SDP is calculated based on the orthogonal component signal Q940.
Therefore, according to the second embodiment, the obtained depth image has higher accuracy.

[4] Third Embodiment The difference between this third embodiment and each of the above embodiments is that the depth signal SDP is selected based on both the intensity signal difference and the region determination feature amount.
FIG. 13 is an explanatory diagram of an example of a functional block of the signal processing unit of the third embodiment.
In FIG. 13, the same parts as those in FIG. 7 or 10 are designated by the same reference numerals.
The signal processing unit 23 in the third embodiment can be roughly classified into a first RAW image storage unit 40-1, a second RAW image storage unit 40-2, a first I_Q signal calculation unit 41-1, and a second I_Q signal calculation unit 41-2. , 1st depth conversion unit 42-1, 2nd depth conversion unit 42-2, 1st area determination feature amount calculation unit 43-1, 2nd area determination feature amount calculation unit 43-2, 1st reflection intensity calculation unit 31 -1, the second reflection intensity calculation unit 31-2, the intensity signal difference calculation unit 32, the selection signal generation unit 51, and the selection unit 45 are provided.

Here, the operation of the third embodiment will be described with reference to FIGS. 9 and 11 again.
The irradiation unit 21 of the image processing apparatus 20 measures the depth of the distance measuring light L including the light having a wavelength λ1 = 850 nm as the first distance measuring light L1 and the light having a wavelength λ2 = 940 nm as the second distance measuring light L2. When the object OBJ of the target (distance measuring target) is irradiated, the light receiving unit 22 receives the reflected light of the first ranging light L1 and the reflected light of the second ranging light L2, and the RAW image data RAW850 The RAW image data RAW940 is generated, the RAW image data RAW850 is output to the first I_Q signal calculation unit 41-1, and the RAW image data RAW940 is output to the second I_Q signal calculation unit 41-2.

As a result, the first RAW image storage unit 40-1 of the signal processing unit 23 acquires the RAW image data RAW850.
The first I_Q signal calculation unit 41-1 has a first orthogonality corresponding to the first in-phase component signal I850 and the first in-phase component signal I850 for each pixel based on the RAW image data RAW850 read from the first RAW image storage unit 40-1. The component signal Q850 is calculated and output to the first depth conversion unit 42-1.

As a result, the first depth conversion unit 42-1 calculates the first depth signal SDP1 corresponding to the distance to the object OBJ for each pixel based on the combination of the first in-phase component signal I850 and the first orthogonal component signal Q850. The output is output to the first region determination feature amount calculation unit 43-1 and the selection unit 45.

Subsequently, the first region determination feature amount calculation unit 43-1 calculates the reliability and outputs the integrated reliability RL1 to the selection signal generation unit 51.

Similarly, the second RAW image storage unit 40-2 acquires the RAW image data RAW940 (step S45).
As a result, the second I_Q signal calculation unit 41-2 calculates the second orthogonal component signal Q940 corresponding to the second in-phase component signal I940 and the second in-phase component signal I940 for each pixel based on the RAW image data RAW940. Output to the 2-depth conversion unit 42-2 (step S46).

The second depth conversion unit 42-2 calculates the second depth signal SDP2 corresponding to the distance to the object OBJ for each pixel based on the combination of the second in-phase component signal I940 and the second orthogonal component signal Q940, and the second It is output to the area determination feature amount calculation unit 43-2 and the selection unit 45 (step S47).
As a result, the second region determination feature amount calculation unit 43-2 calculates the reliability and outputs the integrated reliability RL2 to the selection signal generation unit 51.

On the other hand, the first reflection intensity calculation unit 31-1 inputs RAW image data RAW850 corresponding to light having a wavelength λ1 = 850 nm as the first ranging light L1 from the light receiving unit 22. As a result, the first reflection intensity calculation unit 31-1 calculates the reflection intensity signal C850 for each pixel and outputs it to the intensity signal difference calculation unit 32.

The second reflection intensity calculation unit 31-2 receives RAW image data RAW940 corresponding to light having a wavelength of λ2 = 940 nm as the second ranging light L2 from the light receiving unit 22. As a result, the second reflection intensity calculation unit 31-2 calculates the reflection intensity signal C940 for each pixel and outputs it to the intensity signal difference calculation unit 32.

The intensity signal difference calculation unit 32 calculates the difference P between the input reflection intensity signal C850 and the reflection intensity signal C940 by the following equation and outputs it to the selection signal generation unit 51.
P = (C850-C940) / (C850 + C940)

As a result, the comparison unit 44 sets the integrated reliability RL1 output by the first area determination feature amount calculation unit 43-1, the integrated reliability RL2 output by the second area determination feature amount calculation unit 43-2, and the difference P. Based on this, it is determined which of the depth signal SDP1 and the depth signal SDP2 should be selected, and the selection signal sel is output to the selection unit 45.

In this case, if the comparison result of the integrated reliability RL1 and the integrated reliability RL2 and the selection result of the difference P are the same, the selection result is adopted.
When the difference between the integrated reliability RL1 and the integrated reliability RL2 is large and the difference P is small, the comparison result of the integrated reliability RL1 and the integrated reliability RL2 is adopted as the selection result.

Further, when the difference between the integrated reliability RL1 and the integrated reliability RL2 is small and the difference P is large, the selection result based on the difference P is adopted.
Further, when the difference between the integrated reliability RL1 and the integrated reliability RL2 is small and the difference P is also small, it is considered that the result is not so different regardless of which one is selected, so it is arbitrarily selected (for example, the depth signal SDP1 is depthed in advance). Set to select as signal SDP).

As a result of these, the selection unit 45 outputs either the first depth signal SDP1 or the second depth signal SDP2 as the depth signal SDP for each pixel based on the selection signal sel.

As described above, according to the third embodiment, whether or not each pixel is affected by sunlight is more reliable based on the reliability of the depth image and / or the intensity signal difference. Since the high depth signal SDP is adopted, the obtained depth image becomes more accurate.

[5] Fourth Embodiment In each of the above embodiments, as the range-finding light L, the first range-finding light L1 (for example, wavelength 850 nm), which is a component contained in a large amount in sunlight, and the sunlight spectrum are spectroscopic. The case where the second ranging light L2 (for example, wavelength 940 nm), which is a component having a relatively low radiation intensity, that is, which is not so contained in sunlight, is used has been described. On the other hand, in the fourth embodiment, the first ranging light L1 in the frequency band having high reflectance and the second ranging light L2 in the frequency band having low reflectance, which are specific to the type of the object to be measured, are used. It is an embodiment when it is used. That is, the reflectance of the first ranging light L1 (reflectance of the distance measuring object at the first wavelength) is the reflectance of the second ranging light L2 (reflectance of the ranging object at the second wavelength). ), This is an embodiment in which the first ranging light L1 and the second ranging light L2 are set so as to be significantly higher than.

First, the principle of the fourth embodiment will be described.
It is known that the intensity of reflection in each wavelength band differs depending on the type of substance to be measured (for example, plant, soil, water, etc.).

That is, once the substance to be measured is determined, the frequency band having high reflectance and the frequency band having low reflectance are determined.
Therefore, the object to be measured exists in a region where the difference between the reflected signal in the frequency band having high reflectance and the reflected signal in the frequency band having low reflectance is large depending on the substance of the object to be measured. Can be regarded as.

FIG. 14 is an explanatory diagram of the fourth embodiment.
FIG. 14 shows the relationship between the wavelength and the reflection intensity on plants, soil, water, the ground surface, and the water surface as measurement objects.
For example, as a first example, when a plant is an object to be measured, a reflection signal near 800 nm (corresponding to the reflectance of the first ranging light L1) having a high reflectance of the plant and the reflection of the plant. When the difference of the reflected signal (corresponding to the reflectance of the second ranging light L2) near 500 nm, which has a low reflectance, if the difference is large, it can be determined that a plant exists in the region. It will be possible.

Therefore, when the plant is the object to be measured, for pixels with a large difference in the reflected signal, the distance (depth) to the plant can be stabilized by selecting the reflected signal near 800 nm, which has a high reflectance of the plant. You can get it.

Further, as a second example, when the water surface is an object to be measured, a reflected signal near 400 nm (corresponding to the reflectance of the first distance measuring light L1) having a high reflectance of water and a reflection of water. When the difference of the reflected signal (corresponding to the reflectance of the second ranging light L2) near 800 nm, which has a low reflectance, is taken, if the difference is large, it can be determined that water exists in the region. It will be possible.

Therefore, when water is the object to be measured, for pixels with a large difference in reflected signals, the distance (depth) to the water surface can be stabilized by selecting a reflected signal near 400 nm, which has a high reflectance of water. You can get it.

As described above, by using the first distance measuring light L1 having a high reflectance of the measurement object and the second distance measuring light L2 having a low reflectance of the measurement object, the existence of the measurement object is present. It is possible to accurately determine the presence / absence and the distance (depth) to the measurement object if it exists.

[6] Modifications of the Embodiment In the above-mentioned first to third embodiments, the distance measuring light L is the first distance measuring light L1 (for example, a wavelength of 850 nm) which is a component that is abundantly contained in sunlight. ) And the case where the second ranging light L2 (for example, wavelength 940 nm), which is a component not so much contained in sunlight, is used. However, the wavelength is not limited to this, and any combination of wavelengths that can eliminate the influence of sunlight can be appropriately applied.

Note that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

In addition, this technology can also adopt the following configurations.
(1)
The first ranging at the same ranging position based on the received signal of the reflected light of the first ranging light of the first wavelength and the received signal of the reflected light of the second ranging light of the second wavelength. A difference calculation unit that calculates the difference between the reflection intensity of the light for distance measurement and the reflection intensity of the second distance measurement light.
Based on the difference, the in-phase component signal corresponding to either the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light and the said An output unit that outputs a depth signal corresponding to the orthogonal component signal,
Image processing device equipped with.
(2)
The output unit is a selection unit that selects and outputs either a received signal of the reflected light of the first ranging light or a received signal of the reflected light of the second ranging light based on the difference.
A calculation unit that calculates an in-phase component signal and an orthogonal component signal from the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light selected by the selection unit.
Based on the calculation result of the calculation unit, the in-phase component signal corresponding to either the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light and the orthogonality A conversion unit that performs depth conversion on the component signal and outputs the depth signal,
The image processing apparatus according to (1).
(3)
The output unit includes a calculation unit that calculates an in-phase component signal and an orthogonal component signal from the received signal of the reflected light of the first ranging light and the received signal of the reflected light of the second ranging light, respectively.
Depth conversion is performed on the in-phase component signal and the orthogonal component signal calculated from the received signal of the reflected light of the first range-finding light, and the first depth signal as the depth signal and the reflected light of the second range-finding light are performed. A conversion unit that performs depth conversion on the in-phase component signal and the orthogonal component signal calculated from the received signal of the above and outputs the second depth signal as the depth signal.
A selection unit that selects and outputs either the first depth signal or the second depth signal based on the difference.
The image processing apparatus according to (1).
(4)
The first ranging light has a relatively high spectral radiant intensity in the sunlight spectrum, the second ranging light has a relatively low spectral radiant intensity, and the second wavelength has a second wavelength. Longer than 1 wavelength,
The image processing apparatus according to any one of (1) to (3).
(5)
The first wavelength has a center wavelength of 850 nm, and the second wavelength has a center wavelength of 940 nm.
(4) The image processing apparatus according to the above.
(6)
The first wavelength and the second wavelength are set so that the reflectance of the distance measuring object at the first wavelength is significantly higher than the reflectance at the second wavelength.
The image processing apparatus according to any one of (1) to (4).
(7)
A method performed by an image processor
The first ranging at the same ranging position based on the received signal of the reflected light of the first ranging light of the first wavelength and the received signal of the reflected light of the second ranging light of the second wavelength. The process of calculating the difference between the reflection intensity of the light for distance measurement and the reflection intensity of the second ranging light, and
Based on the difference, the in-phase component signal corresponding to either the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light and the said The process of outputting the depth signal corresponding to the orthogonal component signal and
A method equipped with.
(8)
An irradiation unit that irradiates the first ranging light of the first wavelength and the second ranging light of the second wavelength,
The reflected light of the first ranging light and the reflected light of the second ranging light are received, and the received signal of the reflected light of the first ranging light and the reflected light of the second ranging light are received. The image pickup unit that outputs the received signal of
Based on the received signal of the reflected light of the first ranging light and the received signal of the reflected light of the second ranging light, the reflection intensity of the first ranging light at the same ranging position and the said A difference calculation unit that calculates the difference between the reflection intensity of the second distance measuring light and
Based on the difference, the in-phase component signal corresponding to either the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light and the said An output unit that outputs a depth signal corresponding to the orthogonal component signal,
Electronic equipment equipped with.

10 Image processing device 11 Difference calculation unit 12 Output unit 20 Image processing device 21 Irradiation unit 22 Light receiving unit 22A Light receiving lens 22B Beam splitter 22C-1, 22C-2 Filter 22D-1 1st TOF sensor 22D-2 2nd TOF sensor 23 Signal processing Part 25 TOF sensor 25A Lens array unit 25B Filter unit 25C Light receiving unit 30-1 1st RAW image storage unit 30-2 2nd RAW image storage unit 31-1 1st reflection intensity calculation unit 31-2 2nd reflection intensity calculation unit 32 Strength Signal difference calculation unit 33 Selection signal generation unit 34-1, 41-1 1st I_Q signal calculation unit 34-2, 41-2 2nd I_Q signal calculation unit 35 Selection unit 36 Depth conversion unit 40-1 1st RAW image storage unit 40- 2 2nd RAW image storage unit 42-1 1st depth conversion unit 42-2 2nd depth conversion unit 43-1 1st area judgment feature amount calculation unit 4-3 2nd area judgment feature amount calculation unit 44 Comparison unit 45 Selection unit 51 Selection signal generator C1 Pixel cell C2 Pixel cell C850 Reflection intensity signal C940 Reflection intensity signal d Difference DP1 First depth signal DP2 Second depth signal E, F Reliability GDP1 First depth image GDP2 Second depth image I1, I2 In-phase Component signal L1 1st ranging light L2 2nd ranging light I850 1st in-phase component signal I940 2nd in-phase component signal L ranging light LS lens Q850 1st orthogonal component signal Q940 2nd orthogonal component signal RAW850 RAW image Data RAW940 RAW image data OBJ Object P difference PC light receiving cell Q Orthogonal component signal Q1 Orthogonal component signal Q2 Orthogonal component signal RL1 Integrated reliability RL2 Integrated reliability SDP depth signal SDP1 1st depth signal SDP2 2nd depth signal sel selection signal SL1 Signal SL2 Received signal th Threshold λ1 First wavelength λ2 Second wavelength σ1, σ2 SN ratio

Claims (8)

1. The first ranging at the same ranging position based on the received signal of the reflected light of the first ranging light of the first wavelength and the received signal of the reflected light of the second ranging light of the second wavelength. A difference calculation unit that calculates the difference between the reflection intensity of the light for distance measurement and the reflection intensity of the second distance measurement light.
   Based on the difference, the in-phase component signal and the orthogonal component corresponding to either the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light for each distance measuring position. An output unit that outputs a depth signal based on the signal,
   Image processing device equipped with.
2. The output unit is a selection unit that selects and outputs either a received signal of the reflected light of the first ranging light or a received signal of the reflected light of the second ranging light based on the difference.
   A calculation unit that calculates an in-phase component signal and an orthogonal component signal from the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light selected by the selection unit.
   Based on the calculation result of the calculation unit, the in-phase component signal corresponding to either the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light and the orthogonality A conversion unit that performs depth conversion on the component signal and outputs the depth signal,
   The image processing apparatus according to claim 1.
3. The output unit includes a calculation unit that calculates an in-phase component signal and an orthogonal component signal from the received signal of the reflected light of the first ranging light and the received signal of the reflected light of the second ranging light, respectively.
   Depth conversion is performed on the in-phase component signal and the orthogonal component signal calculated from the received signal of the reflected light of the first range-finding light, and the first depth signal as the depth signal and the reflected light of the second range-finding light are performed. A conversion unit that performs depth conversion on the in-phase component signal and the orthogonal component signal calculated from the received signal of the above and outputs the second depth signal as the depth signal.
   A selection unit that selects and outputs either the first depth signal or the second depth signal based on the difference.
   The image processing apparatus according to claim 1.
4. The first ranging light has a relatively high spectral radiant intensity in the sunlight spectrum, the second ranging light has a relatively low spectral radiant intensity, and the second wavelength has a second wavelength. Longer than 1 wavelength,
   The image processing apparatus according to claim 1.
5. The first wavelength has a center wavelength of 850 nm, and the second wavelength has a center wavelength of 940 nm.
   The image processing apparatus according to claim 4.
6. The first wavelength and the second wavelength are set so that the reflectance of the distance measuring object at the first wavelength is significantly higher than the reflectance at the second wavelength.
   The image processing apparatus according to claim 1.
7. A method performed by an image processor
   The first ranging at the same ranging position based on the received signal of the reflected light of the first ranging light of the first wavelength and the received signal of the reflected light of the second ranging light of the second wavelength. The process of calculating the difference between the reflection intensity of the light for distance measurement and the reflection intensity of the second ranging light, and
   Based on the difference, the in-phase component signal and the orthogonal component corresponding to either the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light for each distance measuring position. The process of outputting a depth signal based on the signal and
   A method equipped with.
8. An irradiation unit that irradiates the first ranging light of the first wavelength and the second ranging light of the second wavelength,
   The reflected light of the first ranging light and the reflected light of the second ranging light are received, and the received signal of the reflected light of the first ranging light and the reflected light of the second ranging light are received. The image pickup unit that outputs the received signal of
   Based on the received signal of the reflected light of the first ranging light and the received signal of the reflected light of the second ranging light, the reflection intensity of the first ranging light at the same ranging position and the said A difference calculation unit that calculates the difference between the reflection intensity of the second ranging light and
   Based on the difference, the in-phase component signal and the orthogonal component corresponding to either the received signal of the reflected light of the first ranging light or the received signal of the reflected light of the second ranging light for each distance measuring position. An output unit that outputs a depth signal based on the signal,
   Electronic equipment equipped with.

Images