WO2021022696A1

WO2021022696A1 - Image acquisition apparatus and method, electronic device and computer-readable storage medium

Info

Publication number: WO2021022696A1
Application number: PCT/CN2019/116308
Authority: WO
Inventors: 张超; 臧凯
Original assignee: 深圳市灵明光子科技有限公司
Priority date: 2019-08-02
Filing date: 2019-11-07
Publication date: 2021-02-11
Also published as: CN117156114A; CN110493587A; CN110493587B

Abstract

An image acquisition apparatus and method, an electronic device, and a computer-readable storage medium. The image acquisition apparatus comprises an acquisition module, a registration module, a fusion module, and an up-sampling module. The acquisition module acquires scenes so as to obtain a first two-dimensional image, a second two-dimensional image and a three-dimensional image that have different resolutions and different image information. The registration module performs relevant registration on the images among which there are differences, and obtains a mapping relationship between the different images, so that according to said mapping relationship, the fusion module fuses the images that have different resolutions and different image information. Moreover, the resolution of the fused image is further improved by means of the up-sampling module, so that the image acquisition apparatus obtains a high-resolution three-dimensional depth image, and at the same time achieves the fusion of different feature information such as color information, grayscale information and depth information of the images, thereby improving the integrity of image information.

Description

Image acquisition device and method, electronic equipment, computer readable storage medium

Technical field

The present invention relates to the field of imaging technology, in particular to an image acquisition device and method, electronic equipment, and computer-readable storage medium.

Background technique

With the rapid development of digital technology, semiconductor manufacturing technology, and the Internet, the two-dimensional imaging technology of traditional image sensors has become increasingly mature, and the images collected can be in color mode or grayscale mode, and have high resolution. However, the collected image information is incomplete to a certain extent, such as lack of depth information.

With the development of science and technology, more and more industry fields no longer satisfy the plane information provided by the two-dimensional display and require a true reflection of the actual three-dimensional world. Therefore, there are more and more applications of depth sensors that can output both three-dimensional image information and two-dimensional image information. However, compared with the traditional image sensor, the image collected by the depth sensor has low resolution and limited color mode, which cannot meet the needs of users.

Therefore, the exemplary image acquisition method cannot meet the user's needs for high-resolution images and complete image information.

Summary of the invention

Based on this, it is necessary to provide an image acquisition device and method, electronic equipment, and computer-readable storage medium that can simultaneously improve image resolution and image information integrity.

In order to achieve the purpose of the present invention, the present invention adopts the following technical solutions:

An image acquisition device, including:

An acquisition module configured to acquire a first two-dimensional image, a second two-dimensional image, and a three-dimensional image of the scene; the resolution of the first two-dimensional image is higher than the resolution of the second two-dimensional image;

A registration module, connected to the acquisition module, configured to perform image registration on the first two-dimensional image and the second two-dimensional image, and obtain the first two-dimensional image and the second two-dimensional image The mapping relationship;

A fusion module, which connects the acquisition module and the registration module, and is configured to perform image fusion on the three-dimensional image and the first two-dimensional image according to the mapping relationship to obtain a fused image;

The up-sampling module is connected to the fusion module and is configured to up-sample the fused image to obtain a high-resolution three-dimensional depth image.

In one of the embodiments, the collection module includes:

The first collection unit is respectively connected to the registration module and the fusion module, and is configured to collect image light reflected by the scene to form a first two-dimensional image;

The second collection unit is respectively connected to the registration module and the fusion module, and is configured to collect a second two-dimensional image of the scene and output to the registration module, and collect the three-dimensional image of the scene and output to the fusion module .

In one of the embodiments, the second collection unit includes:

The control circuit is configured to control the light source to emit the first pulsed light to the scene;

A detection circuit, connected to the control circuit, configured to receive the second pulsed light reflected back from the scene, and detect the two-dimensional gray value and/or depth information according to the first pulsed light and the second pulsed light;

The readout circuit, the detection circuit, the registration module, and the fusion module, respectively, are configured to read out the second two-dimensional image information according to the two-dimensional gray value, and read out the three-dimensional image according to the depth information information.

In one of the embodiments, the detection circuit includes:

A photodetector, connected to the control circuit, configured to detect the second pulsed light reflected by the scene and generate a trigger signal;

The conversion circuit is connected to the photodetector and the readout circuit, and is configured to detect the two-dimensional gray value according to the trigger signal in the two-dimensional mode, and according to the first pulsed light and the second pulsed light in the three-dimensional mode. Pulse light obtains distance information.

In one of the embodiments, the conversion circuit includes:

The first switch, the second switch, the counter, the oscillator and the decoder;

The static contact of the first switch is connected to the input end of the counter, the first dynamic contact of the first switch, the first end of the oscillator, and the output end of the photodetector are connected in common, so The second dynamic contact of the first switch, the second end of the oscillator, and the first end of the decoder are connected in common, and the second end of the decoder is connected to the static contact of the second switch, The dynamic contact of the second switch and the output end of the counter are commonly connected to the input end of the readout circuit.

In one of the embodiments, the collection module further includes:

The filter unit is arranged between the scene and the first collection unit, and/or between the scene and the second collection unit, and is configured to perform spectral filtering on the scene reflected light.

In one of the embodiments, the collection module further includes:

The dimming unit is arranged between the scene and the first collection unit, and/or between the scene and the second collection unit, and is configured to adjust the light intensity of the scene reflected light.

In one of the embodiments, the collection module further includes:

The switch unit is respectively connected to the second acquisition unit, the registration module, and the fusion module, and is configured to control the connection state of the second acquisition unit and the registration module, and control the second acquisition unit Connection status with the fusion module.

In one of the embodiments, the registration module includes:

An extraction unit, connected to the acquisition module, and configured to respectively extract feature point groups corresponding to the first two-dimensional image and the second two-dimensional image;

The mapping unit is connected to the extraction unit and the fusion module, and is configured to obtain the mapping relationship between the first two-dimensional image and the second two-dimensional image according to a feature point group.

In one of the embodiments, the fusion module includes:

A transformation unit, connected to the acquisition module and the registration module, and configured to acquire a projective transformed image according to the three-dimensional image and the mapping relationship;

The image fusion unit is connected to the transformation unit and the up-sampling module, and is configured to fuse the projection transformed image and the first two-dimensional image to obtain a fused image.

An image acquisition method, including:

Acquiring a first two-dimensional image, a second two-dimensional image, and a three-dimensional image of the scene; the resolution of the first two-dimensional image is higher than the resolution of the second two-dimensional image;

Performing image registration on the first two-dimensional image and the second two-dimensional image, and acquiring a mapping relationship between the first two-dimensional image and the second two-dimensional image;

Performing image fusion on the three-dimensional image and the first two-dimensional image according to the mapping relationship to obtain a fused image;

Up-sampling the fused image to obtain a high-resolution three-dimensional depth image.

An electronic device includes a memory and a processor, and a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the image acquisition method described above.

A computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the steps of the image acquisition method described above are realized.

The above-mentioned image acquisition device and image acquisition method collect a scene to obtain a first two-dimensional image, a second two-dimensional image and a three-dimensional image with different resolutions and different image information, and then perform relevant registration on the different images , Obtain the mapping relationship between different images, merge images with different resolutions and different image information according to the mapping relationship, and further improve the resolution of the fused image through upsampling, so as to obtain a high-resolution three-dimensional depth image, and realize the image The fusion of different characteristic information such as color information, gray information and depth information of the image information improves the integrity of image information.

Description of the drawings

FIG. 1 is a schematic diagram of the structure of an image acquisition device in an embodiment;

2 is a schematic diagram of the structure of a registration module in an embodiment;

Figure 3 is a schematic structural diagram of a fusion module in an embodiment;

FIG. 4 is a schematic diagram of the structure of the acquisition module in an embodiment;

Figure 5 is a schematic diagram of the structure of a second collection unit in an embodiment;

FIG. 6 is a schematic diagram of the structure of a detection circuit in an embodiment;

Fig. 7 is a circuit diagram of a conversion circuit in an embodiment;

8a-8d are schematic diagrams of the structure of the acquisition module in another embodiment;

FIG. 9 is a schematic structural diagram of a collection module in another embodiment;

FIG. 10 is a schematic circuit diagram of a switch unit in an embodiment;

Fig. 11 is a flowchart of an image acquisition method in an embodiment.

detailed description

In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The drawings show preferred embodiments of the present invention. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments, and are not intended to limit the present invention.

Refer to FIG. 1, which is a schematic structural diagram of an image acquisition device in an embodiment.

In this embodiment, the image acquisition device includes an acquisition module 100, a registration module 200, a fusion module 300, and an up-sampling module 400.

The acquisition module 100 is configured to acquire a first two-dimensional image, a second two-dimensional image, and a three-dimensional image of the scene, and the resolution of the first two-dimensional image is higher than the resolution of the second two-dimensional image.

The registration module 200, connected to the acquisition module 100, is configured to perform image registration on the first two-dimensional image and the second two-dimensional image, and obtain the mapping relationship between the first two-dimensional image and the second two-dimensional image.

The fusion module 300 is connected to the acquisition module 100 and the registration module 200, and is configured to perform image fusion between the three-dimensional image and the first two-dimensional image according to the mapping relationship to obtain a fused image.

The up-sampling module 400, connected to the fusion module 300, is configured to up-sample the fused image information to obtain a high-resolution three-dimensional depth image.

In this embodiment, the collection module 100 integrates multiple collection units, and can collect scenes from the same location or different locations in different collection methods, so as to obtain the first two-dimensional image, the second two-dimensional image, and the three-dimensional image. The image of the source channel.

Among them, the scene is the target collection object, which is composed of one or more objects, and the image corresponding to the scene can be obtained by collecting the scene.

Wherein, the first two-dimensional image is an image with only two-dimensional plane information of the scene but no depth direction information; the first two-dimensional image may be an RGB color mode image or a grayscale mode image. The first two-dimensional image has a high resolution. The second two-dimensional image refers to the two-dimensional grayscale image information of the scene, and the resolution is lower than the first two-dimensional image; the three-dimensional image refers to the image with the depth direction information of the scene, and the resolution is lower than the first two-dimensional image. A three-dimensional depth image refers to an image with high-resolution image information and depth direction information. Among them, the first two-dimensional image, the second two-dimensional image, and the three-dimensional image may respectively correspond to multiple images; and the first two-dimensional image, the second two-dimensional image, and the three-dimensional image may simultaneously correspond to the same scene; or the second two-dimensional image The image and a certain first two-dimensional image correspond to the same scene, and the three-dimensional image and another certain first two-dimensional image correspond to the same scene.

Among them, different acquisition methods include traditional image sensing acquisition methods and depth sensing acquisition methods. When collecting the same scene, there is an overlapping area of the collected perspectives, that is, a common area. In the overlapping area, the first two-dimensional image, the second two-dimensional image, and the three-bit depth image have corresponding points, such as corresponding feature points.

Wherein, the collection module 100 can collect the first two-dimensional image, the second two-dimensional image, and the three-dimensional image at different times when collecting different images of the same scene. For example, the first two-dimensional image and the second two-dimensional image can be collected first. After the registration module 200 obtains the mapping relationship, the first two-dimensional image and the three-dimensional image are collected for fusion processing; the first two-dimensional image, the second two-dimensional image, and the three-dimensional image can also be collected at the same time. The module 200 acquires the mapping relationship, and the fusion module 300 performs fusion processing on the first two-dimensional image and the three-dimensional image; or, when the image acquisition device has stored the mapping relationship, the acquisition module 100 may only acquire the first two-dimensional image during subsequent image processing. The first two-dimensional image and the three-dimensional image are then directly fused by the fusion module 300 to perform the fusion processing on the first two-dimensional image and the three-dimensional image, avoiding repeated acquisition of the second two-dimensional image, and reducing the power consumption of the acquired image.

In this embodiment, because the acquisition module 100 integrates multiple acquisition units, the acquired first two-dimensional image and the second two-dimensional image are different due to differences in the acquisition method, location, field of view, and resolution. There is a certain degree of difference between the information between the two, the registration module 200 as the pre-processing module of the fusion module 300 can perform relevant registration on the different images, and obtain the mapping relationship between different images, so that subsequent images are aligned according to the mapping relationship Fusion.

In an embodiment, the registration module 200 can obtain the common area of the first two-dimensional image and the second two-dimensional image through image processing and existing algorithm operations, and extract the corresponding area or correspondence between the two images in the common area. Feature points and establish a coordinate system, and calculate the mapping relationship between images according to the location coordinates of the corresponding area or corresponding feature points. Among them, the existing algorithm operations include Homography matrix (H matrix), interpolation algorithm and the combination of the two. Specifically, referring to FIG. 2 for assistance, the registration module 200 includes an extraction unit 201 and a mapping unit 202.

Wherein, the extraction unit 201 is connected to the acquisition module 100 and is configured to extract feature point groups corresponding to the first two-dimensional image and the second two-dimensional image respectively. Optionally, the characteristic points include edges, contours, intersections on curved surfaces, and points of high curvature. In an embodiment, the extracting unit 201 may extract corresponding feature point groups between different images on the common area of the first two-dimensional image and the second two-dimensional image by using existing algorithms.

Further, the extraction unit 201 can directly extract the respective high-frequency component images in the two common areas, extract the corresponding feature point group from the high-frequency component images, and obtain the mapping relationship, thereby reducing the computational complexity and improving the registration rate. Specifically, the extraction unit 201 includes a filter and a processor. The filter is configured to perform filtering processing on the first two-dimensional image and the second two-dimensional image respectively to obtain the first two-dimensional image high-frequency component and the second two-dimensional image high-frequency component; the processor is connected to the filter and is configured to The feature point groups of the first two-dimensional image high-frequency component and the second two-dimensional image high-frequency component are respectively extracted.

The mapping unit 202 is connected to the extraction unit 201 and the fusion module 300, and is configured to obtain the mapping relationship between the first two-dimensional image and the second two-dimensional image according to the feature point group. In an embodiment, the mapping unit 202 first calculates a discrete coordinate mapping table based on the discrete feature point group, and then generates a complete coordinate mapping table based on the discrete coordinate mapping table and difference calculation to obtain the mapping relationship between images. The mapping unit 202 may be a data processing chip connected to the extraction unit 201.

It should be noted that after the registration module 200 acquires the mapping relationship, the mapping relationship may be stored, so that the subsequent image acquisition device can directly use the mapping relationship for image alignment and fusion when it is used again.

In this embodiment, the fusion module 300 can fuse a plurality of different images collected by the collection module 100 to obtain a fusion image with different resolutions and different image information. Among them, the fusion image includes both three-dimensional image information and two-dimensional image information. Therefore, by fusing the image 300, the resolution of the original captured image can be improved, and at the same time, the color information, gray information, and depth information of the image can be merged to improve the integrity of the image information.

In an embodiment, please refer to FIG. 3 for assistance. The fusion module 300 includes a transformation unit 301 and an image fusion unit 302.

Wherein, the transformation unit 301 is connected to the acquisition module 100 and the registration module 200, and is configured to acquire the projection transformed image according to the three-dimensional image and the mapping relationship. For example, the transform unit 301 transforms the three-dimensional image by using an interpolation algorithm and a mapping relationship to form a projected transformed image. Wherein, the projection transformation image and the three-dimensional image have a common area image, and the position coordinates of each point are the same as the position coordinates of the corresponding points of the three-dimensional image. The transformation unit 301 may be an image processor.

Wherein, the image fusion unit 302 is connected to the transformation unit 301 and the up-sampling module 400, and is configured to fuse the projection transformed image and the first two-dimensional image to obtain a fused image. Optionally, the image fusion unit 302 separately obtains the common areas of the projection transformed image and the three-dimensional image, adds the two common areas and averages them, or divides the areas into different weights for weighted synthesis as needed to obtain the fused image; or Algorithms such as multi-resolution tower image fusion, wavelet transform, and Kalman filtering are used to fuse the projection transformed image and the first two-bit image. The image fusion unit 302 may be an image fusion device, an image processor, a fusion controller, or the like.

In this embodiment, the up-sampling module 400 can up-sample the fused image obtained in the above-mentioned embodiment to further improve the resolution of image display, and generate a high-resolution three-dimensional depth image. Specifically, the up-sampling module 400 may enlarge the fused image through an interpolation algorithm to improve the image enlargement effect. Among them, interpolation algorithms include but are not limited to nearest neighbor interpolation, bilinear interpolation, and cubic interpolation.

It should be noted that, in order to facilitate viewing or evaluating the three-dimensional depth images obtained in the foregoing embodiments, the image acquisition device may also be provided with an image display module to visually display the three-dimensional depth images, or directly add to the fusion module 300 or the up-sampling module 400 The display unit displays the image simultaneously during the fusion or up-sampling process.

It should be noted that the division of each module in the image acquisition device described above is only for illustration. In other embodiments, the image acquisition device can be divided into different modules as needed to complete all or part of the functions of the image acquisition device. .

Each module in the above-mentioned image acquisition device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules. For example, the above-mentioned registration module, fusion module, and up-sampling module are embedded in hardware or independent of the processor in the computer device.

The image acquisition device provided in this embodiment includes an acquisition module, a registration module, a fusion module, and an up-sampling module. The acquisition module collects a scene to obtain a first two-dimensional image and a second two-dimensional image with different resolutions and different image information. Two-dimensional images and three-dimensional images, the registration module performs relevant registration on the images with differences, and obtains the mapping relationship between different images, so that the fusion module will merge images with different resolutions and different image information according to the mapping relationship, and pass The up-sampling module further improves the resolution of the fused image, enabling the image acquisition device to obtain a high-resolution three-dimensional depth image, and at the same time realizes the fusion of different characteristic information such as color information, gray information and depth information of the image, and improves the integrity of image information .

Refer to FIG. 4, which is a schematic diagram of the structure of the acquisition module in an embodiment.

In this embodiment, the collection module 100 includes a first collection unit 101 and a second collection unit 102.

The first collection unit 101 is respectively connected to the registration module 200 and the fusion module 300, and is configured to collect image light reflected by the scene to form a first two-dimensional image.

The second acquisition unit 102 is respectively connected to the registration module 200 and the fusion module 300, and is configured to collect a second two-dimensional image of the scene and output to the registration module, and collect the three-dimensional image of the scene and output to the fusion module.

In an embodiment, the first acquisition unit 101 includes an image sensor, and the output ends of the image sensor are respectively connected to the registration module 200 and the fusion module 300 to convert the image light reflected by the scene into electrical signals. The electrical signals obtained can be directly derived from the image The sensor independently reads and processes data, or can assist other electronic components to read and process data to form the first two-dimensional image. Optionally, the image sensor may be a pn junction diode, a CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor, complementary metal oxide semiconductor) image sensor.

It should be noted that the light source of the image light collected by the first collection unit 101 may be a built-in light source of the unit, an ambient light source or other external light sources. The light source is not limited to visible light RGB or infrared light.

In one embodiment, please refer to FIG. 5 for assistance. The second acquisition unit 102 includes a control circuit 1021, a detection circuit 1022, and a readout circuit 1023.

The control circuit 1021 is configured to control the light source to emit the first pulsed light to the scene.

The detection circuit 1022 is connected to the control circuit 1021 and is configured to receive the second pulsed light reflected from the scene, and detect the two-dimensional gray value and/or depth information according to the first pulsed light and the second pulsed light.

The readout circuit 1023 is respectively connected to the detection circuit 1022, the registration module 200 and the fusion module 300, and is configured to read out the second two-dimensional image information according to the two-dimensional gray value and the three-dimensional image information according to the depth information.

In this embodiment, the control circuit 1021 controls the light source to emit the first pulsed light to the scene according to the start instruction, and at the same time controls the detection circuit 1022 to start work and make the detection circuit 1022 perform detection in a preset manner. Wherein, the first pulsed light may be a continuous pulse of near infrared. The control circuit 1021 controls the emission direction, emission time and emission intensity of the light source. In an embodiment, the control circuit 1021 includes one or more processors and memories, and a light source.

In this embodiment, the detection circuit 1022 receives the second pulsed light emitted back from the scene and converts the optical signal into an electrical signal, thereby recording the emission time of the first pulsed light, the receiving time of the second pulsed light, and the received pulse according to the electrical signal According to the number of photons in the second pulsed light, the two-dimensional gray value of the acquired image is calculated according to the number of photons of the second pulsed light, the time interval is acquired according to the emission time and the reception time, and then the depth information is acquired. It should be noted that the detection circuit 1022 can also obtain a two-dimensional gray value by detecting ambient light.

Optionally, the detection circuit 1022 is configured to have two switchable working modes. In the two-dimensional mode, the two-dimensional gray value of the acquired image is calculated according to the number of photons of the second pulsed light, and in the three-dimensional mode according to the emission time and The receiving time obtains the time interval and obtains the depth information, so as to switch the mode according to actual needs and obtain the corresponding information.

In an embodiment, the detection circuit 1022 includes a photodetector and a conversion circuit.

Specifically, the photodetector is configured to detect the second pulsed light reflected by the scene and generate a trigger signal. The photodetector includes SPAD (Single-Photon Avalanche Diode), or other photodetectors.

Specifically, the conversion circuit is connected to the photodetector, and is configured to detect the two-dimensional gray value according to the trigger signal in the two-dimensional mode, and obtain distance information according to the first pulsed light and the second pulsed light in the three-dimensional mode. Through the conversion circuit, different modes can be selected to obtain gray information or depth information correspondingly, so as to realize the sharing of internal circuits and save circuit costs.

The number of photodetectors can be one or more. When the number of photodetectors is one, the conversion circuit corresponds to one, and the detection circuit scans point by point to detect the two-dimensional gray value or depth information, and obtain the two-dimensional gray value or depth information of the plane after multi-point detection. When the photodetectors are distributed in an array, the corresponding conversion circuits are also distributed in an array and one conversion circuit corresponds to one photodetector (see Figure 6, taking the photodetector as SPAD as an example, where the photodetector is 601, and the conversion The circuit is 602), the detection circuit detects multiple points at the same time to obtain the two-dimensional gray value or depth information of the plane.

In one embodiment, please refer to FIG. 7 (taking the photodetector including SPAD and the number as an example), the conversion circuit includes a first switch K1, a second switch K2, a counter J1, an oscillator Z1, and a decoder Y1 , The static contact 0 of the first switch K1 is connected to the input end of the counter J1, the first dynamic contact 2D of the first switch K1, the first end of the oscillator Z1 and the output end of the photodetector SPAD are connected together, the first switch The second dynamic contact 3D of K1, the second end of the oscillator Z1, and the first end of the decoder Y1 are connected together. The second end of the decoder Y1 is connected to the static contact 1 of the second switch K2. The second switch The dynamic contact 2 of K2 and the output terminal of the counter J1 are connected to the input terminal of the readout circuit 1023 in common.

In the two-dimensional mode, the first dynamic contact 2D of the first switch K1 is closed, the second dynamic contact 3D of the first switch K1 is opened, the second switch K2 is opened, and the counter J1 is driven by SPAD to record the activation of SPAD In order to collect the two-dimensional gray value of a single point, and then obtain the two-dimensional gray value of other points point by point by scanning, and finally obtain the image information of the surface. Among them, scanning methods include mechanical scanning, MEMS scanning and optical phased array scanning.

In the three-dimensional mode, the first dynamic contact 2D of the first switch K1 is open, the second dynamic contact 3D of the first switch K1 is closed, the second switch K2 is closed, the counter J1, the oscillator Z1 and the decoder Y1 are combined A TDC (Time-to-Digital Converter) is formed. The counter J1 and the decoder Y1 are driven by the oscillator Z1 to record the flight time of the photon, so as to obtain the distance information of a single point, and then obtain it point by point by scanning The distance information of other points, and finally the depth information of the surface is obtained.

In another embodiment, the detection circuit 1022 can also be configured to collect two kinds of information at the same time, that is, to detect a two-dimensional gray value according to a trigger signal, and to obtain a value according to the first pulsed light and the second pulsed light. Distance information. Therefore, the two-dimensional gray value and the depth information can be obtained at the same time, so that the readout circuit 1023 can obtain the second two-dimensional image information and the three-dimensional image information at the same time.

In this embodiment, the readout circuit 1023 reads out the second two-dimensional image information according to the two-dimensional gray value, and reads out the three-dimensional image information according to the depth information. When the detection circuit 1022 is configured to have two switchable working modes, correspondingly, the readout circuit 1023 reads out the second two-dimensional image information in the two-dimensional mode, and reads out the three-dimensional image information in the three-dimensional mode. When the detection circuit 1022 is configured to collect two kinds of information at the same time, correspondingly, the readout circuit 1023 reads out the second two-dimensional image information and the three-dimensional image information at the same time. Among them, the specific device of the readout circuit 1023 is not limited, as long as it can realize the data reading function, for example, it may include a combination of resistors, capacitors, amplifiers, and samplers.

Thus, through the control circuit 1021, the detection circuit 1022, and the readout circuit 1023, the function of the second acquisition unit 102 to acquire the second two-dimensional image in the two-dimensional mode, the three-dimensional image in the three-dimensional mode, or the second acquisition can be realized. The unit 102 has the function of simultaneously collecting a second two-dimensional image and a three-dimensional image. Wherein, when the second acquisition unit 102 has a mode switching function, the acquisition module 100 can switch to the two-dimensional mode when the registration module 200 requires registration, and only output the first two-dimensional image and the second two-dimensional image; in the fusion module When 300 needs to be fused, it switches to 3D mode and only outputs the first 2D image and 3D image, which can simultaneously optimize power consumption, chip area, and cost.

It should be noted that the first collection unit 101 and the second collection unit 102 of the foregoing embodiment can be integrated in a multi-sensor camera, for example, integrated in a camera with an image sensor and a depth sensor; or they can be respectively disposed in two cameras with multiple sensors. In the camera with independent sensor and auxiliary circuit.

Refer to Fig. 8a-8d, Fig. 8a-8d is a schematic diagram of the structure of the acquisition module in another embodiment.

Further, on the basis of the above-mentioned embodiment, in order to realize the spectral filtering function, the collection module 100 further includes a filter unit 103. The filter unit 103 is arranged between the scene and the first acquisition unit 101, and/or is arranged between the scene and the first collection unit 101. Between the second collection units 102 (Figure 8a is set between the scene and the first collection unit 101, and between the scene and the second collection unit 102 as an example), are configured to perform spectral filtering on the light reflected by the scene . In this way, the wavelength of the image light can be selected to realize single-wavelength imaging, or realize the acquisition of scene color information.

The filter unit 103 may include one or more filters, or may be configured as a color filter array according to the number of image sensors in the first acquisition unit 101 and the number of photodetectors in the second acquisition unit 102, respectively. Wherein, the color filter array is respectively registered with the image sensor array and the photodetector array, so that each color filter covers at least one image sensor and/or one photodetector, and the colors of the color filters corresponding to different devices can be the same or Different; or, each color filter can be optically coupled to at least two image sensors and/or two photodetectors.

Further, in order to improve the accuracy of image information collection, the collection module 100 further includes a dimming unit 104. The dimming unit 104 is arranged between the scene and the first collection unit 101, and/or between the scene and the second collection unit 102. (Figure 8b takes the setting between the scene and the first collection unit 101, and the setting between the scene and the second collection unit 102 as examples), is configured to adjust the intensity of the light reflected by the scene, so that the collection The information is more accurate. In an embodiment, the dimming unit 104 includes one or more flash lights; or includes a light accentuation damper.

It should be noted that the collection module 100 includes one or two of the filter unit 103 and the dimming unit 104. When the collection module 100 includes both the filter unit 103 and the dimming unit 104, the filter unit 103 and the dimming unit 104 The relative position of the unit 104 is not limited and can be adjusted according to actual conditions.

Take the first collection unit 101 as an example for description. For example, when the filter unit 103 and the dimming unit 104 are on the same optical axis, referring to FIG. 8c, the filter unit 103 may be provided in the dimming unit 104 and the first collection unit. Between 101, dimming is performed first and then filtering; or referring to FIG. 8d, the dimming unit 104 is arranged between the filtering unit 103 and the first collection unit 101, and the dimming is performed first and then dimming. When the filter unit 103 and the dimming unit 104 do not need to be on the same optical axis, for example, the dimming unit 104 is a flash, then the flash can be set anywhere between the scene and the first collection unit 101, and the filter unit 103 The location of the setting can be irrelevant.

Refer to FIG. 9, which is a schematic structural diagram of a collection module in another embodiment.

Further, on the basis of the above-mentioned embodiment, in order to realize the function of selecting the image output type and then to select the image processing stage, the acquisition module 100 further includes a switch unit 105, which is respectively connected to the second acquisition unit 102 and the registration module 200 And the fusion module 300 is configured to control the connection state of the second acquisition unit 102 and the registration module 200, and control the connection state of the second acquisition unit 102 and the fusion module 300.

When the device needs to perform registration processing, the switch unit 105 turns on the connection between the second acquisition unit 102 and the registration module 200, and the second acquisition unit 102 outputs the second two-dimensional image to the registration module 200; when the device needs to perform fusion processing When the switch unit 105 turns on the connection between the second acquisition unit 102 and the fusion module 300, the second acquisition unit 102 outputs a three-dimensional image to the fusion module 300; when the device needs to perform registration processing and fusion processing at the same time, turn on the second acquisition unit at the same time. The collection unit 102 is connected to the registration module 200 and the fusion module 300.

In an embodiment, please refer to FIG. 10 for assistance, the switch unit 105 includes a bidirectional switch K3.

The static contact of the bidirectional switch K3 is connected to the second acquisition unit 102, the first movable contact of the bidirectional switch K3 is connected to the input terminal of the registration module 200, and the second movable contact of the bidirectional switch K3 is connected to the input terminal of the fusion module 300. In the two-dimensional mode, the first movable contact of the bidirectional switch K3 is closed, and the second acquisition unit 102 outputs a second two-dimensional image to the registration module 200; in the three-dimensional mode, the second movable contact of the bidirectional switch K3 is closed, The second acquisition unit 102 outputs a three-dimensional image to the fusion module 300.

Corresponding to the image acquisition device provided in the foregoing embodiment, FIG. 11 shows a flowchart of the image acquisition method provided in this embodiment.

In this embodiment, the image acquisition method includes steps S100, S200, S300, and S400. The details are as follows:

Step S100: Collect a first two-dimensional image, a second two-dimensional image, and a three-dimensional image of the scene; the resolution of the first two-dimensional image is higher than the resolution of the second two-dimensional image.

Step S200: Perform image registration on the first two-dimensional image and the second two-dimensional image, and obtain a mapping relationship between the first two-dimensional image and the second two-dimensional image.

Step S300: Perform image fusion on the three-dimensional image and the first two-dimensional image according to the mapping relationship to obtain a fused image.

Step S400: Up-sampling the fused image to obtain a high-resolution three-dimensional depth image.

In this embodiment, step S100 is executed by the acquisition module in the foregoing embodiment, and images of multiple source channels can be obtained. For a specific description of step S100, refer to the relevant description of the acquisition module in the foregoing embodiment. In one embodiment, step S100 includes: step S101 and step S102.

Step S101: Collect image light reflected by the scene to form a first two-dimensional image.

Step S102: Acquire a second two-dimensional image of the scene in the two-dimensional mode, and collect a three-dimensional image of the scene in the three-dimensional mode. Specifically, step S102 controls the light source to emit the first pulsed light to the scene, and receives the second pulsed light reflected back from the scene, and detects the two-dimensional gray value or depth information according to the ambient light, the first pulsed light and the second pulsed light, and then The second two-dimensional image information is read out according to the two-dimensional gray value, and the three-dimensional image information is read out according to the depth information.

In another embodiment, before step S102, step S100 further includes: step S103 and step S104.

Step S103: Perform spectral filtering on the image light reflected by the scene, so that the wavelength of the image light can be selected to realize single-wavelength imaging, or to realize the acquisition of scene color information.

In step S104, the intensity of the image light reflected by the scene is adjusted, so that the collected information is more accurate.

Among them, step S101 is performed by the first collection unit of the above embodiment, step S102 is performed by the second collection unit of the above embodiment, step S103 is performed by the filter unit of the above embodiment, and step S104 is performed by the dimming unit of the above embodiment. Implementation, not repeat them here.

In this embodiment, step S200 is performed by the registration module in the above-mentioned embodiment, which can perform relevant registration on images with differences, and obtain the mapping relationship between different images, so that subsequent images are aligned and fused according to the mapping relationship; step S200 For a specific description of, refer to the relevant description of the registration module in the foregoing embodiment. In one embodiment, step S100 includes: step S201 and step S202.

Step S201, extracting feature point groups corresponding to the first two-dimensional image and the second two-dimensional image respectively.

Step S202: Obtain the mapping relationship between the first two-dimensional image and the second two-dimensional image according to the feature point group.

Among them, step S201 is performed by the extraction unit of the above-mentioned embodiment, and step S102 is performed by the mapping unit of the above-mentioned embodiment, which will not be repeated here.

In this embodiment, step S300 is performed by the fusion module in the above-mentioned embodiment, which can fuse images with different resolutions and different image information into a three-dimensional depth image, improve the resolution of the original acquired image, and realize the integration of different feature information. Fusion improves the integrity of the image information; for the specific description of step S300, refer to the related description of the fusion module in the foregoing embodiment. In one embodiment, step S300 includes: step S301 and step S302.

Step S301: Obtain a projected transformed image according to the three-dimensional image and the mapping relationship.

Step S302, fusing the projected transformed image and the first two-dimensional image to obtain a fused image.

Wherein, step S301 is performed by the transformation unit of the above-mentioned embodiment, and step S302 is performed by the image fusion unit of the above-mentioned embodiment, which will not be repeated here.

In this embodiment, step S400 is performed by the up-sampling module in the above-mentioned embodiment, which can further improve the resolution of image display and generate a high-resolution three-dimensional depth image; for the specific description of step S400, refer to the up-sampling module in the above-mentioned embodiment The related description of, I won’t repeat it here.

It should be noted that, in order to facilitate viewing or evaluating the three-dimensional depth image obtained in the above-mentioned embodiment, the image obtaining method may further include step 500 to visually display the three-dimensional depth image. Step S500 may be after step S300 or step S400, or with step S300. S300 or step S400 are executed simultaneously.

In the above image acquisition method, the first two-dimensional image, the second two-dimensional image, and the three-dimensional image with different resolutions and different image information are obtained by collecting the scene, and the images with differences are correlated to obtain different images. Then, according to the mapping relationship, images with different resolutions and different image information are fused, and the resolution of the fused image is further improved through upsampling, and finally a high-resolution three-dimensional depth image is obtained, while the color information of the image is realized The fusion of different characteristic information such as gray information and depth information improves the integrity of image information.

It should be understood that although the various steps in the flowchart of FIG. 11 are displayed in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless specifically stated in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least part of the steps in FIG. 11 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution of these sub-steps or stages The sequence is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

An embodiment of the present application also provides an electronic device, including a memory and a processor, and a computer program is stored in the memory. When the computer program is executed by the processor, the processor is caused to execute the image acquisition method described in the above embodiment. A step of.

The embodiment of the present application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, cause the processors to execute the image acquisition method in any of the above embodiments step.

Any reference to memory, storage, database or other media used in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which acts as external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, All should be considered as the scope of this specification.

The above-mentioned embodiments only express several embodiments of the present invention, and the descriptions are more specific and detailed, but they should not be understood as limiting the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims

An image acquisition device, characterized by comprising:

An acquisition module configured to acquire a first two-dimensional image, a second two-dimensional image, and a three-dimensional image of the scene; the resolution of the first two-dimensional image is higher than the resolution of the second two-dimensional image;

A registration module, connected to the acquisition module, configured to perform image registration on the first two-dimensional image and the second two-dimensional image, and obtain the first two-dimensional image and the second two-dimensional image The mapping relationship;

A fusion module, which connects the acquisition module and the registration module, and is configured to perform image fusion on the three-dimensional image and the first two-dimensional image according to the mapping relationship to obtain a fused image;

The up-sampling module is connected to the fusion module and is configured to up-sample the fused image to obtain a high-resolution three-dimensional depth image.
The image acquisition device according to claim 1, wherein the acquisition module comprises:

The first collection unit is respectively connected to the registration module and the fusion module, and is configured to collect image light reflected by the scene to form a first two-dimensional image;

The second collection unit is respectively connected to the registration module and the fusion module, and is configured to collect a second two-dimensional image of the scene and output to the registration module, and collect the three-dimensional image of the scene and output to the fusion module .
The image acquisition device according to claim 2, wherein the second acquisition unit comprises:

The control circuit is configured to control the light source to emit the first pulsed light to the scene;

A detection circuit, connected to the control circuit, configured to receive the second pulsed light reflected back from the scene, and detect the two-dimensional gray value and/or depth information according to the first pulsed light and the second pulsed light;

The readout circuit, the detection circuit, the registration module, and the fusion module, respectively, are configured to read out the second two-dimensional image information according to the two-dimensional gray value, and read out the three-dimensional image according to the depth information information.
The image acquisition device according to claim 3, wherein the detection circuit comprises:

A photodetector, connected to the control circuit, configured to detect the second pulsed light reflected by the scene and generate a trigger signal;

The conversion circuit is connected to the photodetector and the readout circuit, and is configured to detect the two-dimensional gray value according to the trigger signal in the two-dimensional mode, and according to the first pulsed light and the second pulsed light in the three-dimensional mode. Pulse light obtains distance information.
The image acquisition device according to claim 4, wherein the conversion circuit comprises:

The first switch, the second switch, the counter, the oscillator and the decoder;

The static contact of the first switch is connected to the input end of the counter, the first dynamic contact of the first switch, the first end of the oscillator, and the output end of the photodetector are connected in common, so The second dynamic contact of the first switch, the second end of the oscillator, and the first end of the decoder are connected in common, and the second end of the decoder is connected to the static contact of the second switch, The dynamic contact of the second switch and the output end of the counter are commonly connected to the input end of the readout circuit.
The image acquisition device according to claim 2, wherein the acquisition module further comprises:

The filter unit is arranged between the scene and the first collection unit, and/or between the scene and the second collection unit, and is configured to perform spectral filtering on light reflected by the scene.
The image acquisition device according to claim 2, wherein the acquisition module further comprises:

The dimming unit is arranged between the scene and the first collection unit, and/or between the scene and the second collection unit, and is configured to adjust the light intensity of the light reflected by the scene.
The image acquisition device according to claim 2, wherein the acquisition module further comprises:

The switch unit is respectively connected to the second acquisition unit, the registration module, and the fusion module, and is configured to control the connection state of the second acquisition unit and the registration module, and control the second acquisition unit Connection status with the fusion module.
8. The image acquisition device according to any one of claims 1-8, wherein the registration module comprises:

An extraction unit, connected to the acquisition module, and configured to respectively extract feature point groups corresponding to the first two-dimensional image and the second two-dimensional image;

The mapping unit is connected to the extraction unit and the fusion module, and is configured to obtain the mapping relationship between the first two-dimensional image and the second two-dimensional image according to a feature point group.
8. The image acquisition device according to any one of claims 1-8, wherein the fusion module comprises:

A transformation unit, connected to the acquisition module and the registration module, and configured to acquire a projective transformed image according to the three-dimensional image and the mapping relationship;

The image fusion unit is connected to the transformation unit and the up-sampling module, and is configured to fuse the projection transformed image and the first two-dimensional image to obtain a fused image.
An image acquisition method, characterized by comprising:

Acquiring a first two-dimensional image, a second two-dimensional image, and a three-dimensional image of the scene; the resolution of the first two-dimensional image is higher than the resolution of the second two-dimensional image;

Performing image registration on the first two-dimensional image and the second two-dimensional image, and acquiring a mapping relationship between the first two-dimensional image and the second two-dimensional image;

Performing image fusion on the three-dimensional image and the first two-dimensional image according to the mapping relationship to obtain a fused image;

Up-sampling the fused image to obtain a high-resolution three-dimensional depth image.
An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the image according to claim 11 Get the steps of the method.
A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the image acquisition method according to claim 11 when the computer program is executed by a processor.