WO2018232631A1

WO2018232631A1 - Image processing method, device and system, and computer storage medium

Info

Publication number: WO2018232631A1
Application number: PCT/CN2017/089380
Authority: WO
Inventors: 阳光
Original assignee: 深圳配天智能技术研究院有限公司
Priority date: 2017-06-21
Filing date: 2017-06-21
Publication date: 2018-12-27
Also published as: CN109429560B; CN109429560A

Abstract

Disclosed is an image processing method, comprising: when an object moves, obtaining a current image collected at the current moment from the perspective of the object; calculating depth information of the current image; and adjusting the depth information of the current image by using the depth information of the collected image. Also disclosed are a corresponding device and system. The present invention effectively improves the resolution of the depth information, thereby improving the resolution of the ambient environment of the object obtained when the object moves.

Description

Image processing method, device, system and computer storage medium

【技术领域】[Technical Field]

The present invention relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, system, and computer storage medium.

【背景技术】【Background technique】

3D vision is a necessary technology in vehicle-mounted automatic or assisted driving systems. Currently, 3D vision is implemented based on the fusion of laser radar and vision. In this method, laser radar provides low-resolution depth information. Vision provides high-resolution texture information and then fuses the two to achieve 3D vision. However, in this method, the resolution of the laser radar is low, which causes visual errors, especially in some complicated situations. For example, the lack of texture, the depth information changes dramatically (such as grid railings, etc.).

【发明内容】 [Summary of the Invention]

The object of the present invention is to provide an image processing method, device, system and computer storage medium for solving problems caused by realizing low-resolution depth information of laser radar in 3D vision in the prior art, and improving resolution of depth information. Rate, which in turn increases the resolution of the surrounding environment obtained by the moving target.

The technical solution adopted by the present invention to solve the above technical problem is to provide an image processing method, comprising: acquiring a current image acquired by using the target as a current time at a current time when the target moves; and calculating depth information of the current image. Adjusting the depth information of the current image by using the depth information of the previously acquired image.

Another technical solution adopted by the present invention to solve the above technical problem is to provide an image processing system including an image capturing device and an image processing device; the image capturing device is configured to use the target as a viewpoint when the target moves Collecting a current image of the current time; the image processing device is configured to calculate depth information of the current image when acquiring the current image, and use depth information of the previously acquired image to depth of the current image Information is adjusted.

Another technical solution adopted by the present invention to solve the above technical problem is to provide a vehicle, wherein the vehicle includes an image capturing device and an image processing device; and the image capturing device is configured to Observing an environment image in real time; the image processing device is configured to acquire a current image acquired by the image capturing device at a current time, calculate depth information of the current image, and utilize depth information of the previously acquired image, Adjusting the depth information of the current image.

Another technical solution adopted by the present invention to solve the above technical problems is to provide an image processing apparatus including a memory and a processor for storing computer instructions configured to be executed by the processor; The processor executes the computer instruction, configured to: acquire a current image acquired by the current time from the target when the target moves; calculate depth information of the current image; and use depth information of the previously acquired image And adjusting the depth information of the current image.

Another technical solution adopted by the present invention to solve the above technical problems is to provide a computer storage medium in which computer instructions executable by a processor are stored, the computer instructions being used to execute the image processing method described above.

The invention has the beneficial effects that the depth information of the current image is adjusted by using the depth information of the image at the previous moment, and the depth resolution of the current image is improved, so that the resolution of the moving target to obtain the surrounding environment becomes higher.

【附图说明】 [Description of the Drawings]

The present invention will be further described below in conjunction with the accompanying drawings and embodiments, in which:

1 is a flow chart of an embodiment of an image processing method of the present invention;

Figure 2 is a flowchart of step S106 in the above embodiment of the present invention;

3-4 are schematic diagrams of application scenarios of the above embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an embodiment of an image processing system of the present invention; FIG.

Figure 6 is a block diagram showing an embodiment of an image processing apparatus of the present invention.

【具体实施方式】【Detailed ways】

The technical solutions of the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in FIG. 1 , it is a flowchart of an embodiment of an image processing method of the present invention, which can be applied to a moving target to acquire an image of a surrounding environment thereof, for example, realizing 3D vision in a vehicle assisted driving, that is, moving vehicle acquisition. An image of the driving environment, the method comprising the steps of:

Step S102: When the target moves, acquire the current image acquired by the current time from the perspective of the target.

In step S102, the current image is acquired from the perspective of the target, indicating that the collection device is set on the target, and when the target moves, the acquisition device moves accordingly, and the acquired image is also different with the movement, for example, the target is a vehicle. In order to collect the driving environment of the vehicle, the collecting device may be a laser radar.

Step S104: Calculate depth information of the current image.

Step S106: Adjust the depth information of the current image by using the depth information of the previously acquired image.

In step S106, the previously acquired image refers to the image acquired before the current time, and may include but not limited to one or more sheets. As the target moves, the previously acquired images are not completely identical, and thus the depth thereof. The information is not completely the same, and the depth information of the current image is adjusted by the depth information of the images that are not identical, and the resolution of the depth information of the current image is improved, thereby improving the resolution of the surrounding environment that can be seen by the moving target.

Through the implementation of the foregoing embodiment, the depth information of the current image is adjusted by using the depth information of the image at the previous time, and the depth resolution of the current image is improved, so that the resolution of the surrounding environment of the moving target is increased.

Specifically, in an embodiment, as shown in FIG. 2, step S106 includes:

Step S1061: Find a superimposed area between the current image and the previous time image acquired at the previous moment.

In one embodiment, finding a superimposed region between the current image and a previous moment image acquired at a previous moment includes: first, finding a reference object in the current image and the previous moment image, wherein the reference object is at the target The ground is stationary while moving relative to the target during the move. Then, the reference object is compared with the position of the current image and the previous time image to obtain the pose difference between the current moment and the previous moment; finally, the pose between the current image and the previous moment image is obtained according to the pose difference. Overlay area. Thus, in some complicated cases, due to the effect of the depth superposition, the resolution of the adjusted depth information is high, so that the moving target has a high resolution in obtaining its surrounding environment. Further, in an embodiment, searching for the reference object in the current image and the previous time image comprises: first, acquiring a static object having the set feature, and setting the feature to the depth information and the texture information respectively satisfying the preset condition The feature, for example, the depth information and the texture information of a certain feature are respectively greater than the corresponding preset values, for example, street lights, buildings, plants, etc. along the way when the target moves; subsequently, determining the moving direction when the target moves; Finding a static object that meets a preset condition in the moving direction as a reference object, specifically, in one embodiment, the preset condition includes a preset parameter, wherein the preset parameter may be a static object in the current image and The degree of display in the image at the previous moment, for example, in the moving direction, a certain static object whose degree of display is above 80% in the images at the two moments can be used as a reference object. Of course, the preset parameter may also be a static object and The distance of the target, for example, when the static object is at a preset range in the image of the two moments, the static object can also Selecting as a reference.

In another embodiment, finding the superimposed area between the current image and the previous time image acquired at the previous moment includes: if the reference object is not found in the current image and the previous time image, for example, not found here. The static object with the set feature is displayed in the image at both time points with the degree of more than 80%; according to the movement parameter of the target movement, the position difference between the current time and the previous time is estimated; the current position and the position deviation are obtained according to the current position and position deviation. The superimposed area between the image and the image of the previous moment. The movement parameters in which the target moves include the moving speed and the like.

Step S1062: Superimpose the depth information of the previous time image into the depth information of the superimposed area of the current image to obtain the adjusted depth information of the current image. The depth information of the image at the previous moment is obtained by adjusting the depth information of the image before the previous moment and the depth information of the image of the previous moment.

In one embodiment, the depth information of the current image is adjusted by using the depth information of the previously acquired image, and further includes:

Record the deviation of the pose of the target in the image acquired at each moment, because during the running of the vehicle, the captured image may also have errors or errors due to visual errors, resulting in the vehicle obtained by comparing the images. The difference between the pose difference and the actual pose difference of the vehicle may be deviated; and the cumulative deviation of the pose of the target in the image at the previous moment is calculated, and the cumulative deviation refers to the sum of the deviations of all the moments before and immediately before the previous moment; Whether the cumulative deviation of the pose of the target in the time image exceeds the preset value; if not exceeded, the superimposed area between the current image and the previous time image is found, and if it is exceeded, the previous moment collected at the previous moment is directly The image is discarded. At this time, the depth information of the current image is not adjusted, and the depth information of the current image is used as a basis for adjusting the image acquired when the subsequent target is moved.

One of the application scenarios of the present invention will be described below based on the description of the above embodiments.

Assuming that the target is the vehicle car1, in order to obtain a high-resolution vehicle driving environment, as shown in FIGS. 3 and 4, respectively, the vehicle travels to the first time. 1 and the second time Time 2, the first time Time 1 corresponds to the first position Position 1, and the first image A1 acquired at this time; the second time Time 2 corresponds to the second position Position 3, and the second image A2 acquired at this time, and at the same time, obj 1-3 are all static objects. First time Time 1 and second time Time The interval of 2 can be determined by the frequency of the image acquisition device. Assume that the second time Time 2 is the current time of the vehicle, correspondingly, the second position Position 3 is the current position of the vehicle, and the second image A2 collected at the second time Time 2 is the current image, and the first image A1 collected at the first time Time 1 is the previous time image.

It is assumed that when the second image A2 is processed, the static object obj 2 is used as a reference object, and further by the reference object obj. 2 Aligning the positions in the second image A2 and the first image A1, the second position Position 3 of the vehicle in the second image A2 is obtained, and the vehicle is at the second time Time 2 and the first time Time 1 o'clock position difference, thus depending on the position difference and the second position Position 3, a superimposed region between the second image and the first image may be obtained. At this time, the depth information in the superimposed region in the first image may be superimposed on the depth information of the superimposed region in the second image as the second image. The depth information is such that the resolution of the depth information of the superimposed region becomes higher. Repeated in sequence, during the subsequent vehicle movement, the second time Time The depth information of the second image A2 collected is used as a reference for depth adjustment of the image acquired at a later time, and the second time is used. The depth information of the second acquired image A2 is depth-adjusted for the image acquired at a later time.

It is assumed that when the second image A2 is processed, it cannot be traced to the static objects obj 1 and obj. 2, while processing the first image A1, the static object obj 3 can be traced. At this time, it is estimated that the vehicle car1 can estimate the vehicle car1 at the second time according to the moving speed of the vehicle car1. 2 with the first moment Time At 1 o'clock, the pose is poor, so that the superimposed region between the second image A2 and the first image A1 can be obtained according to the pose difference. At this time, the depth information in the superimposed region in the first image A1 can be superimposed to the second image. The depth information of the superimposed region in the image A2 is taken as the depth information in the second image A2, so that the resolution of the depth information of the superimposed region becomes high. Repeated in sequence, during the subsequent vehicle movement, the second time Time The depth information of the second image A2 collected is used as a reference for depth adjustment of the image acquired at a later time, and the second time is used. The depth information of the second acquired image A2 is depth-adjusted for the image acquired at a later time.

As shown in FIG. 5, which is a schematic structural diagram of an embodiment of an image processing system of the present invention, the system 500 includes an image capture device 510 and an image processing device 520, wherein the image capture device is coupled to the image processing device 520.

The image capturing device 510 is configured to collect a current image of the current moment from a target perspective when the target moves;

The image processing device 520 is configured to calculate depth information of the current image when the current image is acquired, and adjust depth information of the current image by using depth information of the previously acquired image.

The present invention also provides a vehicle, such as the vehicle car1 of Figures 3 and 4, which includes the image capture device 510 and image processing device 520 of the above-described embodiments.

Specifically, the image processing device 520 is as follows. The image processing device 520 will be described in detail below, and will not be described here.

As shown in FIG. 6, which is a schematic structural diagram of an embodiment of an image processing apparatus of the present invention, the image processing apparatus 600 can be mounted on a moving target, including a memory 610, a processor 620, and a bus 630.

The memory 610 is used to store computer instructions configured to be executed by the processor 620 and data that needs to be saved or cached during operation of the processor 620, such as depth information of previously acquired images.

In this embodiment, the processor 620 is configured to execute by calling a computer instruction stored in the memory 610:

When the target moves, the current image acquired at the current time is acquired from the perspective of the target;

Calculating depth information of the current image;

The depth information of the current image is adjusted using the depth information of the previously acquired image.

Further, specifically, the processor 620 adjusts the depth information of the current image by using the depth information of the previously acquired image, including: searching for a superimposed area between the current image and the previous time image acquired at a previous moment. And superimposing the depth information of the superimposed area of the previous time image into the depth information of the superimposed area of the current image to obtain the adjusted depth information of the current image. Further, in an embodiment, searching for a superimposed region between the current image and the previous moment image acquired at a previous moment includes: first, finding a reference object in the current image and the previous moment image, wherein the reference object The ground is stationary while moving relative to the target during the movement of the target. Then, the reference object is compared with the position of the current image and the previous time image to obtain a pose difference between the current time and the previous time; finally, the superposition between the current image and the previous time image is obtained according to the pose difference. region. In another embodiment, finding a superimposed area between the current image and a previous time image acquired at a previous moment includes: if the reference object is not found in the current image and the previous time image; The parameter estimates the difference between the current position and the previous moment of the target; and the superimposed area between the current image and the previous time image is obtained according to the current position and the pose difference.

Further, the processor 620 searches for the reference object in the current image and the previous time image, and includes: acquiring a static object having the set feature, wherein the setting feature is that the depth information and the texture information respectively satisfy the preset condition. For example, the depth information and the texture information of a certain feature are respectively greater than the corresponding preset values; determining the moving direction when the target moves; and finding a static object satisfying the preset condition in the moving direction as a reference object. For details, please refer to the description of the above method embodiment.

Further, the processor 620 is further configured to record the deviation of the pose of the target in the image acquired at each moment, and calculate the cumulative deviation of the pose of the target in the image at the previous moment; and determine the image at the previous moment. Whether the cumulative deviation of the pose of the middle target exceeds a preset value; if not, finds a superimposed area between the current image and the previous time image. If it is exceeded, the image of the previous moment collected at the previous moment is directly discarded.

The processor 620 may also be referred to as a CPU (Central Processing). Unit, central processing unit). Memory 610 can include read only memory and random access memory and provides instructions and data to processor 620. A portion of memory 610 may also include non-volatile random access memory (NVRAM). In a specific application, the foregoing components of the mobile terminal are coupled together by a bus 630. The bus 630 may include a power bus, a control bus, a status signal bus, and the like in addition to the data bus. However, for clarity of description, various buses are labeled as bus 630 in the figure.

The method disclosed in the foregoing embodiments of the present invention may be applied to the processor 620 or implemented by the processor 620. Processor 620 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 620 or an instruction in a form of software. The processor 620 described above may be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or discrete hardware. Component. The methods, steps, and logical block diagrams disclosed in the embodiments of the present invention may be implemented or carried out. The general purpose processor may be a microprocessor or the processor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present invention may be directly implemented by the hardware decoding processor, or may be performed by a combination of hardware and software modules in the decoding processor. The software module can be located in a conventional storage medium such as random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, and the like. The storage medium is located in the memory 610, and the processor 620 reads the information in the memory 610 and completes the steps of the above method in combination with its hardware.

To this end, the present invention also provides a computer readable storage medium, which is specifically usable as a memory 610 as shown in FIG. 6, which stores computer instructions executable on the processor 620, in particular, In the present embodiment, computer instructions can be executed to implement the image processing method in the above embodiment.

It is noted that the computer readable storage medium storing computer instructions executable on the processor 630 corresponds to the steps in the above method embodiments.

In this embodiment, the depth information of the current image is adjusted by using the depth information of the image at the previous time, and the depth resolution of the current image is improved, so that the resolution of the surrounding environment of the moving target is increased.

The above are only the embodiments of the present invention, and are not intended to limit the scope of the invention, and the equivalent structure or equivalent process transformations made by the description of the present invention and the drawings are directly or indirectly applied to other related technical fields. The same is included in the scope of patent protection of the present invention.

Claims

An image processing method, comprising:

Obtaining a current image acquired by the current time from the perspective of the target when the target moves;

Calculating depth information of the current image;

The depth information of the current image is adjusted using the depth information of the previously acquired image.
The method according to claim 1, wherein the adjusting the depth information of the current image by using the depth information of the previously acquired image comprises:

Finding a superimposed area between the current image and the previous time image;

And superposing the depth information of the superimposed area of the previous time image into the depth information of the superimposed area of the current image to obtain the adjusted depth information of the current image.
The method according to claim 2, wherein

And searching for the superimposed area between the current image and the previous time image, including:

Finding a reference object in the current image and the previous time image, wherein the reference object is stationary with respect to the ground when the target moves during the moving of the target;

Comparing the reference object with the position of the current image and the previous time image to obtain a pose difference of the target at the current time and the previous time;

And obtaining an overlapping area between the current image and the previous time image according to the pose difference.
The method according to claim 3, wherein

The finding the reference object in the current image and the previous time image includes:

Obtaining a static object with a set feature;

Determining the direction of movement of the target as it moves;

A static object that satisfies a preset condition is found in the moving direction as the reference object.
The method according to claim 2, wherein

And searching for the superimposed area between the current image and the previous time image, including:

If the reference object is not found in the current image and the previous time image, the pose difference of the target at the current time and the previous time is estimated according to the movement parameter of the target movement;

And obtaining an overlapping area between the current image and the previous time image according to the pose difference.
The method according to claim 2, wherein

The adjusting the depth information of the current image by using the depth information of the previously acquired image, further includes:

Recording a deviation of the pose of the target in the image acquired at each moment, and calculating a cumulative deviation of the pose of the target in the image at the previous moment;

Determining whether a cumulative deviation of the pose of the target in the image of the previous moment exceeds a preset value;

If not, the superimposed area between the current image and the previous time image is found.
An image processing system including an image capture device and an image processing device;

The image capturing device is configured to collect a current image of the current time with the target as a viewpoint when the target moves;

The image processing apparatus is configured to calculate depth information of the current image when the current image is acquired, and adjust depth information of the current image by using depth information of the previously acquired image.
A vehicle, wherein the vehicle includes an image capture device and an image processing device;

The image capturing device is configured to collect an environment image in real time at a viewing angle of the vehicle when the vehicle moves;

The image processing device is configured to acquire a current image acquired by the image capturing device at a current time, calculate depth information of the current image, and use the depth information of the previously acquired image to obtain depth information of the current image. Make adjustments.
An image processing apparatus including a memory and a processor;

The memory is for storing computer instructions configured to be executed by the processor;

The processor executes the computer instructions for:

Obtaining a current image acquired by the current time from the perspective of the target when the target moves;

Calculating depth information of the current image;

The depth information of the current image is adjusted using the depth information of the previously acquired image.
The image processing device according to claim 9, wherein

The processor performs the adjustment of the depth information of the current image by using the depth information of the previously acquired image, including:

Finding a superimposed area between the current image and a previous time image acquired at a previous moment;

And superposing the depth information of the superimposed area of the previous time image into the depth information of the superimposed area of the current image to obtain the adjusted depth information of the current image.
The image processing device according to claim 10, wherein

The processor performs a search for a superimposed area between the current image and a previous time image acquired at a previous time, including:

Finding a reference object in the current image and the previous time image, wherein the reference object is stationary with respect to the ground when the target moves during the moving of the target;

Comparing the reference object with the position of the current image and the previous time image to obtain a pose difference of the target at the current time and the previous time;

And obtaining an overlapping area between the current image and the previous time image according to the pose difference.
The image processing device according to claim 11, wherein

The processor performs a search for a reference object in the current image and the previous time image, including:

Obtaining a static object with a set feature;

Determining a moving direction when the target moves;

A static object that satisfies a preset condition is found in the moving direction as the reference object.
The image processing device according to claim 10, wherein

The processor performs the searching for the superimposed area between the current image and the previous time image acquired by the previous moment, including:

If the reference object is not found in the current image and the previous time image, the pose difference of the target at the current time and the previous time is estimated according to the movement parameter of the target movement;

And obtaining an overlapping area between the current image and the previous time image according to the pose difference.
The image processing device according to claim 10, wherein

The processor performs the adjustment of the depth information of the current image by using the depth information of the previously acquired image, and further includes:

Recording a deviation of the pose of the target in the image acquired at each moment, and calculating a cumulative deviation of the pose of the target in the image at the previous moment;

Determining whether a cumulative deviation of the pose of the target in the image of the previous moment exceeds a preset value;

If not, the superimposed area between the current image and the previous time image is found.
A computer storage medium storing computer instructions executable by a processor for performing the image processing method of any one of claims 1 to 6.