CN114677425A - Method and device for determining depth of field of object - Google Patents

Abstract

The application provides a method and a device for determining depth of field of an object. The method comprises the following steps: acquiring a plurality of pieces of historical image data, wherein the plurality of pieces of historical image data are obtained by shooting historical target objects at the same distance and/or different distances away from a vehicle in a historical time period by adopting a plurality of image acquisition devices with different field angles; training by adopting each historical image data and the depth of field of the historical target corresponding to each historical image data to obtain a normalized model; acquiring real-time image data, and determining a real-time target object corresponding to the real-time image data by adopting a normalization model to calculate the depth of field; and performing inverse normalization processing on the calculated depth of field of the real-time target object by adopting related parameters of image acquisition equipment for shooting the real-time image data to obtain the real depth of field of the real-time target object corresponding to the real-time image data. The normalization model of the scheme is suitable for image acquisition equipment with different visual angles and target objects with different distances.

Description

Method and device for determining the depth of field of an object

technical field

The present application relates to the field of computer vision, and in particular, to a method, an apparatus, a computer-readable storage medium, a processor, a vehicle and a system for determining the depth of field of an object.

Background technique

Computer vision is mainly a simulation of biological vision through computers and related visual sensors. First, the visual sensor is used to obtain the external image, and then the external image is converted into a digital signal to realize the data processing of the image.

Estimating the depth of field of an object is an important branch of computer vision. The existing solutions often use a simple monocular scene depth estimation method and a binocular scene depth estimation method. The monocular camera is used to obtain fewer features, and the binocular camera is used. Stereo image matching needs to be performed, and the calculation is relatively complicated. Both the monocular scene depth estimation method and the binocular scene depth estimation method cause the problem of low accuracy.

SUMMARY OF THE INVENTION

The main purpose of the present application is to provide a method, apparatus, computer-readable storage medium, processor, vehicle and system for determining the depth of field of an object, so as to at least solve the problem of low accuracy of the method for estimating the depth of field of an object.

In order to achieve the above object, according to an aspect of the present application, a method for determining the depth of field of an object is provided, the method is applied to a vehicle driving system, and the vehicle driving system includes a vehicle and a vehicle mounted on the vehicle with different viewing angles The multiple image acquisition devices, including: acquiring multiple pieces of historical image data, the multiple pieces of the historical image data are obtained by using a plurality of the image acquisition devices with different field of view, shooting the same distance from the vehicle in the historical time period and/or historical objects at different distances; use each of the historical image data and the depth of field of the historical object corresponding to each of the historical image data, and perform training to obtain a normalized model; obtain real-time image data , and use the normalized model to determine the real-time target corresponding to the real-time image data to calculate the depth of field; use the relevant parameters of the image acquisition device that captures the real-time image data to calculate the depth of field for the real-time target. Perform inverse normalization processing to obtain the real depth of field of the real-time target object corresponding to the real-time image data.

Further, using the relevant parameters of the image acquisition device that shoots the real-time image data, the real-time target object is calculated with a depth of field to perform inverse normalization processing to obtain the real-time target object corresponding to the real-time image data. The real depth of field includes: determining the ratio relationship between the real-time target calculated depth of field and the real depth of field of the real-time target to be determined according to the relevant parameters of the image acquisition device; according to the ratio relationship and the real-time target object Calculate the depth of field, and determine the real depth of field of the real-time target.

Further, using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data, performing training to obtain a normalized model, including: filtering and thresholding each of the historical image data. Segmentation processing to obtain processed historical image data corresponding to each of the historical image data; using each of the processed historical image data and the depth of field of the historical target object corresponding to each of the processed historical image data , and train to get a normalized model.

Further, using each of the processed historical image data and the depth of field of the historical target corresponding to each of the processed historical image data, performing training to obtain a normalized model, including: extracting the processed historical image data. A variety of different characteristic parameters of the historical image data, and one color channel of the processed historical image data represents one kind of the characteristic parameter; adopting a plurality of different characteristics of each of the processed historical image data parameters, and the depth of field of the historical object corresponding to each of the processed historical image data are trained to obtain a normalized model.

Further, when there are three image acquisition devices, use each of the historical image data and the depth of field of the historical object corresponding to each of the historical image data to perform training to obtain a normalized model, including: Constructing a training set, the training set includes a first quantity of the historical image data captured by an image acquisition device with a first field of view, and a second quantity of historical image data captured by an image capture device with a second field of view , the third quantity of the historical image data captured by the image acquisition device at the third angle of view, wherein the first quantity is at least determined by the size of the first angle of view, the image of the first angle of view The relative positional relationship between the acquisition device and the vehicle and the relative positional relationship between the historical target and the vehicle are determined, and the second quantity is at least determined by the size of the second field of view, the second field of view The relative positional relationship between the field angle image acquisition device and the vehicle and the relative positional relationship between the historical target and the vehicle are determined, and the third quantity is at least determined by the size of the third field of view, the The relative positional relationship between the third field of view image acquisition device and the vehicle and the relative positional relationship between the historical target and the vehicle are determined; the training set is used for training to obtain the normalized model.

Further, in the process of training using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data, the method further includes: acquiring the data obtained by the normalized model. an error between the output result and the depth of field of the historical target; adjust at least one of the first number, the second number and the third number according to the error.

Further, by adopting the relevant parameters of the image acquisition device that captures the real-time image data, the real-time target object is calculated depth of field and is subjected to inverse normalization processing to obtain the real-time target object corresponding to the real-time image data. After the real depth of field is obtained, the method further includes: determining the driving speed and the driving acceleration of the vehicle according to the real depth of field of the real-time target.

Further, the relevant parameters of the image capture device include the relative pose between the coordinates of the image capture device and the world coordinates, the position of the optical center of the image capture device, and the distortion amount of the image capture device.

Further, the angle of view is one of the following: 30°, 60°, 90°, 120°.

Further, the normalization model is a convolutional neural network model, and the convolutional neural network model includes an input layer, an output layer and a hidden layer.

According to another aspect of the present application, there is provided an apparatus for determining the depth of field of an object, the apparatus being applied to a vehicle driving system, the vehicle driving system including a vehicle and a plurality of images mounted on the vehicle with different viewing angles The acquisition device includes: a first acquisition unit for acquiring a plurality of pieces of historical image data, the plurality of pieces of the historical image data are obtained by using a plurality of the image acquisition devices with different field of view, and the shooting distance within the historical time period is Obtained from historical objects at the same distance and/or at different distances from the vehicle; the training unit is used to use each of the historical image data and the depth of field of the historical object corresponding to each of the historical image data to perform training to obtain a normalized object. A normalized model; a second acquisition unit for acquiring real-time image data, and using the normalized model to determine a real-time target corresponding to the real-time image data to calculate the depth of field; a processing unit for capturing the real-time image by using According to the relevant parameters of the image acquisition device, the calculated depth of field of the real-time target is denormalized, and the real depth of field of the real-time target corresponding to the real-time image data is obtained.

According to another aspect of the present application, a computer-readable storage medium is provided, and the computer-readable storage medium includes a stored program, wherein when the program is executed, a device on which the computer-readable storage medium is located is controlled to execute any arbitrary program. a method as described.

According to another aspect of the present application, there is provided a processor for running a program, wherein any one of the methods is executed when the program is run.

According to yet another aspect of the present application, there is provided a vehicle comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured For execution by the one or more processors, the one or more programs include means for performing any of the described methods.

According to yet another aspect of the present application, a system is provided, comprising the vehicle and a plurality of image acquisition devices with different viewing angles, the image acquisition devices are installed on the vehicle, and the image acquisition device communicate with the vehicle.

By applying the technical solution of the present application, by acquiring a plurality of pieces of historical image data, the plurality of pieces of the above-mentioned historical image data are obtained by using a plurality of the above-mentioned image acquisition devices with different field of view, and shooting the same distance and/or different distances from the above-mentioned vehicle within a historical time period. The historical target object at the distance is obtained, using each of the above-mentioned historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned historical image data, conduct training to obtain a normalized model, obtain real-time image data, and use the above-mentioned normalization. The model determines the real-time target object corresponding to the above-mentioned real-time image data to calculate the depth of field, and uses the relevant parameters of the above-mentioned image acquisition device that captures the above-mentioned real-time image data to perform inverse normalization processing on the above-mentioned real-time target object to calculate the depth of field, and obtain the real-time image with the above-mentioned real-time image. The real depth of field of the real-time target object corresponding to the data. Since the normalized model is obtained by using a plurality of the above-mentioned image acquisition devices with different field of view to collect historical objects at the same distance and/or different distances from the above-mentioned vehicle, the obtained normalized model is suitable for images from different perspectives Acquisition equipment, targets at different distances. That is, a general model is obtained by training, which can obtain the depth of field of the target object corresponding to the images collected by the above image acquisition devices with different field angles, and then through inverse normalization processing to obtain the real depth of field of the real-time target object .

Description of drawings

The accompanying drawings that form a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute improper limitations on the present application. In the attached image:

1 shows a flowchart of a method for determining the depth of field of an object according to an embodiment of the present application;

FIG. 2 shows a schematic diagram of an apparatus for determining the depth of field of an object according to an embodiment of the present application;

FIG. 3 shows a schematic diagram of solving a normalized multiplier according to an embodiment of the present application.

Detailed ways

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances for the embodiments of the application described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. Also, in the specification and claims, when an element is described as being "connected" to another element, the element can be "directly connected" to the other element or "connected" to the other element through a third element.

As described in the background art, the method for estimating the depth of field of an object in the prior art has low accuracy. In order to solve the problem of low accuracy of the method for estimating the depth of field of an object, the embodiments of the present application provide a Methods, apparatus, computer-readable storage media, processors, vehicles, and systems for object depth of field.

According to an embodiment of the present application, a method for determining the depth of field of an object is provided.

FIG. 1 is a flowchart of a method for determining the depth of field of an object according to an embodiment of the present application. The above-mentioned method is applied to a vehicle driving system, and the above-mentioned vehicle driving system includes a vehicle and a plurality of image acquisition devices with different field of view installed on the above-mentioned vehicle. As shown in FIG. 1 , the method includes the following steps:

Step S101, obtaining a plurality of pieces of historical image data, the plurality of pieces of the above-mentioned historical image data are obtained by using a plurality of above-mentioned image acquisition devices with different field of view to shoot historical objects at the same distance and/or different distances from the above-mentioned vehicle within a historical time period obtained;

Step S102, using each of the above-mentioned historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned historical image data, perform training to obtain a normalized model;

Step S103, acquiring real-time image data, and using the above-mentioned normalization model to determine the real-time target object corresponding to the above-mentioned real-time image data to calculate the depth of field;

Step S104, using the relevant parameters of the image acquisition device that captures the real-time image data, to perform inverse normalization processing on the real-time target object to calculate the depth of field, to obtain the real depth of field of the real-time target object corresponding to the real-time image data.

Specifically, the above-mentioned image acquisition device may select a camera.

In the above scheme, by acquiring a plurality of pieces of historical image data, the plurality of pieces of the above-mentioned historical image data are obtained by using a plurality of the above-mentioned image acquisition devices with different field of view to shoot images at the same distance and/or different distances from the above-mentioned vehicle within a historical time period. If the historical target object is obtained, use each of the above-mentioned historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned historical image data, carry out training to obtain a normalized model, obtain real-time image data, and use the above-mentioned normalized model to determine. The real-time target object corresponding to the above-mentioned real-time image data is used to calculate the depth of field, and the relevant parameters of the above-mentioned image acquisition device that captures the above-mentioned real-time image data are used to perform inverse normalization processing on the above-mentioned real-time target object to calculate the depth of field to obtain the corresponding real-time image data. The real depth of field of the above real-time target. Since the normalized model is obtained by using a plurality of the above-mentioned image acquisition devices with different field of view to collect historical objects at the same distance and/or different distances from the above-mentioned vehicle, the obtained normalized model is suitable for images from different perspectives Acquisition equipment, targets at different distances. That is, a general model is obtained by training, which can obtain the depth of field of the target object corresponding to the images collected by the above image acquisition devices with different field angles, and then through inverse normalization processing to obtain the real depth of field of the real-time target object .

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be executed in a computer system, such as a set of computer-executable instructions, and, although a logical sequence is shown in the flowcharts, in some cases, Steps shown or described may be performed in an order different from that herein.

In an optional embodiment, the relevant parameters of the above-mentioned image acquisition device that captures the above-mentioned real-time image data are used to perform inverse normalization processing on the above-mentioned real-time target object to calculate the depth of field, so as to obtain the above-mentioned real-time target object corresponding to the above-mentioned real-time image data. The real depth of field includes: determining the ratio relationship between the real-time target calculated depth of field and the real depth of field of the real-time target to be determined according to the relevant parameters of the above-mentioned image acquisition device; according to the above-mentioned ratio relationship and the real-time target Calculate the depth of field, Determine the true depth of field of the above real-time target. That is, after inverse normalization, the corresponding relationship between the calculated depth of field of the real-time target and the real depth of field of the real-time target is obtained. Since the depth of field is the distance information, the ratio between the calculated depth of field of the real-time target and the real depth of field of the real-time target is obtained. , and then calculate the depth of field and the ratio relationship according to the real-time target obtained by the normalized model to determine the real depth of field of the real-time target. Accurate determination of the real depth of field of real-time objects is achieved.

In a specific embodiment of the present application, the method for obtaining the depth of field of an object includes the following steps:

Step 1: Multiply the homogeneous coordinates of the center point of the 3D object to the left by the camera extrinsic parameter matrix, and then left-multiply the camera intrinsic parameter matrix to obtain the two-dimensional coordinates of the center point of the 3D object in the image plane;

Step 2: Normalize the two-dimensional coordinates of the center point of the 3D object in the image plane through the internal reference matrix to obtain the coordinates of the center point of the 3D object on the normalized image plane;

Step 3: As shown in Figure 3, emit a ray from the camera center M to the two-dimensional coordinate N of the 3D object center point on the normalized image plane Z, and intercept the ray at a predetermined depth (for example, 10 meters) to obtain a point P, construct a sphere B (for example, with a diameter of 1 cm) based on point P, and then project the sphere B onto the normalized image plane Z to obtain a circle C, and obtain the number of pixels occupied by the width of the circle. The number of pixels is divided by the diameter of the sphere to get the normalized multiplier;

Step 4: Divide the depth of the marked 3D object (ie the true depth) by the normalization multiplier to obtain the normalized object depth, which is used as the deep learning regression target;

Step 5: Denormalization: For a 2D bounding box of an object on an image, the normalization model estimates a normalized depth. When inverse normalization is performed, the center point of the frame is taken as the two-dimensional coordinate, and the above steps 2 and 3 are performed to obtain the normalized multiplier. Multiply the depth estimated by the model by the normalization multiplier to get the denormalized depth, which is the true depth.

In an optional embodiment, using each of the above-mentioned historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned historical image data, performing training to obtain a normalized model, including: filtering each of the above-mentioned historical image data. and threshold segmentation processing to obtain processed historical image data corresponding to each of the above-mentioned historical image data; using each of the above-mentioned processed historical image data and the depth of field of the above-mentioned historical object corresponding to each of the above-mentioned processed historical image data, carry out Trained to get a normalized model. Specifically, the filtering process is to filter out the noise of historical image data, and the threshold segmentation process is to perform binarization segmentation. Using the processed historical image data is conducive to the training of the model and ensures the accuracy of the normalized model obtained by training.

In an optional embodiment, using each of the above-mentioned processed historical image data and the depth of field of the above-mentioned historical object corresponding to each of the above-mentioned processed historical image data, performing training to obtain a normalized model, including: extracting the above-mentioned processing. A variety of different characteristic parameters of the historical image data after processing, and a color channel of the above-mentioned processed historical image data represents a kind of above-mentioned characteristic parameters; using each of the above-mentioned various different above-mentioned characteristic parameters of the processed historical image data, And the depth of field of the historical target object corresponding to each of the above-mentioned processed historical image data is trained to obtain a normalized model. The training of the model is actually to train the parameters, to extract a variety of different characteristic parameters of the above processed historical image data, and then to train to obtain an accurate normalized model.

Specifically, the processed historical image data has ten color channels, and each color channel can represent a characteristic parameter, and then images corresponding to different color channels can be used for training to obtain a normalized model.

In an optional embodiment, when there are three image acquisition devices, each of the above-mentioned historical image data and the depth of field of the above-mentioned historical object corresponding to each of the above-mentioned historical image data are used for training to obtain a normalized model, comprising: constructing a training set, the training set includes a first quantity of the above-mentioned historical image data captured by an image acquisition device with a first field of view, and a second quantity of the above-mentioned historical image data captured by an image acquisition device with a second field of view, The third quantity of the above historical image data captured by the image acquisition device at the third angle of view, wherein the first quantity is at least determined by the size of the first angle of view, the image acquisition device at the first angle of view and the vehicle The relative positional relationship between the above-mentioned historical target and the above-mentioned vehicle is determined, and the above-mentioned second quantity is at least determined by the size of the above-mentioned second angle of view, the relative position of the image acquisition device of the above-mentioned second angle of view and the above-mentioned vehicle. relationship and the relative positional relationship between the historical object and the vehicle, and the third quantity is at least determined by the size of the third angle of view, the relative positional relationship between the image acquisition device and the vehicle at the third angle of view, and the above It is determined by the relative positional relationship between the historical target and the above-mentioned vehicle; the above-mentioned normalized model is obtained by using the above-mentioned training set for training. That is, the amount of data in the training set can be determined according to parameters such as the size of the field of view, the relative positional relationship between the image acquisition device and the above-mentioned vehicle, and the relative positional relationship between the above-mentioned historical target and the above-mentioned vehicle. The normalized model has better applicability and higher accuracy.

In an optional embodiment, in the process of training using each of the above-mentioned historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned historical image data, the above method further includes: obtaining the above normalized model. The error between the output result and the depth of field of the historical target object; adjust at least one of the first quantity, the second quantity and the third quantity according to the above error. That is, in order to ensure the accuracy of the parameters of the model during the training process, the first quantity, the above-mentioned second quantity and the above-mentioned third quantity can be adjusted according to the error between the output result obtained by the normalized model and the depth of field of the above-mentioned historical target. at least one of.

In another embodiment, the parameters in the model may be adjusted according to the error between the output result obtained by the normalized model and the depth of field of the historical target object. For example, adjust the number of layers of the network.

In an optional embodiment, by using the relevant parameters of the above-mentioned image acquisition device that captures the above-mentioned real-time image data, inverse normalization processing is performed on the above-mentioned real-time target object to calculate the depth of field, and the above-mentioned real-time target corresponding to the above-mentioned real-time image data is obtained. After determining the real depth of field of the object, the method further includes: determining the driving speed and the driving acceleration of the vehicle according to the real depth of field of the real-time target object. That is, according to the real depth of field of the real-time target, the real-time navigation is guided.

In an optional embodiment, the relevant parameters of the image capturing device include the relative pose between the coordinates of the image capturing device and the world coordinates, the optical center position of the image capturing device, and the distortion amount of the image capturing device. Of course, the relevant parameters also include other parameters except the relative pose between the coordinates of the image acquisition device and the world coordinates, the position of the optical center of the above-mentioned image acquisition device, and the distortion amount of the above-mentioned image acquisition device. Choose according to actual needs.

In an optional embodiment, the above-mentioned field of view angle is one of the following: 30°, 60°, 90°, and 120°. Of course, the viewing angle may also be a viewing angle other than 30°, 60°, 90°, and 120°.

In an optional embodiment, the normalization model is a convolutional neural network model, and the convolutional neural network model includes an input layer, an output layer, and a hidden layer.

The embodiment of the present application further provides an apparatus for determining the depth of field of an object. It should be noted that the apparatus for determining the depth of field of an object in the embodiment of the present application may be used to execute the method for determining the depth of field of an object provided by the embodiment of the present application. The device for determining the depth of field of an object provided by the embodiments of the present application will be introduced below.

FIG. 2 is a schematic diagram of an apparatus for determining the depth of field of an object according to an embodiment of the present application. The above-mentioned device is applied to a vehicle driving system, and the above-mentioned vehicle driving system includes a vehicle and a plurality of image acquisition devices with different viewing angles installed on the above-mentioned vehicle. As shown in FIG. 2 , the device includes:

The first acquisition unit 10 is used to acquire a plurality of pieces of historical image data. The plurality of pieces of the above-mentioned historical image data are obtained by using a plurality of the above-mentioned image acquisition devices with different field of view, and shooting the same distance and/or different distances from the above-mentioned vehicle within a historical time period. Obtained from historical objects at distance;

The training unit 20 is configured to use each of the above-mentioned historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned historical image data to perform training to obtain a normalized model;

The second acquiring unit 30 is configured to acquire real-time image data, and use the above-mentioned normalization model to determine the real-time target object corresponding to the above-mentioned real-time image data to calculate the depth of field;

The processing unit 40 is configured to perform inverse normalization processing on the calculated depth of field of the above-mentioned real-time target by using the relevant parameters of the above-mentioned image acquisition device that captures the above-mentioned real-time image data to obtain the real depth of field of the above-mentioned real-time target corresponding to the above-mentioned real-time image data. .

In the above solution, the first acquisition unit acquires a plurality of pieces of historical image data, and the plurality of pieces of the above-mentioned historical image data is obtained by using a plurality of above-mentioned image acquisition devices with different field of view, and shooting the same distance and/or different distances from the above-mentioned vehicle within a historical time period. The historical target object at the distance is obtained, the training unit adopts each of the above-mentioned historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned historical image data, and performs training to obtain a normalized model, and the second acquisition unit acquires real-time image data, And the above-mentioned normalization model is used to determine the real-time target object corresponding to the above-mentioned real-time image data to calculate the depth of field, and the processing unit adopts the relevant parameters of the above-mentioned image acquisition device that shoots the above-mentioned real-time image data to calculate the depth of field of the above-mentioned real-time target object. Then, the real depth of field of the real-time target object corresponding to the real-time image data is obtained. Since the normalized model is obtained by using a plurality of the above-mentioned image acquisition devices with different field of view to collect historical objects at the same distance and/or different distances from the above-mentioned vehicle, the obtained normalized model is suitable for images from different perspectives Acquisition equipment, targets at different distances. That is, a general model is obtained by training, which can obtain the depth of field of the target object corresponding to the images collected by the above image acquisition devices with different field angles, and then through inverse normalization processing to obtain the real depth of field of the real-time target object .

In an optional embodiment, the processing unit includes a first determination module and a second determination module, and the first determination module is used to determine the real-time target object to calculate the depth of field and the real-time target to be determined according to the relevant parameters of the above-mentioned image acquisition device. The second determining module is configured to calculate the depth of field according to the ratio relationship and the real-time target, and determine the real depth of field of the real-time target. That is, after inverse normalization, the corresponding relationship between the calculated depth of field of the real-time target and the real depth of field of the real-time target is obtained. Since the depth of field is the distance information, the ratio between the calculated depth of field of the real-time target and the real depth of field of the real-time target is obtained. , and then calculate the depth of field and the ratio relationship according to the real-time target obtained by the normalized model to determine the real depth of field of the real-time target. Accurate determination of the real depth of field of real-time objects is achieved.

In an optional embodiment, the training unit includes a processing module and a first training module, and the processing module is used to perform filtering processing and threshold segmentation processing on each of the above-mentioned historical image data to obtain a processed image corresponding to each of the above-mentioned historical image data. Historical image data; the first training module is configured to use each of the processed historical image data and the depth of field of the historical object corresponding to each of the processed historical image data to perform training to obtain a normalized model. Specifically, the filtering process is to filter out the noise of historical image data, and the threshold segmentation process is to perform binarization segmentation. Using the processed historical image data is conducive to the training of the model and ensures the accuracy of the normalized model obtained by training.

In an optional embodiment, the first training module includes an extraction sub-module and a training sub-module, and the extraction sub-module is used to extract a variety of different characteristic parameters of the processed historical image data, and the processed historical A color channel of the image data represents a kind of above-mentioned characteristic parameter; the training submodule is used for adopting various above-mentioned characteristic parameters of each above-mentioned processed historical image data, and the above-mentioned historical target corresponding to each above-mentioned processed historical image data The depth of field of the object is trained to obtain a normalized model. The training of the model is actually to train the parameters, to extract a variety of different characteristic parameters of the above processed historical image data, and then to train to obtain an accurate normalized model.

In an optional embodiment, when there are three image acquisition devices, the training unit includes a building module and a second training module, and the building module is used to construct a training set, and the training set includes the first field of view image acquisition. The first amount of the above-mentioned historical image data captured by the device, the second amount of the above-mentioned historical image data captured by the second field of view image acquisition device, and the third amount of the above-mentioned historical image data captured by the third field of view image capture device Image data, wherein the first number is at least determined by the size of the first angle of view, the relative positional relationship between the image acquisition device of the first angle of view and the vehicle, and the relative positional relationship between the historical target and the vehicle The above-mentioned second quantity is at least determined by the size of the above-mentioned second angle of view, the relative positional relationship between the image acquisition device of the above-mentioned second angle of view and the above-mentioned vehicle, and the relative positional relationship between the above-mentioned historical target and the above-mentioned vehicle. The third quantity is determined at least by the size of the third angle of view, the relative positional relationship between the image acquisition device for the third angle of view and the vehicle, and the relative positional relationship between the historical target and the vehicle; the second training module It is used for training with the above-mentioned training set to obtain the above-mentioned normalized model. That is, the amount of data in the training set can be determined according to parameters such as the size of the field of view, the relative positional relationship between the image acquisition device and the above-mentioned vehicle, and the relative positional relationship between the above-mentioned historical target and the above-mentioned vehicle. The normalized model has better applicability and higher accuracy.

In an optional embodiment, the above-mentioned device further includes a third acquisition unit and an adjustment unit, and the third acquisition unit is used for using each of the above-mentioned historical image data and the depth of field of the above-mentioned historical object corresponding to each of the above-mentioned historical image data, During the training process, the error between the output result obtained by the normalized model and the depth of field of the historical target object is obtained; the adjustment unit is used to adjust the above-mentioned first quantity, the above-mentioned second quantity and the above-mentioned third quantity according to the above-mentioned error. at least one of them. That is, in order to ensure the accuracy of the parameters of the model during the training process, the first quantity, the above-mentioned second quantity and the above-mentioned third quantity can be adjusted according to the error between the output result obtained by the normalized model and the depth of field of the above-mentioned historical target. at least one of.

In an optional embodiment, the above-mentioned apparatus further includes a determination unit, which is configured to perform inverse normalization processing on the calculated depth of field of the above-mentioned real-time target by using the relevant parameters of the above-mentioned image acquisition device that captures the above-mentioned real-time image data, After the real depth of field of the real-time target object corresponding to the real-time image data is obtained, the driving speed and the driving acceleration of the vehicle are determined according to the real depth of field of the real-time target object. That is, according to the real depth of field of the real-time target, the real-time navigation is guided.

The device for determining the depth of field of an object includes a processor and a memory, the above-mentioned first acquisition unit, training unit, second acquisition unit and processing unit are all stored in the memory as program units, and the processor executes the above-mentioned program stored in the memory. unit to achieve the corresponding function.

The processor includes a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to one or more, by adjusting the kernel parameters to accurately determine the depth of field of the object.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory, such as read only memory (ROM) or flash memory (flash RAM), the memory including at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium includes a stored program, wherein when the program runs, the device where the computer-readable storage medium is located is controlled to execute the determining of the depth of field of the object Methods.

An embodiment of the present invention provides a processor for running a program, wherein the method for determining the depth of field of an object is executed when the program is running.

An embodiment of the present invention provides a vehicle, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more programs. Executed by a processor, the above-mentioned one or more programs include any one of the above-mentioned methods.

An embodiment of the present invention provides a system including the above vehicle and a plurality of image acquisition devices with different viewing angles, the image acquisition devices are installed on the vehicle, and the image acquisition device communicates with the vehicle.

An embodiment of the present invention provides a device. The device includes a processor, a memory, and a program stored in the memory and running on the processor. The processor implements at least the following steps when executing the program:

Step S101, obtaining a plurality of pieces of historical image data, the plurality of pieces of the above-mentioned historical image data are obtained by using a plurality of above-mentioned image acquisition devices with different field of view to shoot historical objects at the same distance and/or different distances from the above-mentioned vehicle within a historical time period obtained;

Step S102, using each of the above-mentioned historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned historical image data, perform training to obtain a normalized model;

Step S103, acquiring real-time image data, and using the above-mentioned normalization model to determine the real-time target object corresponding to the above-mentioned real-time image data to calculate the depth of field;

Step S104, using the relevant parameters of the image acquisition device that captures the real-time image data, to perform inverse normalization processing on the real-time target object to calculate the depth of field, to obtain the real depth of field of the real-time target object corresponding to the real-time image data.

The devices in this article can be servers, PCs, PADs, mobile phones, and so on.

The present application also provides a computer program product that, when executed on a data processing device, is adapted to execute a program initialized with at least the following method steps:

Step S101, obtaining a plurality of pieces of historical image data, the plurality of pieces of the above-mentioned historical image data are obtained by using a plurality of above-mentioned image acquisition devices with different field of view to shoot historical objects at the same distance and/or different distances from the above-mentioned vehicle within a historical time period obtained;

Step S102, using each of the above-mentioned historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned historical image data, perform training to obtain a normalized model;

Step S103, acquiring real-time image data, and using the above-mentioned normalization model to determine the real-time target object corresponding to the above-mentioned real-time image data to calculate the depth of field;

Step S104, using the relevant parameters of the image acquisition device that captures the real-time image data, to perform inverse normalization processing on the real-time target object to calculate the depth of field, to obtain the real depth of field of the real-time target object corresponding to the real-time image data.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory in the form of, for example, read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media includes both persistent and non-permanent, removable and non-removable media, and storage of information may be implemented by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture or apparatus that includes the element.

From the above description, it can be seen that the above-mentioned embodiments of the present application achieve the following technical effects:

1), the method for determining the depth of field of an object of the present application, by obtaining multiple pieces of historical image data, the multiple pieces of the above-mentioned historical image data are a plurality of above-mentioned image acquisition devices with different field of view, and the shooting distance is the same as the above-mentioned vehicle in the historical time period. distances and/or historical objects at different distances are obtained, using each of the above-mentioned historical image data and the depth of field of the above-mentioned historical objects corresponding to each of the above-mentioned historical image data, performing training to obtain a normalized model, and obtaining real-time image data, and The above-mentioned normalization model is used to determine the real-time target object corresponding to the above-mentioned real-time image data to calculate the depth of field, and the relevant parameters of the above-mentioned image acquisition device that captures the above-mentioned real-time image data are used to perform inverse normalization processing on the above-mentioned real-time target object to calculate the depth of field, The real depth of field of the real-time target object corresponding to the real-time image data is obtained. Since the normalized model is obtained by using a plurality of the above-mentioned image acquisition devices with different field of view to collect historical objects at the same distance and/or different distances from the above-mentioned vehicle, the obtained normalized model is suitable for images from different perspectives Acquisition equipment, targets at different distances. That is, a general model is obtained by training, which can obtain the depth of field of the target object corresponding to the images collected by the above image acquisition devices with different field angles, and then through inverse normalization processing to obtain the real depth of field of the real-time target object .

2), the device for determining the depth of field of an object of the present application, the first acquisition unit acquires a plurality of historical image data, and the above-mentioned historical image data is a plurality of above-mentioned image acquisition devices with different angles of view, and the shooting distance in the historical time period. Obtained from the historical objects at the same distance and/or different distances from the above-mentioned vehicles, the training unit uses each of the above-mentioned historical image data and the depth of field of the above-mentioned historical objects corresponding to each of the above-mentioned historical image data, and performs training to obtain a normalized model. The second acquisition unit acquires real-time image data, and uses the above-mentioned normalization model to determine the real-time target corresponding to the above-mentioned real-time image data to calculate the depth of field, and the processing unit uses the relevant parameters of the above-mentioned image acquisition device that captures the above-mentioned real-time image data. The target object calculates the depth of field and performs inverse normalization processing to obtain the real depth of field of the above-mentioned real-time target object corresponding to the above-mentioned real-time image data. Since the normalized model is obtained by using a plurality of the above-mentioned image acquisition devices with different field of view to collect historical objects at the same distance and/or different distances from the above-mentioned vehicle, the obtained normalized model is suitable for images from different perspectives Acquisition equipment, targets at different distances. That is, a general model is obtained by training, which can obtain the depth of field of the target object corresponding to the images collected by the above image acquisition devices with different field angles, and then through inverse normalization processing to obtain the real depth of field of the real-time target object .

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims (15)

1. A method for determining a depth of field of an object, the method being applied to a vehicle driving system including a vehicle and a plurality of image capturing devices mounted on the vehicle and having different angles of view, comprising:

acquiring a plurality of pieces of historical image data, wherein the plurality of pieces of historical image data are obtained by shooting historical target objects at the same distance and/or different distances away from the vehicle in a historical time period by adopting a plurality of image acquisition devices with different field angles;

training to obtain a normalized model by adopting each historical image data and the depth of field of the historical target corresponding to each historical image data;

acquiring real-time image data, and determining a real-time target object corresponding to the real-time image data by adopting the normalized model to calculate the depth of field;

And performing inverse normalization processing on the calculated depth of field of the real-time target object by using the related parameters of the image acquisition equipment for shooting the real-time image data to obtain the real depth of field of the real-time target object corresponding to the real-time image data.

2. The method according to claim 1, wherein the denormalizing the real-time object computed depth of field using parameters associated with the image capture device capturing the real-time image data to obtain the real depth of field of the real-time object corresponding to the real-time image data comprises:

determining a ratio relation between the calculated depth of field of the real-time target object and the real depth of field of the real-time target object to be determined according to the related parameters of the image acquisition equipment;

and calculating the depth of field according to the ratio relation and the real-time target object, and determining the real depth of field of the real-time target object.

3. The method of claim 1, wherein training using each of the historical image data and the depth of field of the historical object corresponding to each of the historical image data to obtain a normalized model comprises:

filtering and threshold segmentation processing are carried out on each historical image data to obtain processed historical image data corresponding to each historical image data;

And training to obtain a normalized model by adopting each processed historical image data and the depth of field of the historical target corresponding to each processed historical image data.

4. The method of claim 3, wherein training to obtain a normalized model using each of the processed historical image data and the depth of field of the historical object corresponding to each of the processed historical image data comprises:

extracting a plurality of different characteristic parameters of the processed historical image data, wherein one color channel of the processed historical image data represents one characteristic parameter;

and training to obtain a normalized model by adopting a plurality of different characteristic parameters of each processed historical image data and the depth of field of the historical target corresponding to each processed historical image data.

5. The method according to claim 1, wherein in a case where there are three image capturing devices, training using each of the historical image data and the depth of field of the historical target corresponding to each of the historical image data to obtain a normalized model includes:

Constructing a training set, wherein the training set comprises a first number of the historical image data captured by a first field angle image capture device, a second number of the historical image data captured by a second field angle image capture device, and a third number of the historical image data captured by a third field angle image capture device, wherein the first number is determined by at least the size of the first field angle, the relative positional relationship between the first field angle image capture device and the vehicle, and the relative positional relationship between the historical object and the vehicle, wherein the second number is determined by at least the size of the second field angle, the relative positional relationship between the second field angle image capture device and the vehicle, and the relative positional relationship between the historical object and the vehicle, and wherein the third number is determined by at least the size of the third field angle, The relative position relation between the third field angle image acquisition device and the vehicle and the relative position relation between the historical target object and the vehicle are determined;

and training by adopting the training set to obtain the normalized model.

6. The method of claim 5, wherein during training using each of the historical image data and the depth of field of the historical object corresponding to each of the historical image data, the method further comprises:

Acquiring an error between an output result obtained by the normalized model and the depth of field of the historical target object;

adjusting at least one of the first number, the second number, and the third number based on the error.

7. The method according to any one of claims 1 to 6, wherein after performing denormalization processing on the calculated depth of field of the real-time object using parameters related to the image capturing device capturing the real-time image data to obtain a true depth of field of the real-time object corresponding to the real-time image data, the method further comprises:

and determining the running speed and the running acceleration of the vehicle according to the real depth of field of the real-time target object.

8. The method according to any one of claims 1 to 6, wherein the relevant parameters of the image acquisition device comprise relative pose between coordinates of the image acquisition device and world coordinates, optical center position of the image acquisition device, distortion amount of the image acquisition device.

9. The method according to any of claims 1 to 6, wherein the field angle is one of:

30°、60°、90°、120°。

10. the method of any one of claims 1 to 6, wherein the normalized model is a convolutional neural network model comprising an input layer, an output layer, and a hidden layer.

11. An apparatus for determining depth of field of an object, the apparatus being applied to a vehicle driving system including a vehicle and a plurality of image capturing devices mounted on the vehicle and having different angles of view, comprising:

the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring a plurality of pieces of historical image data, and the plurality of pieces of historical image data are obtained by shooting historical target objects at the same distance and/or different distances away from the vehicle in a historical time period by adopting a plurality of image acquisition devices with different field angles;

the training unit is used for training by adopting the historical image data and the depth of field of the historical target object corresponding to the historical image data to obtain a normalized model;

the second acquisition unit is used for acquiring real-time image data and determining a real-time target object corresponding to the real-time image data by adopting the normalization model to calculate the depth of field;

and the processing unit is used for performing inverse normalization processing on the calculated depth of field of the real-time target object by using the related parameters of the image acquisition equipment for shooting the real-time image data to obtain the real depth of field of the real-time target object corresponding to the real-time image data.

12. A computer-readable storage medium, comprising a stored program, wherein the program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform the method of any one of claims 1 to 10.

13. A processor configured to run a program, wherein the program when executed performs the method of any one of claims 1 to 10.

14. A vehicle comprising one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any of claims 1-10.

15. A system comprising the vehicle of claim 14 and a plurality of image capturing devices of different field angles, the image capturing devices being mounted on the vehicle, the image capturing devices being in communication with the vehicle.

Images