WO2021128314A1

WO2021128314A1 - Image processing method and device, image processing system and storage medium

Info

Publication number: WO2021128314A1
Application number: PCT/CN2019/129364
Authority: WO
Inventors: 徐斌; 陈晓智
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-12-27
Filing date: 2019-12-27
Publication date: 2021-07-01
Also published as: CN113661513A

Abstract

An image processing method and device, an image processing system and a storage medium. Said method comprises: acquiring an initial image photographed by a photographing apparatus; processing the initial image, to determine a two-dimensional target image area of the initial image and semantic information of a target object contained in the two-dimensional target image area; according to a projection model of the two-dimensional target image area and the semantic information of the target object, determining the initial three-dimensional coordinates of the target object; performing area feature point extraction on the two-dimensional target image area, and determining, on the basis of the area feature points, three-dimensional coordinate adjustment information of the target object; and according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information, determining the target three-dimensional coordinates of the target object. By means of this implementation, three-dimensional target detection can be adaptively performed on different types of images, improving the efficiency and effectiveness of image processing.

Description

Image processing method, equipment, image processing system and storage medium

Technical field

The embodiments of the present invention relate to the field of image processing technology, and in particular to an image processing method, device, image processing system, and storage medium.

Background technique

At present, most of the three-dimensional target detection schemes for monocular images are for images obtained by ordinary small-hole imaging projection methods, and there are few schemes for detecting other projection models. If the ordinary monocular 3D target detection algorithm is directly applied to the image obtained by other projection methods (such as fisheye image), the 3D target detection effect of the image will be reduced. Therefore, how to more effectively perform three-dimensional target detection processing on different types of images has become an urgent problem to be solved.

Summary of the invention

The embodiment of the invention discloses an image processing method, equipment, image processing system and storage medium, which can adaptively perform three-dimensional target detection on different types of images, thereby improving the efficiency and effectiveness of image processing.

In the first aspect, an embodiment of the present invention provides an image processing method, including:

Acquiring an initial image taken by the photographing device;

Processing the initial image to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area;

Determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object;

Performing regional feature point extraction on the two-dimensional target image area, and determining the three-dimensional coordinate adjustment information of the target object based on the regional feature points;

Determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information.

In the second aspect, an embodiment of the present invention provides an image processing device, including: a memory and a processor,

The memory is used to store programs;

The processor is configured to execute a program stored in the memory, and when the program is executed, the processor is configured to:

Acquiring an initial image taken by the photographing device;

In a third aspect, an embodiment of the present invention provides an image processing system, including:

A photographing device and the image processing device described in the first aspect described above.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium in which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented. step.

The embodiment of the present invention can process the initial image captured by the shooting device to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, and according to the projection of the two-dimensional target image area The semantic information of the model and the target object, determine the initial three-dimensional coordinates of the target object, and extract the regional feature points of the two-dimensional target image area, and determine the three-dimensional coordinate adjustment information of the target object based on the regional feature points, so as to according to the initial three-dimensional coordinates And three-dimensional coordinate adjustment information to determine the target three-dimensional coordinates of the target object. Through this implementation, three-dimensional target detection can be adaptively performed on different types of images, which improves the efficiency and effectiveness of image processing.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative labor, other drawings can be obtained from these drawings.

Fig. 1 is a schematic structural diagram of an image processing system provided by an embodiment of the present invention;

2 is a schematic flowchart of an image processing method provided by an embodiment of the present invention;

3 is a schematic diagram of determining three-dimensional coordinates based on a small hole imaging model according to an embodiment of the present invention;

4 is a schematic diagram of determining three-dimensional coordinates based on an isometric projection model according to an embodiment of the present invention;

5 is a schematic flowchart of another image processing method provided by an embodiment of the present invention;

FIG. 6 is a schematic flowchart of another image processing method provided by an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of an image processing method provided by an embodiment of the present invention;

FIG. 8 is a schematic flowchart of yet another image processing method provided by an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of an image processing method provided by an embodiment of the present invention;

Fig. 10 is a schematic structural diagram of an image processing device provided by an embodiment of the present invention.

Detailed ways

The following clearly describes the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

In the following, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

In the fields of Advanced Driving Assistant System (ADAS), autonomous driving, drones and robot navigation, how to obtain the three-dimensional coordinate position of surrounding dynamic obstacles is a very important matter. At present, among many sensors, the vision camera can capture surrounding scene information. Among them, the monocular camera has a lower cost, so it is a sensor that has more choices. Therefore, how to perform three-dimensional detection of dynamic obstacles through a monocular camera, including information such as the three-dimensional coordinate position, three-dimensional size, and orientation of the obstacle, is a problem that needs to be solved urgently. At the same time, there are a variety of lenses and projection models. In addition to the common lens that uses small aperture imaging, there are also fisheye lenses with a super large field of view, and the projection method used is different from the usual aperture imaging method. Therefore, it is more difficult to detect three-dimensional targets on various monocular cameras.

In one embodiment, the two-dimensional or three-dimensional detection of the target object on the image mostly adopts the convolutional neural network in deep learning. For two-dimensional target detection, the convolutional neural network can be used to extract features from the image, and then the category and circumscribed rectangle of the corresponding target object can be estimated directly from the features.

For the three-dimensional target detection of a specific target object, the detection results can generally be divided into three aspects. One is the three-dimensional size of the target object, which is usually expressed by the length, width, and height of the circumscribed cuboid of the target object obtained by the detection. ; One is the three-dimensional position of the target object, usually represented by the center of the circumscribed cuboid on the left side of the camera coordinate system, and can be converted to coordinates in other coordinate systems if necessary; one is the orientation information of the target object, usually It is expressed by the pitch angle, yaw angle, and roll angle of the circumscribed cuboid. In the three-dimensional detection of the target object of the monocular camera, it is often necessary to use multiple frames of images at different moments in time sequence combined with the motion information of the camera or the movable platform equipped with the camera to achieve; if you want to use the single frame image of the monocular camera for the target Three-dimensional detection requires semantic recognition of the target, combined with the preset three-dimensional information experience value corresponding to the semantic information of the target object detection, so that the result obtained is relatively inaccurate and can only provide a rough three-dimensional information .

For example, if a monocular camera mounted on a vehicle captures an image around the vehicle, and the image contains a target object (such as another vehicle), then the image can be semantically recognized through the neural network to identify the target object If it is a car, it can be given preset three-dimensional information corresponding to the car, such as a height of 1.5 meters, and combine this preset information and the camera parameters of the monocular camera to finally obtain the three-dimensional information of the car.

In one embodiment, for any photographing device such as a camera, its parameters are known, such as focal length (f) and optical center coordinates (cx, cy). Therefore, when performing three-dimensional object detection, it can be based on its image The size of the upper part is a rough estimate of its three-dimensional coordinates. In layman's terms, it is the near-large and far-small. The smaller the object on the image, the farther the distance.

The embodiment of the present invention provides an image processing method, which processes the initial image captured by the camera to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, and According to the projection model of the two-dimensional target image area and the semantic information of the target object, the initial three-dimensional coordinates of the target object are determined, and the area feature points of the two-dimensional target image area are extracted, and the three-dimensional coordinates of the target object are determined based on the area feature points Adjust the information to determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information. Through this implementation, three-dimensional target detection can be adaptively performed on different types of images, which improves the efficiency and effectiveness of image processing.

The image processing method provided in the embodiments of the present invention may be executed by an image processing system, where the image processing system may include a photographing device and an image processing device. In some embodiments, the photographing device may be combined with the image processing system. Equipment integration; in some embodiments, the photographing device may be spatially independent of the image processing equipment, and the photographing device and the image processing equipment establish a communication connection in a wired or wireless manner. In some embodiments, the photographing device may be set on a movable platform configured with a load (such as a photographing device, an infrared detection device, a surveying instrument, etc.); in other embodiments, the photographing device may be spatially Independent of the movable platform. In some embodiments, the photographing device may include, but is not limited to, a sports camera, a panoramic camera, a fisheye camera, and the like. In some embodiments, the number of the photographing device may be one or more. In some embodiments, the movable platform includes movable equipment such as drones, robots capable of autonomous movement, unmanned vehicles, and unmanned ships. In some embodiments, the image processing device may include one or more of a smart phone, a tablet computer, a laptop computer, and a wearable device. In some embodiments, the embodiments of the present invention can be applied to the detection of obstacles in mobile platforms such as ADAS, autonomous driving, robots, drones, etc., to realize obstacle avoidance functions and subsequent path planning functions. In one embodiment, the movable platform may be a vehicle, the imaging device may be a fisheye camera, and the image processing device may be a computing platform mounted on the vehicle; specifically, the fisheye camera is installed on the vehicle The rearview mirror is used to obtain the environmental image of the side of the vehicle and then send it to the on-board computing platform for processing, and can provide a wider field of view than ordinary lenses, thereby reducing or avoiding the blind area of the camera's field of view.

The image processing system provided by the embodiment of the present invention will be schematically described below with reference to FIG. 1 as an example.

Please refer to FIG. 1, which is a schematic structural diagram of an image processing system provided by an embodiment of the present invention. The image processing system includes a photographing device 11 and an image processing device 12, wherein one of the photographing device 11 and the image processing device 12 The communication connection can be established through wireless communication connection. Among them, in some scenarios, a communication connection between the photographing device 11 and the image processing device 12 may also be established through a wired communication connection. In some embodiments, the photographing device 11 may be provided on the image processing device 12. In other embodiments, the photographing device 11 and the image processing device 12 are independent of each other, and the image processing device 12 may include one or more of a smart phone, a tablet computer, a laptop computer, and a wearable device.

In the embodiment of the present invention, the image processing device 12 may obtain the initial image taken by the photographing device 11, and process the initial image to determine the two-dimensional target image area of the initial image and the semantics of the target object contained in the two-dimensional target image area Information, and according to the projection model of the two-dimensional target image area and the semantic information of the target object, determine the initial three-dimensional coordinates of the target object, extract the area feature points of the two-dimensional target image area, and determine the three-dimensional target object based on the area feature points The coordinate adjustment information, and the target three-dimensional coordinates of the target object are determined according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information. Through this implementation, three-dimensional target detection can be adaptively performed on different types of images, which improves the efficiency and effectiveness of image processing.

The image processing method provided by the embodiment of the present invention will be schematically described below in conjunction with the accompanying drawings.

Please refer to FIG. 2 for details. FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present invention. The method may be executed by an image processing device, and the specific explanation of the image processing device is as described above. Specifically, the method of the embodiment of the present invention includes the following steps.

S201: Acquire an initial image photographed by the photographing device.

In the embodiment of the present invention, the image processing device may obtain the initial image captured by the photographing device. In some embodiments, the initial image may be obtained by photographing the target object by a photographing device. In some embodiments, the explanation of the photographing device is as described above, and will not be repeated here.

S202: Process the initial image, and determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area.

In the embodiment of the present invention, the image processing device may process the initial image to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area.

In an embodiment, the image processing device may perform image processing and semantic analysis on the initial image to determine the semantic information of the target object. In some embodiments, the semantic information includes, but is not limited to, the category and location information of the target object. In one example, the semantic information of the target object includes the semantics of different types of objects such as cars and drones. In some embodiments, the location information includes two-dimensional coordinates of the target object.

In one embodiment, when the image processing device processes the initial image to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, it may The initial image is processed according to the first neural network, and the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area are determined. In some embodiments, the first neural network is a convolutional neural network.

By determining the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, it is helpful to subsequently determine the initial three-dimensional coordinates of the target object.

S203: Determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object.

In the embodiment of the present invention, the image processing device may determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object.

In some embodiments, the projection model includes, but is not limited to, any one of a pinhole imaging model, a Snell window projection model, an isometric projection model, an isometric projection model, and a stereo projection model.

In one embodiment, the image processing device may acquire the two-dimensional target when determining the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object. The projection model of the image area, and obtain the parameters of the photographing device, and determine the two-dimensional target image according to the projection model of the two-dimensional target image area, the semantic information of the target object and the parameters of the photographing device The initial 3D coordinates of the area. In some embodiments, the projection model may be a preset projection model.

In some embodiments, the parameters of the photographing device include an internal parameter and an external parameter, the internal parameter includes the focal length of the photographing device, and the external parameter includes the optical center of the photographing device.

In one embodiment, when the image processing device acquires the projection model of the two-dimensional target image area, it may determine the category information of the two-dimensional target image area according to the feature point information of the two-dimensional target image area, And according to the category information of the two-dimensional target image area, a projection model of the two-dimensional target image area is determined.

For example, assuming that the image processing device determines that the category information of the two-dimensional target image area is a fisheye image according to the feature point information of the two-dimensional target image area, it may be determined that the projection model of the two-dimensional target image area is Any one of a Snell window projection model, an equal area projection model, an equal distance projection model, and a stereo projection model corresponding to the fisheye image.

In one embodiment, when the image processing device acquires the projection model of the two-dimensional target image area, it may process the initial image according to the first neural network to obtain the projection model of the initial image, and determine The projection model of the initial image is a projection model of the two-dimensional target image area.

In one embodiment, when the image processing device acquires the projection model of the two-dimensional target image area, it may process the initial image according to a third neural network to obtain the projection model of the initial image, and determine The projection model of the initial image is a projection model of the two-dimensional target image area. In some embodiments, the third neural network may be a convolutional neural network, and the third neural network is different from the first neural network.

In one embodiment, the projection model includes a first projection model; the image processing device determines according to the projection model of the two-dimensional target image area, the semantic information of the target object and the parameters of the photographing device When the initial three-dimensional coordinates of the two-dimensional target image area, the image height of the target object in the two-dimensional target image area and the actual height of the target object can be obtained, and the actual height of the target object can be obtained according to the image height, the actual height, The semantic information of the target object and the parameters of the photographing device are used to determine the initial three-dimensional coordinates of the two-dimensional target image area by using the first projection model. In some embodiments, the first projection model includes a pinhole imaging model.

In one embodiment, the image processing device uses the first projection model to determine the two-dimensional image based on the image height, the actual height, the semantic information of the target object, and the parameters of the photographing device. In the initial three-dimensional coordinates of the target image area, the first projection model may be used to determine the actual distance of the target object from the shooting device according to the image height, the actual height, and the parameters of the shooting device, and Obtain the two-dimensional coordinates of the target object in the image coordinate system, and use the first projection model according to the parameters of the photographing device, the actual distance, the two-dimensional coordinates, and the semantic information of the target object Determine the initial three-dimensional coordinates of the two-dimensional target image area.

Take FIG. 3 as an example for description. FIG. 3 is a schematic diagram of determining three-dimensional coordinates based on a small hole imaging model according to an embodiment of the present invention. As shown in Figure 3, f is the focal length of the camera, h is the image height of the target object in the two-dimensional target image area, H is the actual height of the target object, and D is the distance from the target object to the camera. From Figure 3, f/ D=h/H. For object detection, h is the height of the target object in the two-dimensional target image area. Since the category of the target object in each two-dimensional target image area can be predicted, H can be roughly obtained (for example, assuming that the target object is small For a car, it can be assumed that H is 1 meter), f is the camera's internal parameters, so the actual distance D of the target object from the camera can be calculated. That is, the coordinates of the target object in the Z direction in the camera coordinate system. In the same way, using the small hole imaging projection model, according to the camera's optical center cx, cy, and the actual distance D of the target object from the camera, the X and Y coordinates of the target object in the camera coordinate system can be calculated, as shown in the following formula (1) Show:

Among them, u and v are the coordinates of the target object in the image coordinate system, which can be approximated by using the coordinates of the center point of the target object.

In one embodiment, after the image processing device calculates the two-dimensional coordinates of the target object in the image coordinate system, it can use the two-dimensional coordinates of the target object in the image coordinate system, the camera internal parameters, and the semantic information of the target object, The initial three-dimensional coordinates of the two-dimensional target image area where the target object is located are calculated.

In one embodiment, the projection model includes a second projection model; the image processing device determines according to the projection model of the two-dimensional target image area, the semantic information of the target object, and the parameters of the photographing device When the initial three-dimensional coordinates of the two-dimensional target image area, the image distance of the target object in the two-dimensional target image area can be obtained, and the two-dimensional coordinates of the target object in the image coordinate system can be obtained, according to The image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the photographing device are used to determine the initial three-dimensional coordinates of the two-dimensional target image area by using the second projection model. In some embodiments, the second projection model includes any one of a Snell window projection model, an isometric projection model, an isometric projection model, and a stereo projection model; in other embodiments, the second projection model The two-projection model may also include other projection models, which are not specifically limited here.

In one embodiment, the image processing device uses the second projection model to determine the second projection model based on the image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the shooting device. When determining the initial three-dimensional coordinates of the target image area, the second projection model may be used to determine the shooting angle of view of the shooting device according to the image distance, the two-dimensional coordinates, and the parameters of the shooting device, and to obtain the The actual distance of the target object from the photographing device, so as to use the second projection model to determine the initial two-dimensional target image area according to the photographing angle of view, the actual distance, and the semantic information of the target object. Three-dimensional coordinates.

In one embodiment, the image processing device uses the second projection model to determine the initial two-dimensional target image area based on the shooting angle of view, the actual distance, and the semantic information of the target object. In the case of three-dimensional coordinates, the second projection model may be used to determine the physical distance of the target object according to the shooting angle of view and the actual distance, and the physical distance of the target object may be determined according to the parameters of the shooting device, the shooting angle of view, and the two-dimensional The coordinates, the physical distance and the semantic information of the target object determine the initial three-dimensional coordinates of the two-dimensional target image area.

Specifically, FIG. 4 can be used as an example for description. FIG. 4 is a schematic diagram of determining three-dimensional coordinates based on an isometric projection model according to an embodiment of the present invention. As shown in Figure 4, where f is the focal length of the camera, t is the distance of the target object in the two-dimensional target image area, r is the actual distance of the target object, and z is the distance from the camera to the object. According to the principle of isometric projection, there are the following Formula (2):

Among them, u, v are the pixel coordinates of the target object in the image coordinate system, and f, cx, and cy are the internal camera parameters.

For target object detection, h is the height of the target object in the two-dimensional target image area, and H is the actual height of the target object, then the following formula (3) exists:

In one embodiment, the category to which the target object in the two-dimensional target image area belongs can be predicted based on the semantic information of the target object, and then H can be roughly obtained (for example, if the target object is a car, it can be assumed that H is 1 meter) . It is known that u and v are the coordinates of the target object in the image coordinate system, which can be approximated by using the coordinates of the center point of the target object. Therefore, the θ can be calculated first, and then the r value can be calculated. Finally, the X, Y, and Z coordinates of the target object in the camera coordinate system can be obtained, that is, the initial three-dimensional coordinates.

In one embodiment, the above describes how to calculate the internal parameters of the camera, the two-dimensional coordinates of the target object in the image coordinate system, the semantic information of the target object, the image distance of the target object in the two-dimensional target image area, and the projection model. The initial three-dimensional coordinates of the target object mainly describe in detail the two models of small hole imaging and equidistant projection. For other projection models, the initial X, Y, and Z coordinates can also be calculated, which will not be repeated here.

In an embodiment, the image processing device may also detect the two-dimensional target image area according to a second neural network to obtain the area information of the two-dimensional target image area; wherein, the area information includes the Any one or more of category information, three-dimensional size information, orientation information, and two-dimensional circumscribed matrix information of the two-dimensional target image area. In some embodiments, the second neural network may be a convolutional neural network, and the second neural network is different from the first neural network and the second neural network.

S204: Perform area feature point extraction on the two-dimensional target image area, and determine three-dimensional coordinate adjustment information of the target object based on the area feature points.

In the embodiment of the present invention, the image processing device may perform area feature point extraction on the two-dimensional target image area, and determine the three-dimensional coordinate adjustment information of the target object based on the area feature points.

In one embodiment, when the image processing device extracts the area feature points of the two-dimensional target image area, it may process the two-dimensional target image area according to the second neural network to extract the two-dimensional target image area. Feature point information of the target image area. In some embodiments, the second neural network is a convolutional neural network. In some embodiments, the second neural network is different from the first neural network and the third neural network.

S205: Determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information.

In the embodiment of the present invention, the image processing device may determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information.

In one embodiment, when the image processing device determines the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information, the image processing device may compare the two-dimensional target image according to the second neural network. The area is processed, the three-dimensional coordinate adjustment information of the two-dimensional target image area is determined, and the target three-dimensional coordinates of the two-dimensional target image area are determined according to the three-dimensional coordinate adjustment information and the initial three-dimensional coordinates.

In one embodiment, when the image processing device determines the target three-dimensional coordinates of the two-dimensional target image area according to the three-dimensional coordinate adjustment information and the initial three-dimensional coordinates, it may perform an adjustment to the three-dimensional coordinate adjustment information. The initial three-dimensional coordinates are adjusted, and the adjusted three-dimensional coordinates are determined to be the target three-dimensional coordinates of the two-dimensional target image area.

In the embodiment of the present invention, the image processing equipment can process the initial image captured by the photographing device to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, and according to the two-dimensional The projection model of the target image area and the semantic information of the target object are used to determine the initial three-dimensional coordinates of the target object, and by extracting regional feature points from the two-dimensional target image area, the three-dimensional coordinate adjustment information of the target object is determined based on the regional feature points, thereby According to the initial 3D coordinates and 3D coordinate adjustment information, the target 3D coordinates of the target object are determined. Through this implementation, three-dimensional target detection can be adaptively performed on different types of images, which improves the efficiency and effectiveness of image processing.

Please refer to FIG. 5 for details. FIG. 5 is a schematic flowchart of another image processing method provided by an embodiment of the present invention. The method may be executed by an image processing device, and the specific explanation of the image processing device is as described above. Specifically, the method of the embodiment of the present invention includes the following steps.

S501: Acquire an initial image photographed by the photographing device.

In the embodiment of the present invention, the image processing device may obtain the initial image captured by the photographing device. In some embodiments, the explanation of the photographing device is as described above, and will not be repeated here.

S502: Process the initial image, and determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area.

In the embodiment of the present invention, the image processing device may process the initial image to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area. The specific implementation is as described above and will not be repeated here.

S503: Acquire a projection model of the two-dimensional target image area, where the projection model is a preset projection model.

In the embodiment of the present invention, the image processing device may obtain a projection model of the two-dimensional target image area, and the projection model is a preset projection model. In some embodiments, the explanation of the projection model is as described above, and will not be repeated here.

S504: Determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object.

In the embodiment of the present invention, the image processing device may determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object. The specific implementation is as described above and will not be repeated here.

S505: Perform area feature point extraction on the two-dimensional target image area, and determine three-dimensional coordinate adjustment information of the target object based on the area feature points.

In the embodiment of the present invention, the image processing device may perform area feature point extraction on the two-dimensional target image area, and determine the three-dimensional coordinate adjustment information of the target object based on the area feature points. The specific implementation is as described above and will not be repeated here.

S506: Determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information.

In the embodiment of the present invention, the image processing device may determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information. The specific implementation is as described above and will not be repeated here.

In the embodiment of the present invention, the image processing equipment can process the initial image captured by the photographing device to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, and according to the preset The projection model and the semantic information of the target object are used to determine the initial three-dimensional coordinates of the target object, and by extracting regional feature points from the two-dimensional target image area to determine the three-dimensional coordinate adjustment information of the target object based on the regional feature points, so as to adjust the information according to the initial three-dimensional Coordinates and three-dimensional coordinate adjustment information determine the target three-dimensional coordinates of the target object. Through this implementation, three-dimensional target detection can be adaptively performed on different types of images, which improves the efficiency and effectiveness of image processing.

Please refer to FIG. 6 for details. FIG. 6 is a schematic flowchart of another image processing method provided by an embodiment of the present invention. The method may be executed by an image processing device, and the specific explanation of the image processing device is as described above. Specifically, the method of the embodiment of the present invention includes the following steps.

S601: Acquire an initial image photographed by the photographing device.

S602: Process the initial image, and determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area.

S603: Process the initial image according to the third neural network to obtain a projection model of the initial image, and determine that the projection model of the initial image is the projection model of the two-dimensional target image area.

In the embodiment of the present invention, the image processing device may process the initial image according to a third neural network to obtain a projection model of the initial image, and determine that the projection model of the initial image is the two-dimensional target image The projection model of the area.

Specifically, FIG. 7 can be taken as an example. FIG. 7 is a schematic structural diagram of an image processing method provided by an embodiment of the present invention. As shown in FIG. 7, the image processing device may process the initial image 72 through the first neural network 71. To obtain a two-dimensional target image area 73. The two-dimensional target image area is detected according to the second neural network 74 to obtain the area information 75 of the two-dimensional target image area, where the area information 75 includes category information, three-dimensional size information, and orientation of the two-dimensional target image area. Any one or more of information and two-dimensional circumscribed matrix information. The projection model 77 of the two-dimensional target image area is determined by the third neural network 76, wherein the projection model 77 is at least one determined from a plurality of projection models. Determine the initial three-dimensional coordinates 791 of the target object according to the projection model 77 of the two-dimensional target image area, the semantic information of the target object and the parameters 78 of the photographing device; perform regional feature points on the two-dimensional target image area Extract and determine the three-dimensional coordinate adjustment information 792 of the target object based on the regional feature points; determine the target three-dimensional coordinate 710 of the target object according to the initial three-dimensional coordinate 791 and the three-dimensional coordinate adjustment information 792.

Through this implementation mode, the corresponding projection model can be automatically selected, and various projection models can be adapted within the same algorithm framework to adapt to all types of monocular images, which helps to improve the efficiency and efficiency of image processing. Effectiveness.

S604: Determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object.

S605: Perform area feature point extraction on the two-dimensional target image area, and determine three-dimensional coordinate adjustment information of the target object based on the area feature points.

S606: Determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information.

In the embodiment of the present invention, the image processing device may determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information. The specific implementation is as described above, and will not be repeated here.

In the embodiment of the present invention, the image processing device may process the initial image captured by the photographing device to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, and according to the passage The projection model of the two-dimensional target image area and the semantic information of the target object obtained by the three-neural network processing the initial image, determine the initial three-dimensional coordinates of the target object, and extract the regional feature points of the two-dimensional target image area to The three-dimensional coordinate adjustment information of the target object is determined based on the regional feature points, so that the target three-dimensional coordinate of the target object is determined according to the initial three-dimensional coordinate and the three-dimensional coordinate adjustment information. Through this implementation manner, the projection model can be determined through a neural network to adaptively perform three-dimensional target detection on different types of images, which improves the efficiency and effectiveness of image processing.

Please refer to FIG. 8 for details. FIG. 8 is a schematic flowchart of another image processing method provided by an embodiment of the present invention. The method may be executed by an image processing device, and the specific explanation of the image processing device is as described above. Specifically, the method of the embodiment of the present invention includes the following steps.

S801: Acquire an initial image photographed by the photographing device.

S802: Process the initial image, and determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area.

S803: Process the initial image according to the first neural network to obtain a projection model of the initial image, and determine that the projection model of the initial image is the projection model of the two-dimensional target image area.

In the embodiment of the present invention, the image processing device may process the initial image according to the first neural network to obtain the projection model of the initial image, and determine that the projection model of the initial image is the one of the two-dimensional target image area. Projection model.

In one embodiment, the image processing device may process the two-dimensional target image area according to the second neural network to extract feature point information of the two-dimensional target image area, and according to the characteristics of the two-dimensional target image area The feature point information determines the category information of the two-dimensional target image area, and the projection model of the two-dimensional target image area is determined according to the category information of the two-dimensional target image area.

Specifically, FIG. 9 is an example. FIG. 9 is a schematic structural diagram of an image processing method provided by an embodiment of the present invention. As shown in FIG. Processing is performed to extract feature point information 93 of the two-dimensional target image area 92. The two-dimensional target image area is detected according to the second neural network 94 to obtain the area information 95 of the two-dimensional target image area. The projection model 96 of the two-dimensional target image area 92 is determined by the first neural network 91, and the projection model 96 of the two-dimensional target image area, the parameters 97 of the photographing device, and the semantic information of the target object are determined to determine the The initial three-dimensional coordinates 981 of the target object; perform regional feature point extraction on the two-dimensional target image area, and determine the three-dimensional coordinate adjustment information 982 of the target object based on the regional feature points; according to the initial three-dimensional coordinates 981 and the The three-dimensional coordinate adjustment information 982 determines the target three-dimensional coordinate 99 of the target object.

Through this embodiment, the neural network for judging the projection model and the neural network for extracting the feature point information of the two-dimensional target image area can be shared, reducing the amount of calculation and complexity, and further improving the efficiency of image processing.

S804: Determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object.

In one embodiment, the imaging device may obtain the parameters of the imaging device when determining the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object, and The initial three-dimensional coordinates of the two-dimensional target image area are determined according to the projection model of the two-dimensional target image area, the semantic information of the target object, and the parameters of the photographing device. In some embodiments, the parameters of the photographing device include an internal parameter and an external parameter, the internal parameter includes the focal length of the photographing device, and the external parameter includes the optical center of the photographing device.

S805: Perform area feature point extraction on the two-dimensional target image area, and determine three-dimensional coordinate adjustment information of the target object based on the area feature points.

S806: Determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information.

In the embodiment of the present invention, the image processing equipment may process the initial image captured by the photographing device to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, and according to the first The neural network processes the initial image to obtain a projection model of the two-dimensional target image area, and according to the projection model and the semantic information of the target object, the initial three-dimensional coordinates of the target object are determined, and the two-dimensional target image area is determined by The regional feature point is extracted to determine the three-dimensional coordinate adjustment information of the target object based on the regional feature point, so as to determine the target three-dimensional coordinate of the target object according to the initial three-dimensional coordinate and the three-dimensional coordinate adjustment information. Through this embodiment, the projection model of the two-dimensional target image area can be determined by the category information of the two-dimensional target image area determined according to the feature point information, so as to adaptively perform three-dimensional target detection on different types of images, thereby improving the image The efficiency and effectiveness of processing.

Please refer to FIG. 10, which is a schematic structural diagram of an image processing device according to an embodiment of the present invention. Specifically, the image processing device includes: a memory 1001 and a processor 1002.

In an embodiment, the image processing device further includes a data interface 1003, and the data interface 1003 is used to transfer data information between the image processing device and other devices.

The memory 1001 may include a volatile memory (volatile memory); the memory 1001 may also include a non-volatile memory (non-volatile memory); the memory 1001 may also include a combination of the foregoing types of memories. The processor 1002 may be a central processing unit (CPU). The processor 1002 may further include a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), or any combination thereof.

The memory 1001 is used to store programs, and the processor 1002 can call the programs stored in the memory 1001 to perform the following steps:

Acquiring an initial image taken by the photographing device;

Further, when the processor 1002 processes the initial image to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, it is specifically used for:

The initial image is processed according to the first neural network, and the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area are determined.

Further, when the processor 1002 performs regional feature point extraction on the two-dimensional target image region, it is specifically configured to:

The two-dimensional target image area is processed according to the second neural network to extract feature point information of the two-dimensional target image area.

Further, the processor 1002 is further configured to:

Detecting the two-dimensional target image area according to the second neural network to obtain the area information of the two-dimensional target image area;

Wherein, the area information includes any one or more of category information, three-dimensional size information, orientation information, and two-dimensional circumscribed matrix information of the two-dimensional target image area.

Further, when the processor 1002 determines the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object, it is specifically configured to:

Acquiring a projection model of the two-dimensional target image area;

Acquiring the parameters of the photographing device;

The initial three-dimensional coordinates of the two-dimensional target image area are determined according to the projection model of the two-dimensional target image area, the semantic information of the target object, and the parameters of the photographing device.

Further, the projection model is a preset projection model.

Further, when the processor 1002 acquires the projection model of the two-dimensional target image area, it is specifically configured to:

Determining the category information of the two-dimensional target image area according to the feature point information of the two-dimensional target image area;

According to the category information of the two-dimensional target image area, a projection model of the two-dimensional target image area is determined.

Further, when the processor 1002 obtains the projection model of the two-dimensional target image area, it is specifically configured to:

Processing the initial image according to the first neural network to obtain a projection model of the initial image;

It is determined that the projection model of the initial image is the projection model of the two-dimensional target image area.

Processing the initial image according to a third neural network to obtain a projection model of the initial image;

Further, the projection model includes a first projection model; the processor 1002 determines the two-dimensional projection model according to the projection model of the two-dimensional target image area, the semantic information of the target object, and the parameters of the photographing device The initial three-dimensional coordinates of the target image area are specifically used for:

Acquiring the image height of the target object in the two-dimensional target image area and the actual height of the target object;

The first projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the image height, the actual height, the semantic information of the target object, and the parameters of the photographing device.

Further, the processor 1002 uses the first projection model to determine the size of the two-dimensional target image area according to the image height, the actual height, the semantic information of the target object, and the parameters of the photographing device. When initial three-dimensional coordinates, it is specifically used for:

Using the first projection model to determine the actual distance of the target object from the shooting device according to the image height, the actual height, and the parameters of the shooting device;

Acquiring the two-dimensional coordinates of the target object in the image coordinate system;

According to the parameters of the photographing device, the actual distance, the two-dimensional coordinates, and the semantic information of the target object, the first projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area.

Further, the projection model includes a second projection model; the processor 702 determines the two-dimensional projection model according to the projection model of the two-dimensional target image area, the semantic information of the target object, and the parameters of the photographing device The initial three-dimensional coordinates of the target image area are specifically used for:

Acquiring the image distance of the target object in the two-dimensional target image area;

The second projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the photographing device.

Further, the processor 1002 uses the second projection model to determine the two-dimensional target image area according to the image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the shooting device. When the initial three-dimensional coordinates are specifically used for:

Using the second projection model to determine the shooting angle of view of the shooting device according to the image distance, the two-dimensional coordinates, and the parameters of the shooting device;

Acquiring the actual distance of the target object from the photographing device;

The second projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the shooting angle of view, the actual distance, and the semantic information of the target object.

Further, when the processor 1002 uses the second projection model to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the shooting angle of view, the actual distance, and the semantic information of the target object, Specifically used for:

Determine the physical distance of the target object by using the second projection model according to the shooting angle of view and the actual distance;

Determine the initial three-dimensional coordinates of the two-dimensional target image area according to the parameters of the photographing device, the photographing angle of view, the two-dimensional coordinates, the physical distance, and the semantic information of the target object.

Further, when the processor 1002 determines the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information, it is specifically configured to:

Processing the two-dimensional target image area according to the second neural network, and determining the three-dimensional coordinate adjustment information of the two-dimensional target image area;

Determine the target three-dimensional coordinates of the two-dimensional target image area according to the three-dimensional coordinate adjustment information and the initial three-dimensional coordinates.

Further, the parameters of the shooting device include internal parameters and external parameters, the internal parameters include the focal length of the shooting device, and the external parameters include the optical center of the shooting device.

Further, the first projection model includes a small hole imaging model.

Further, the second projection model includes any one of a Snell window projection model, an isometric projection model, an isometric projection model, and a stereo projection model.

In the embodiment of the present invention, the image processing equipment can process the initial image captured by the photographing device to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, and according to the two-dimensional The projection model of the target image area and the semantic information of the target object are used to determine the initial three-dimensional coordinates of the target object, and by extracting regional feature points from the two-dimensional target image area to determine the three-dimensional coordinate adjustment information of the target object based on the regional feature points, thereby According to the initial 3D coordinates and 3D coordinate adjustment information, the target 3D coordinates of the target object are determined. Through this implementation manner, various projection models can be adapted within the same algorithm framework to adapt to all types of monocular images, which improves the efficiency and effectiveness of image processing.

An embodiment of the present invention also provides an image processing system, including: a photographing device and the above-mentioned image processing equipment. In the embodiment of the present invention, the image processing equipment can process the initial image captured by the photographing device to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area, and according to the two-dimensional The projection model of the target image area and the semantic information of the target object are used to determine the initial three-dimensional coordinates of the target object, and by extracting regional feature points from the two-dimensional target image area to determine the three-dimensional coordinate adjustment information of the target object based on the regional feature points, thereby According to the initial 3D coordinates and 3D coordinate adjustment information, the target 3D coordinates of the target object are determined. Through this implementation, three-dimensional target detection can be adaptively performed on different types of images, which improves the efficiency and effectiveness of image processing.

The embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the present invention is implemented as shown in FIG. 2, FIG. 5, FIG. 6 or FIG. The method described in the embodiment corresponding to 8 can also implement the device in the embodiment corresponding to the present invention described in FIG. 10, and will not be repeated here.

The computer-readable storage medium may be an internal storage unit of the device described in any of the foregoing embodiments, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a smart memory card (Smart Media Card, SMC), or a Secure Digital (SD) card. , Flash Card, etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

The above-disclosed are only some of the embodiments of the present invention, which of course cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope of the present invention.

Claims

An image processing method, characterized in that it comprises:

Acquiring an initial image taken by the photographing device;

Processing the initial image to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area;

Determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object;

Performing regional feature point extraction on the two-dimensional target image area, and determining the three-dimensional coordinate adjustment information of the target object based on the regional feature points;

Determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information.
The method according to claim 1, wherein the processing of the initial image to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area ,include:

The initial image is processed according to the first neural network, and the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area are determined.
The method according to claim 1, wherein said extracting regional feature points of said two-dimensional target image region comprises:

The two-dimensional target image area is processed according to the second neural network to extract feature point information of the two-dimensional target image area.
The method according to claim 1, wherein the method further comprises:

Detecting the two-dimensional target image area according to the second neural network to obtain the area information of the two-dimensional target image area;

Wherein, the area information includes any one or more of category information, three-dimensional size information, orientation information, and two-dimensional circumscribed matrix information of the two-dimensional target image area.
The method according to claim 1 or 2, wherein the determining the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object comprises:

Acquiring a projection model of the two-dimensional target image area;

Acquiring the parameters of the photographing device;

The initial three-dimensional coordinates of the two-dimensional target image area are determined according to the projection model of the two-dimensional target image area, the semantic information of the target object, and the parameters of the photographing device.
The method according to claim 5, wherein the projection model is a preset projection model.
The method according to claim 5, wherein said obtaining the projection model of the two-dimensional target image area comprises:

Determining the category information of the two-dimensional target image area according to the feature point information of the two-dimensional target image area;

According to the category information of the two-dimensional target image area, a projection model of the two-dimensional target image area is determined.
The method according to claim 5, wherein said obtaining the projection model of the two-dimensional target image area comprises:

Processing the initial image according to the first neural network to obtain a projection model of the initial image;

It is determined that the projection model of the initial image is the projection model of the two-dimensional target image area.
The method according to claim 5, wherein said obtaining the projection model of the two-dimensional target image area comprises:

Processing the initial image according to a third neural network to obtain a projection model of the initial image;

It is determined that the projection model of the initial image is the projection model of the two-dimensional target image area.
The method according to any one of claims 5-9, wherein the projection model comprises a first projection model; the projection model according to the two-dimensional target image area, the semantic information of the target object, and The parameters of the photographing device to determine the initial three-dimensional coordinates of the two-dimensional target image area include:

Acquiring the image height of the target object in the two-dimensional target image area and the actual height of the target object;

The first projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the image height, the actual height, the semantic information of the target object, and the parameters of the photographing device.
The method according to claim 10, wherein the first projection model is used to determine the location based on the image height, the actual height, the semantic information of the target object, and the parameters of the shooting device. The initial three-dimensional coordinates of the two-dimensional target image area include:

Using the first projection model to determine the actual distance of the target object from the shooting device according to the image height, the actual height, and the parameters of the shooting device;

Acquiring the two-dimensional coordinates of the target object in the image coordinate system;

According to the parameters of the photographing device, the actual distance, the two-dimensional coordinates, and the semantic information of the target object, the first projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area.
The method according to any one of claims 5-9, wherein the projection model comprises a second projection model; the projection model according to the two-dimensional target image area, the semantic information of the target object, and The parameters of the photographing device to determine the initial three-dimensional coordinates of the two-dimensional target image area include:

Acquiring the image distance of the target object in the two-dimensional target image area;

Acquiring the two-dimensional coordinates of the target object in the image coordinate system;

The second projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the photographing device.
The method according to claim 12, wherein the second projection model is used to determine according to the image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the photographing device The initial three-dimensional coordinates of the two-dimensional target image area include:

Using the second projection model to determine the shooting angle of view of the shooting device according to the image distance, the two-dimensional coordinates, and the parameters of the shooting device;

Acquiring the actual distance of the target object from the photographing device;

The second projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the shooting angle of view, the actual distance, and the semantic information of the target object.
The method according to claim 13, wherein the second projection model is used to determine the two-dimensional target image according to the shooting angle of view, the actual distance, and semantic information of the target object The initial three-dimensional coordinates of the area, including:

Determine the physical distance of the target object by using the second projection model according to the shooting angle of view and the actual distance;

Determine the initial three-dimensional coordinates of the two-dimensional target image area according to the parameters of the photographing device, the photographing angle of view, the two-dimensional coordinates, the physical distance, and the semantic information of the target object.
The method according to claim 4, wherein the determining the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information comprises:

Processing the two-dimensional target image area according to the second neural network, and determining the three-dimensional coordinate adjustment information of the two-dimensional target image area;

Determine the target three-dimensional coordinates of the two-dimensional target image area according to the three-dimensional coordinate adjustment information and the initial three-dimensional coordinates.
The method according to claim 1, wherein the parameters of the shooting device include internal parameters and external parameters, the internal parameters include the focal length of the shooting device, and the external parameters include the optical center of the shooting device.
The method according to claim 10, wherein the first projection model comprises a pinhole imaging model.
The method according to claim 12, wherein the second projection model comprises any one of a Snell window projection model, an equal area projection model, an equal distance projection model, and a stereo projection model.
An image processing device, characterized by comprising: a memory and a processor,

The memory is used to store programs;

The processor is configured to execute a program stored in the memory, and when the program is executed, the processor is configured to:

Acquiring an initial image taken by the photographing device;

Processing the initial image to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area;

Determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object;

Performing regional feature point extraction on the two-dimensional target image area, and determining the three-dimensional coordinate adjustment information of the target object based on the regional feature points;

Determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information.
The device according to claim 19, wherein the processor processes the initial image to determine the two-dimensional target image area of the initial image and the target object contained in the two-dimensional target image area. When semantic information, it is specifically used for:

The initial image is processed according to the first neural network, and the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area are determined.
The device according to claim 19, wherein the processor is specifically configured to: when performing region feature point extraction on the two-dimensional target image region:

The two-dimensional target image area is processed according to the second neural network to extract feature point information of the two-dimensional target image area.
The device according to claim 19, wherein the processor is further configured to:

Detecting the two-dimensional target image area according to the second neural network to obtain the area information of the two-dimensional target image area;

Wherein, the area information includes any one or more of category information, three-dimensional size information, orientation information, and two-dimensional circumscribed matrix information of the two-dimensional target image area.
The device according to claim 19 or 21, wherein when the processor determines the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object, Specifically used for:

Acquiring a projection model of the two-dimensional target image area;

Acquiring the parameters of the photographing device;

The initial three-dimensional coordinates of the two-dimensional target image area are determined according to the projection model of the two-dimensional target image area, the semantic information of the target object, and the parameters of the photographing device.
The device according to claim 23, wherein the projection model is a preset projection model.
The device according to claim 23, wherein when the processor obtains the projection model of the two-dimensional target image area, it is specifically configured to:

Determining the category information of the two-dimensional target image area according to the feature point information of the two-dimensional target image area;

According to the category information of the two-dimensional target image area, a projection model of the two-dimensional target image area is determined.
The device according to claim 23, wherein when the processor obtains the projection model of the two-dimensional target image area, it is specifically configured to:

Processing the initial image according to the first neural network to obtain a projection model of the initial image;

It is determined that the projection model of the initial image is the projection model of the two-dimensional target image area.
The device according to claim 23, wherein when the processor obtains the projection model of the two-dimensional target image area, it is specifically configured to:

Processing the initial image according to a third neural network to obtain a projection model of the initial image;

It is determined that the projection model of the initial image is the projection model of the two-dimensional target image area.
The device according to any one of claims 23-27, wherein the projection model comprises a first projection model; the processor is based on the projection model of the two-dimensional target image area and the semantics of the target object. The information and the parameters of the photographing device are specifically used to determine the initial three-dimensional coordinates of the two-dimensional target image area:

Acquiring the image height of the target object in the two-dimensional target image area and the actual height of the target object;

The first projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the image height, the actual height, the semantic information of the target object, and the parameters of the photographing device.
The device according to claim 28, wherein the processor uses the first projection model according to the image height, the actual height, the semantic information of the target object, and the parameters of the shooting device When determining the initial three-dimensional coordinates of the two-dimensional target image area, it is specifically used for:

Using the first projection model to determine the actual distance of the target object from the shooting device according to the image height, the actual height, and the parameters of the shooting device;

Acquiring the two-dimensional coordinates of the target object in the image coordinate system;

According to the parameters of the photographing device, the actual distance, the two-dimensional coordinates, and the semantic information of the target object, the first projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area.
The device according to any one of claims 23-27, wherein the projection model comprises a second projection model; the processor is based on the projection model of the two-dimensional target image area and the semantics of the target object The information and the parameters of the photographing device are specifically used to determine the initial three-dimensional coordinates of the two-dimensional target image area:

Acquiring the image distance of the target object in the two-dimensional target image area;

Acquiring the two-dimensional coordinates of the target object in the image coordinate system;

The second projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the photographing device.
The device according to claim 30, wherein the processor uses the second projection according to the image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the shooting device. When the model determines the initial three-dimensional coordinates of the two-dimensional target image area, it is specifically used for:

Using the second projection model to determine the shooting angle of view of the shooting device according to the image distance, the two-dimensional coordinates, and the parameters of the shooting device;

Acquiring the actual distance of the target object from the photographing device;

The second projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the shooting angle of view, the actual distance, and the semantic information of the target object.
The device according to claim 31, wherein the processor uses the second projection model to determine the two-dimensional image based on the shooting angle of view, the actual distance, and semantic information of the target object. The initial three-dimensional coordinates of the target image area are specifically used for:

Determine the physical distance of the target object by using the second projection model according to the shooting angle of view and the actual distance;

Determine the initial three-dimensional coordinates of the two-dimensional target image area according to the parameters of the photographing device, the photographing angle of view, the two-dimensional coordinates, the physical distance, and the semantic information of the target object.
The device according to claim 22, wherein the processor is specifically configured to: when determining the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information:

Processing the two-dimensional target image area according to the second neural network, and determining the three-dimensional coordinate adjustment information of the two-dimensional target image area;

Determine the target three-dimensional coordinates of the two-dimensional target image area according to the three-dimensional coordinate adjustment information and the initial three-dimensional coordinates.
The device according to claim 19, wherein the parameters of the shooting device include internal parameters and external parameters, the internal parameters include the focal length of the shooting device, and the external parameters include the optical center of the shooting device.
The device of claim 28, wherein the first projection model comprises a small hole imaging model.
The device according to claim 30, wherein the second projection model comprises any one of a Snell window projection model, an equal area projection model, an equal distance projection model, and a stereo projection model.
An image processing system, characterized in that it comprises:

The photographing device is used for photographing to obtain the initial image;

Image processing equipment for:

Acquiring an initial image taken by the photographing device;

Processing the initial image to determine the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area;

Determine the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object;

Performing regional feature point extraction on the two-dimensional target image area, and determining the three-dimensional coordinate adjustment information of the target object based on the regional feature points;

Determine the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information.
The system according to claim 37, wherein the image processing device processes the initial image to determine a two-dimensional target image area of the initial image and a target object contained in the two-dimensional target image area When the semantic information is specifically used for:

The initial image is processed according to the first neural network, and the two-dimensional target image area of the initial image and the semantic information of the target object contained in the two-dimensional target image area are determined.
The system according to claim 37, wherein when the image processing device extracts regional feature points of the two-dimensional target image area, it is specifically configured to:

The two-dimensional target image area is processed according to the second neural network to extract feature point information of the two-dimensional target image area.
The system according to claim 37, wherein the image processing device is further used for:

Detecting the two-dimensional target image area according to the second neural network to obtain the area information of the two-dimensional target image area;

Wherein, the area information includes any one or more of category information, three-dimensional size information, orientation information, and two-dimensional circumscribed matrix information of the two-dimensional target image area.
The system according to claim 37 or 39, wherein the image processing device determines the initial three-dimensional coordinates of the target object according to the projection model of the two-dimensional target image area and the semantic information of the target object. , Specifically used for:

Acquiring a projection model of the two-dimensional target image area;

Acquiring the parameters of the photographing device;

The initial three-dimensional coordinates of the two-dimensional target image area are determined according to the projection model of the two-dimensional target image area, the semantic information of the target object, and the parameters of the photographing device.
The system according to claim 41, wherein the projection model is a preset projection model.
The system according to claim 41, wherein when the image processing device acquires the projection model of the two-dimensional target image area, it is specifically used for:

Determining the category information of the two-dimensional target image area according to the feature point information of the two-dimensional target image area;

According to the category information of the two-dimensional target image area, a projection model of the two-dimensional target image area is determined.
The system according to claim 41, wherein when the image processing device acquires the projection model of the two-dimensional target image area, it is specifically used for:

Processing the initial image according to the first neural network to obtain a projection model of the initial image;

It is determined that the projection model of the initial image is the projection model of the two-dimensional target image area.
The system according to claim 41, wherein when the image processing device acquires the projection model of the two-dimensional target image area, it is specifically used for:

Processing the initial image according to a third neural network to obtain a projection model of the initial image;

It is determined that the projection model of the initial image is the projection model of the two-dimensional target image area.
The system according to any one of claims 41-45, wherein the projection model comprises a first projection model; the image processing device is based on the projection model of the two-dimensional target image area and the projection of the target object The semantic information and the parameters of the photographing device are specifically used to determine the initial three-dimensional coordinates of the two-dimensional target image area:

Acquiring the image height of the target object in the two-dimensional target image area and the actual height of the target object;

The first projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the image height, the actual height, the semantic information of the target object, and the parameters of the photographing device.
The system according to claim 46, wherein the image processing device uses the first projection according to the image height, the actual height, the semantic information of the target object, and the parameters of the shooting device. When the model determines the initial three-dimensional coordinates of the two-dimensional target image area, it is specifically used for:

Using the first projection model to determine the actual distance of the target object from the shooting device according to the image height, the actual height, and the parameters of the shooting device;

Acquiring the two-dimensional coordinates of the target object in the image coordinate system;

According to the parameters of the photographing device, the actual distance, the two-dimensional coordinates, and the semantic information of the target object, the first projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area.
The system according to any one of claims 41-45, wherein the projection model comprises a second projection model; the image processing device is based on the projection model of the two-dimensional target image area and the projection of the target object The semantic information and the parameters of the photographing device are specifically used to determine the initial three-dimensional coordinates of the two-dimensional target image area:

Acquiring the image distance of the target object in the two-dimensional target image area;

Acquiring the two-dimensional coordinates of the target object in the image coordinate system;

The second projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the photographing device.
The system according to claim 48, wherein the image processing device uses the second image processing device according to the image distance, the two-dimensional coordinates, the semantic information of the target object, and the parameters of the photographing device. When the projection model determines the initial three-dimensional coordinates of the two-dimensional target image area, it is specifically used for:

Using the second projection model to determine the shooting angle of view of the shooting device according to the image distance, the two-dimensional coordinates, and the parameters of the shooting device;

Acquiring the actual distance of the target object from the photographing device;

The second projection model is used to determine the initial three-dimensional coordinates of the two-dimensional target image area according to the shooting angle of view, the actual distance, and the semantic information of the target object.
The system according to claim 49, wherein the image processing device uses the second projection model to determine the two according to the shooting angle of view, the actual distance, and semantic information of the target object. When the initial three-dimensional coordinates of the target image area are dimensioned, it is specifically used for:

Using the second projection model to determine the physical distance of the target object according to the shooting angle of view and the actual distance;

Determine the initial three-dimensional coordinates of the two-dimensional target image area according to the parameters of the photographing device, the photographing angle of view, the two-dimensional coordinates, the physical distance, and the semantic information of the target object.
The system according to claim 40, wherein when the image processing device determines the target three-dimensional coordinates of the target object according to the initial three-dimensional coordinates and the three-dimensional coordinate adjustment information, it is specifically used for:

Processing the two-dimensional target image area according to the second neural network, and determining the three-dimensional coordinate adjustment information of the two-dimensional target image area;

Determine the target three-dimensional coordinates of the two-dimensional target image area according to the three-dimensional coordinate adjustment information and the initial three-dimensional coordinates.
The system according to claim 37, wherein the parameters of the shooting device include internal parameters and external parameters, the internal parameters include the focal length of the shooting device, and the external parameters include the optical center of the shooting device.
The system of claim 46, wherein the first projection model comprises a pinhole imaging model.
The system of claim 48, wherein the second projection model comprises any one of a Snell window projection model, an equal area projection model, an equal distance projection model, and a stereo projection model.
A computer-readable storage medium in which a computer program is stored, characterized in that: when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 18 are implemented .