WO2023116641A1

WO2023116641A1 - Parking space detection model training method and apparatus, and parking space detection method and apparatus

Info

Publication number: WO2023116641A1
Application number: PCT/CN2022/140134
Authority: WO
Inventors: 王东伟; 吉方成
Original assignee: 北京罗克维尔斯科技有限公司
Priority date: 2021-12-21
Filing date: 2022-12-19
Publication date: 2023-06-29
Also published as: CN114445794A

Abstract

Provided are a parking space detection model training method and apparatus, and a parking space detection method and apparatus. The parking space detection model training method comprises: obtaining a sample image of a sample parking space, and obtaining a center position of the sample parking space from the sample image; determining a parking space recognition redundant area and a parking space corner point regression redundant area in the sample image according to the center position; marking the sample image according to the parking space recognition redundant area to generate a category true value heat map, and marking the sample image according to the parking space corner point regression redundant area to generate a regression true value heat map of parking space corner points; and training a parking space detection model on the basis of the sample image, the category true value heat map, and the regression true value heat map to generate a target parking space detection model.

Description

Parking space detection model training method, parking space detection method and device thereof

Cross References to Related Applications

This application is based on a Chinese patent application with application number 202111571230.X and a filing date of December 21, 2021, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated by reference into this application.

technical field

The present disclosure relates to the technical field of automatic driving, in particular to a parking space detection model training method, a parking space detection method and a device thereof.

Background technique

In related technologies, users have an urgent need for assisted parking and automatic parking. In addition to driving on the road, vehicles also need to drive into corresponding parking spaces. This requires accurate identification of parking spaces near the vehicle in order to plan well. Parking route, park safely and accurately. However, in the existing way of calculating corner points of parking spaces, each parking space only outputs a set of corner points of parking spaces, which has strong randomness, and the corner positions of different frames are easy to jump.

Contents of the invention

The embodiment of the first aspect of the present disclosure proposes a parking space detection model training method, by obtaining a sample image of a sample parking space, and obtaining the center position of the sample parking space from the sample image; according to the center position, determine the The parking space recognition redundant area and the parking space corner regression redundant area in the sample image; according to the parking space identification redundant area, the sample image is marked to generate the category true value heat map of the sample image, and according to The parking space corner returns to a redundant area, and the sample image is marked to generate a regression true value thermodynamic map of the parking space corner of the sample image; based on the sample image, the category true value thermodynamic map and the Regression to the true value heat map, training the parking space detection model, and generating the target parking space detection model.

In the training process of the present disclosure, the redundancy in the recognition of the parking space category and the regression calculation of the corner point of the parking space is considered, so that the trained target parking space detection model will consider the redundant area when recognizing the parking space, and can detect the center of the parking space more accurately. location, bay category, and corner location.

The embodiment of the second aspect of the present disclosure proposes a parking space detection method, including: acquiring a target image to be predicted, wherein the target image includes at least one parking space to be tested; inputting the target image into a target parking space detection model, To obtain the target center position of the parking space to be tested, the category target heat map and the regression target heat map, wherein the category target heat map includes a redundant area for identifying the category of the parking space, and the regression target heat map includes The redundant area used for the regression of the corner point of the parking space, the target parking space detection model is generated by training the parking space detection model training method described in the embodiment of the first aspect above; The redundant area used for the regression of the parking space corners is described, and the target parking space category and the target parking space corner points of the parking spaces to be tested are determined.

The embodiment of the third aspect of the present disclosure proposes a parking space detection model training device, including: an acquisition module, configured to acquire a sample image of a sample parking space, and obtain the center position of the sample parking space from the sample image; a determination module, It is used to determine the parking space recognition redundant area and the parking space corner regression redundant area in the sample image according to the central position; the generating module is used to mark the sample image according to the parking space identification redundant area , generating a class true value heat map of the sample image, and marking the sample image according to the regression redundant area of the parking corner point, generating a regression true value heat map of the parking space corner point of the sample image; training A module, configured to train a parking space detection model based on the sample image, the category true value thermodynamic map and the regression true value thermodynamic map, to generate a target parking space detection model.

The embodiment of the fourth aspect of the present disclosure proposes a parking space detection device, including: a first acquisition module, configured to acquire a target image to be predicted, wherein the target image includes at least one parking space to be tested; a second acquisition module, It is used to input the target image into the target parking space detection model to obtain the target center position of the parking space to be tested, the category target heat map and the regression target heat map, wherein the category target heat map includes the parking space category The identified redundant area, the regression target heat map includes a redundant area for the regression of parking space corners, and the target parking space detection model is generated by training the parking space detection model training device described in the embodiment of the third aspect above; The determination module is configured to determine the target parking space category and the target parking space corner of the parking space to be tested based on the redundant area used for identifying the parking space category and the redundant area used for the regression of the parking space corner point.

The embodiment of the fifth aspect of the present disclosure provides an automatic driving vehicle, which includes the above-mentioned parking space detection model training device or parking space detection device.

To achieve the above purpose, the embodiment of the sixth aspect of the present disclosure provides an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores information that can be used by the Instructions executed by at least one processor, the instructions are executed by the at least one processor to implement the parking space detection model training method described in the embodiment of the first aspect of the present disclosure and the parking space detection method described in the embodiment of the second aspect .

To achieve the above purpose, the embodiment of the seventh aspect of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to realize the parking space described in the embodiment of the first aspect of the present disclosure The detection model training method and the parking space detection method described in the embodiment of the second aspect.

To achieve the above purpose, the embodiment of the eighth aspect of the present disclosure proposes a computer program product, including a computer program, when the computer program is executed by a processor, it implements the parking space detection model training method described in the embodiment of the first aspect of the present disclosure And the parking space detection method described in the embodiment of the second aspect.

To achieve the above purpose, the embodiment of the ninth aspect of the present disclosure provides a computer program, the computer program includes computer program code, and when the computer program code is run on the computer, the computer executes the computer program according to the first aspect of the present disclosure. The parking space detection model training method described in the example and the parking space detection method described in the embodiment of the second aspect.

Description of drawings

Fig. 1 is a schematic diagram of a method for training a parking space detection model according to an embodiment of the present disclosure.

Fig. 2 is a schematic diagram of obtaining the central position of a sample parking space from a sample image according to an embodiment of the present disclosure.

Fig. 3 is a schematic diagram of a process of generating a category truth heat map of a sample image according to an embodiment of the present disclosure.

Fig. 4 is a schematic diagram of a process of generating a regression truth heat map according to an embodiment of the present disclosure.

Fig. 5 is a schematic diagram of training a parking space detection model to generate a target parking space detection model according to an embodiment of the present disclosure.

Fig. 6 is a schematic diagram of an algorithm of a parking space detection model according to an embodiment of the present disclosure.

Fig. 7 is a schematic diagram of a parking space detection model training method according to an embodiment of the present disclosure.

Fig. 8 is a schematic diagram of a parking space detection method according to an embodiment of the present disclosure.

Fig. 9(a) is a schematic diagram of a thermal map of a target center position according to an embodiment of the present disclosure.

Fig. 9(b) is a schematic diagram of a category target heat map shown in an embodiment of the present disclosure.

Fig. 9(c) is a schematic diagram of a regression target heat map shown in an embodiment of the present disclosure.

Fig. 10 is a schematic diagram of determining a target parking space category and a target parking space corner point of a parking space to be tested according to an embodiment of the present disclosure.

Fig. 11 is a schematic diagram of a parking space detection method according to an embodiment of the present disclosure.

Fig. 12 is a schematic diagram of a parking space detection model training device according to an embodiment of the present disclosure.

Fig. 13 is a schematic diagram of a parking space detection device according to an embodiment of the present disclosure.

Fig. 14 is a schematic diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure are described in detail below, examples of which are illustrated in the drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present disclosure and should not be construed as limiting the present disclosure.

FIG. 1 is an implementation of a parking space detection model training method proposed in the present disclosure. As shown in FIG. 1 , the parking space detection model training method includes the following steps: S101-S104.

S101. Obtain a sample image of a sample parking space, and obtain a center position of the sample parking space from the sample image.

A sample image corresponding to the sample parking space collected by the image acquisition device is acquired. In some embodiments, the sample images may be images collected by a large number of vehicles during parking, or images of vehicles during parking may be collected by cameras preset near the parking spaces.

In some embodiments, 4 corner points of the sample parking space are identified from the sample image, and the center position of the sample parking space is obtained according to the coordinates of the 4 corner points.

In some embodiments, when the center position of the sample parking space is obtained according to the coordinates of the four corner points, it can be directly marked manually, or it can be input into an algorithm to identify and directly output the center position of the sample parking space.

S102. Determine a parking space recognition redundant area and a parking space corner regression redundant area in the sample image according to the center position.

In order to accurately classify the parking space even if the parking space detection model has a slight error in the category detection of the parking space, the center of the sample parking space can be taken as the center, and the area within a certain range of the center can be selected as the parking space recognition redundancy area.

In order to accurately determine the corner point of the parking space even when the parking space detection model has a slight error in the corner detection of the parking space, the center position of the sample parking space can be taken as the center, and the area within a certain range of the center can be selected as the parking space angle Point regression redundant area.

S103, mark the sample image according to the redundant area of the parking space recognition, generate the category truth heat map of the sample image, and return the redundant area according to the corner point of the parking space, mark the sample image, and generate the regression of the parking space corner point of the sample image True heat map.

According to the parking space recognition redundant area, based on the category of the sample parking space, the parking space recognition redundant area and the non-parking space recognition redundant area on the sample image are respectively marked, and the class true value heat map is generated according to the marked image. In some embodiments, the pixel points in the parking space recognition redundant area of the sample parking space in the sampling sample image are assigned as the class value of the sample parking space, and other positions are assigned as 0. In some embodiments, the category of the sample parking spaces may include horizontal parking spaces, longitudinal parking spaces, oblique parking spaces, etc., different category values may be assigned to different parking spaces, for example, the category value of the horizontal parking spaces may be assigned a value of 1, and Assign a value of 2 to the category value of the vertical parking space.

Since each sample parking space corresponds to four corner points, and each corner point has two dimensions of x and y, so the sample parking space corner regression has a total of 8 dimensions, according to the parking space corner regression redundant area, based on the sample parking space corner point The multiple dimensions of the sample image are marked, and the regression true value heat map of the corner points of the parking space of the sample image is generated.

S104, based on the sample image, the class true value heat map and the regression true value heat map, train the parking space detection model to generate a target parking space detection model.

Obtain the central location heat map, category heat map, and regression true value heat map of the sample parking spaces output by the parking space detection model, and obtain the heat output of the parking space detection model based on the sample image, category true value heat map, and regression true value heat map obtained above The difference between the map and its corresponding true value heat map, and determine the respective difference to generate a loss function, and train the parking space detection model according to the loss function until the training end condition is met, and the final target parking space detection model is obtained.

In some embodiments, the training end condition may be: when the parking space detection model is trained for a preset number of times, the training ends. For example, if the preset number of training times is 1000 times, the training will end after the parking space detection model has been trained 1000 times.

In some embodiments, the training end condition may be: when the error of the trained parking space detection model converges to a preset error value, the training ends.

In some embodiments, the training end condition may be: when the loss function of the trained parking space detection model decreases to a preset loss value, the training ends.

In the training process of the present disclosure, the redundancy in the recognition of the parking space category and the regression calculation of the corner point of the parking space is considered, so that the trained target parking space detection model will consider the redundant area when recognizing the parking space, so that the target parking space detection model can be used for the parking space. Accurate recognition, so that the center position, parking space category and corner position of the parking space can be detected more accurately.

Figure 2 is an implementation of a parking space detection model training method proposed in the present disclosure. Based on the above-mentioned embodiments, as shown in Figure 2, obtaining the center position of the sample parking space from the sample image includes the following steps: S201 -S203.

S201. Identify corner points of the sample parking space from the sample image, and mark them.

Obtain the multi-channel images collected during the parking process, and stitch the multi-channel images to generate a stitched sample image, record the width of the sample image as w, and record the height as h, that is, the size of the original image can be Denote as (w, h). In some embodiments, the sample vehicle may collect multiple images during parking, or a preset multiple camera may collect multiple images during parking.

In some embodiments, the multi-channel fisheye image data during the entire parking process can be collected by the sample vehicle, and the image stitching algorithm is used to stitch the multi-channel fisheye image data into a surround-view mosaic, and the surround-view mosaic is used as a sample image .

Get the 4 corner points of the sample parking space from the sample image, get the center position of the sample parking space according to the 4 corner points, and record the center position of the sample parking space as (x, y). In some embodiments, the corner points of the sample parking spaces can be marked manually, or can be marked by using a target detection algorithm to detect the parking spaces.

S202. Down-sampling the sample image to obtain a sampled sample image.

The sample image obtained above is down-sampled to obtain a down-sampled sample image. In some embodiments, the sample image obtained above may be down-sampled by 4 times to obtain a down-sampled sample image, and the size of the sample image is (w/4, h/4).

S203. Based on the coordinates of the corner points, determine the center position of the sample parking space in the sample image.

Calculate according to the coordinates of the four corner points of the sample parking spaces obtained above in the sampling sample image, determine the center position of the sample parking space in the sampling sample image, and round the center position of the sample parking space in the sampling sample image, that is, the sample The center position of the parking space is recorded as (round(x/4), round(y/4)). The center position of the sample parking space is assigned a value of 1, and other positions are assigned a value of 0.

In the embodiment of the present disclosure, the sample image is down-sampled, the receptive field is increased, and the calculation amount for obtaining the true value of the center position in the sample parking space is reduced.

FIG. 3 is an implementation of a parking space detection model training method proposed in the present disclosure. Based on the above-mentioned embodiments, as shown in FIG. 3 , the generation process of the category true value heat map of the sample image includes the following steps: S301-S303.

S301. Determine a first redundant offset for parking space identification.

Similar to the above, the obtained sample image is down-sampled to obtain a down-sampled sample image. In some embodiments, the sample image obtained above may be down-sampled by 4 times to obtain a down-sampled sample image, and the size of the sample image is (w/4, h/4).

Calculate according to the coordinates of the four corner points of the sample parking space in the sampling sample image, determine the center position of the sample parking space in the sampling sample image, and round the center position of the sample parking space in the sampling sample image, that is, the center position of the sample parking space in the sampling sample image The center position of the sample parking space is recorded as (round(x/4), round(y/4)).

In order to accurately classify the parking space even if the parking space detection model has a slight error in detecting the category of the parking space, a first redundant offset for parking space identification is preset. Wherein, the first redundancy offset includes ΔX1 and ΔY1. Wherein, if the size of the sample parking space is [Wspot, Hspot] in the real situation, then ΔX1 and ΔY1 are within the range of [0, min(Wspot, Hspot)/2].

S302. Taking the central position as the center and using the first redundant offset as the pixel offset radius, expand the sample image to generate a parking space recognition redundant area in the sample image.

Take the central position as the center, take the first redundant offset as the pixel offset radius, expand on the sample image, generate the parking space recognition redundant area in the sample image, and record the parking space identification redundant area as (round(x /4)±ΔX1, round(y/4)±ΔY1).

S303. Based on the category of the sample parking space, respectively mark the values of the pixels in the parking space identification redundant area and the first remaining area except the parking space identification redundant area to generate a category true value heat map.

Based on the category of the sample parking space, the values of the pixel points in the redundant parking space recognition area and the first remaining area except the redundant parking space identification area are respectively marked. In some embodiments, the pixel points in the parking space identification redundant area of the sample parking space are assigned as the class value of the sample parking space, and other positions are assigned as 0. In some embodiments, the category of the sample parking spaces may include horizontal parking spaces, longitudinal parking spaces, oblique parking spaces, etc., different category values may be assigned to different parking spaces, for example, the category value of the horizontal parking spaces may be assigned a value of 1, and Assign a value of 2 to the category value of the vertical parking space. According to the labeled image, generate a heat map of the true value of the category.

The embodiment of the present disclosure sets a first redundant offset for parking space identification, sets a redundant area for parking space identification based on the first redundant offset, and generates a class true value heat map to carry out the parking space detection model Training prevents the generated target parking space detection model from being inaccurate in recognizing the parking space category due to corner jumps, and increases the accuracy and robustness of the subsequent prediction of the parking space category.

FIG. 4 is an implementation of a parking space detection model training method proposed in the present disclosure. Based on the above-mentioned embodiments, as shown in FIG. 4 , the generation process of the regression true value heat map includes the following steps: S401-S403.

S401. Determine a second redundant offset for the parking corner regression.

In order to accurately determine the corner point of the parking space even if the corner point detection of the parking space by the parking space detection model has a slight error, a second redundant offset for the regression of the corner point of the parking space is preset. Wherein, the second redundancy offset includes ΔX2 and ΔY2. Wherein, if the size of the sample parking space is [Wspot, Hspot] in the real situation, then ΔX2 and ΔY2 are within the range of [0, min(Wspot, Hspot)/2].

S402. Taking the central position as the center and using the second redundant offset as the pixel offset radius, expand the sample image to generate a parking space corner regression redundant area in the sample image.

Take the central position as the center, and take the second redundant offset as the pixel offset radius, expand on the sample image to generate the parking space corner regression redundant area in the sample image, and record the parking space corner regression redundant area as (round(x/4)±ΔX2, round(y/4)±ΔY2).

S403. Based on the multiple dimensions of the sample parking corner points, mark the regression redundant area of the parking space corner points as the parking space corner point regression labeling map, and process the parking space corner point regression labeling map to obtain the regression true value of the parking space corner points heat map.

Since each sample parking space corresponds to four corner points, and each corner point has two dimensions of x and y, so the sample parking space corner regression has a total of 8 dimensions. Based on the multiple dimensions of the sample parking space corner points, the parking space corner point The regression redundant area is marked as the regression label map of the parking space corner point, and the regression label map of the parking space corner point is processed to obtain the regression true value heat map of the parking space corner point.

The embodiment of the present disclosure sets a second redundant offset for the regression of the corner of the parking space, obtains the redundant area of the corner of the parking space based on the second redundant offset, and generates the regression truth of the corner of the parking space The value heat map trains the parking space detection model, which prevents the generated target parking space detection model from inaccurate recognition of parking space corners due to corner jumps, and increases the accuracy and robustness of the subsequent prediction of parking space corners. Further, after obtaining the true value heat map of the parking corner point regression, based on the true value heat map of the parking space corner point regression, the corner point regression values from the corner point to the parking space corner point regression redundant area in multiple dimensions are obtained. in. The corner point regression value can be understood as the distance value from the corner point coordinates in the two dimensions corresponding to each corner point to each pixel point in the corner point regression redundancy area of the parking space.

FIG. 5 is an implementation of a parking space detection model training method proposed in the present disclosure. Based on the above-mentioned embodiments, as shown in FIG. 5 , based on the sample image, the category true value heat map and the regression true value heat map, The training of the parking space detection model to generate the target parking space detection model includes the following steps: S501-S504.

S501. Input the sample image into the parking space detection model to perform multi-scale feature extraction.

Figure 6 is a schematic diagram of the algorithm of the parking space detection model. As shown in Figure 6, in order to minimize the calculation amount of the model under the premise of ensuring the detection accuracy, the sample image is input into the parking space detection model for multi-scale feature extraction to obtain multi-scale feature maps. In some embodiments, the feature extraction algorithm may use a convolutional neural network (Convolutional Neural Networks, CNN).

S502. Perform center position prediction, category prediction, and regression prediction based on the extracted multi-scale feature map, so as to obtain the prediction center location, category prediction heat map, and regression prediction heat map.

In order to better propose the characteristics of the sample parking spaces, the feature maps of the target scale in the multi-scale feature map are fused to obtain the fused feature map. The fused feature map is processed by the center position detection head to complete the center position prediction and obtain the predicted center position. The fused feature map is processed by the category detection head to complete the category prediction, and the category prediction heat map is obtained. The fused feature map is processed by the corner regression detection head to complete the regression prediction, and the regression prediction heat map is obtained. Wherein, the center position detection head, the category detection head and the corner point regression detection head may be a network composed of convolutional layers.

In some embodiments, as shown in FIG. 6 , CNN is used for feature extraction on the sample image, and the extracted feature map is subjected to 2 times downsampling, and CNN feature extraction is performed on the 2 times downsampled feature map, and so on. Then perform 4 times downsampling, 8 times downsampling and 16 times downsampling on the feature map in turn, and perform CNN feature extraction on the feature map after each sampling, and obtain 4 times downsampling feature map and 8 times downsampling feature map respectively and 16x downsampled feature maps.

As shown in Figure 6, the 16-fold downsampled feature map is processed, and the processed image is 16-fold up-sampled, and the 16-fold up-sampled image is fused with the 16-fold downsampled feature map before CNN Feature extraction to obtain the first fused image. The first fused image is upsampled by 8 times, fused with the 8 times downsampled feature map, and then CNN feature extraction is performed to obtain the final fused feature map.

S503. Determine the loss function of the parking space detection model based on the predicted center position and the center position, the class prediction heat map and the class true value heat map, and the regression prediction heat map and the regression true value heat map.

The loss function of the parking space detection model is determined based on the predicted center position and the center position, the class prediction heat map and the class true value heat map, and the regression prediction heat map and the regression true value heat map. Among them, the loss function of the parking space detection model is expressed as:

Loss＝Loss_pos+Loss_kps-reg+Loss_cls

In the above formula, Loss_pos is the position error between the predicted center position and the center position, Loss_kps-reg is the regression error between the regression prediction heat map and the regression true value heat map, and Loss_cls is the difference between the class prediction heat map and the class true value heat map classification error.

Among them, the calculation of the position error between the predicted center position and the center position can use the cross entropy loss function (Cross entropy loss); the calculation of the regression error of the regression prediction heat map and the regression true value heat map can use the regression loss function (Smooth L1 Loss) ; To calculate the classification error between the category prediction heatmap and the category true value heatmap, the cross entropy loss function (Cross entropy loss) can be used.

S504. Adjust the parking space detection model based on the loss function and continue training until the training ends to generate a target parking space detection model.

Based on the loss function generated after each sample image is input into the parking space detection model, the parking space detection model is adjusted and the training continues until the loss function reaches the preset standard, then the training ends and the target parking space detection model is generated.

The target parking space detection model obtained by setting redundant area training in the embodiment of the present disclosure reduces the randomness when outputting corner points, and obtains rich semantic information of the sample image through multi-scale feature extraction of the sample image, which improves the accuracy of output. The accuracy and stability of the center position of the parking space to be tested, the type of parking space, and the corner position.

Fig. 7 is an implementation manner of a parking space detection model training method proposed in the present disclosure. As shown in Fig. 7, the parking space detection model training method includes the following steps: S701-S715.

S701. Acquire multiple images collected during parking, and splice the multiple images to generate a spliced sample image.

S702. Identify corner points of the sample parking space from the sample image, and mark them.

S703. Down-sampling the sample image to obtain a sampled sample image.

S704. Based on the coordinates of the corner points, determine the center position of the sample parking space in the sample image.

Regarding the implementation manner of steps S701 to S704, the implementation manners in the embodiments of the present disclosure may be adopted, and details are not repeated here.

S705. Determine a first redundant offset for parking space identification.

S706. Taking the central position as the center and using the first redundant offset as the pixel offset radius, expand the sample image to generate a parking space recognition redundant area in the sample image.

S707. Based on the category of the sample parking space, respectively mark the values of the pixels in the parking space identification redundant area and the first remaining area except the parking space identification redundant area to generate a class true value heat map.

Regarding the implementation manner of steps S705-S707, the implementation manners in the embodiments of the present disclosure may be adopted, and details are not repeated here.

S708. Determine the second redundant offset for the regression of the corner point of the parking space.

S709, expanding on the sample image with the center position as the center and the second redundant offset as the pixel offset radius, to generate a parking space corner regression redundant area in the sample image.

S710: Based on the multiple dimensions of the sample parking corner points, mark the regression redundant area of the parking space corner points as a parking space corner point regression labeling map, and process the parking space corner point regression labeling map to obtain the regression true value of the parking space corner points heat map.

S711. Based on the heat map of the true regression value of the parking space corner, obtain the corner point regression value from the corner point to the parking space corner point regression redundant area in multiple dimensions.

Regarding the implementation manner of steps S708-S711, the implementation manners in the embodiments of the present disclosure may be adopted, and details are not repeated here.

S712. Input the sample image into the parking space detection model to perform multi-scale feature extraction.

S713. Based on the extracted multi-scale feature map, respectively perform center position prediction, category prediction and regression prediction, so as to obtain the prediction center position, category prediction heat map, and regression prediction heat map.

S714. Determine a loss function of the parking space detection model based on the predicted center position and the center position, the class prediction heat map and the class true value heat map, and the regression prediction heat map and the regression true value heat map.

S715. Adjust the parking space detection model based on the loss function and continue training until the training ends to generate a target parking space detection model.

Regarding the implementation manner of steps S712-S715, the implementation manners in the embodiments of the present disclosure may be adopted, and details are not repeated here.

Fig. 8 is an embodiment of a parking space detection method proposed in the present disclosure. The target parking space detection model of the parking space detection method can be obtained by training the parking space detection model training method of any embodiment of the above-mentioned Fig. 1-Fig. 7, as shown in Fig. 8 As shown, the parking space detection method includes the following steps: S801-S803.

S801. Acquire a target image to be predicted, wherein the target image includes at least one parking space to be tested.

An image to be predicted is acquired, and the image to be predicted is used as a target image, wherein the target image includes at least one parking space to be tested. In some embodiments, the target image may be an image collected by the vehicle under test during parking, or may be an image collected by a preset camera during parking.

S802, input the target image into the target parking space detection model to obtain the target center position of the parking space to be tested, the category target heat map and the regression target heat map, wherein the category target heat map includes a redundant area for identifying the category of the parking space, The regression target heatmap includes redundant regions for the parking corner regression.

Input the target image into the target parking space detection model, and the target parking space detection model first performs feature extraction and fusion on the target image to obtain a fused feature map. Then based on the fused feature map, the target center position heat map, category target heat map and regression target heat output of the target parking space detection model are respectively obtained through the center position detection head, category detection head and corner point regression detection head. picture. Among them, Fig. 9(a) is a schematic diagram of the heat map of the target center position of the embodiment of the present disclosure; Fig. 9(b) is a schematic diagram of the heat map of the category target of the embodiment of the present disclosure; Fig. 9(c) is a schematic diagram of the heat map of the embodiment of the present disclosure Schematic representation of the regression target heatmap. As shown in Figure 9(b), the category target heat map includes a redundant area for the recognition of the parking space category; as shown in Figure 9(c), the regression target heat map includes a redundant area for the parking space corner regression .

In some embodiments, continuing to take Fig. 6 as an example, CNN is used for feature extraction on the target image, and the extracted feature map is subjected to 2 times downsampling, and CNN feature extraction is performed on the 2 times downsampled feature map, and so on. , and then sequentially perform 4 times downsampling, 8 times downsampling and 16 times downsampling on the feature map, and perform CNN feature extraction on the feature map after each sampling, and obtain 4 times downsampling feature map and 8 times downsampling feature respectively Figure and 16x downsampling feature map. As shown in Figure 6, the 16-fold downsampled feature map is processed, and the processed image is 16-fold up-sampled, and the 16-fold up-sampled image is fused with the 16-fold downsampled feature map before CNN Feature extraction to obtain the first fused image. The first fused image is upsampled by 8 times, fused with the 8 times downsampled feature map, and then CNN feature extraction is performed to obtain the final fused feature map.

S803. Based on the redundant area used for identifying the parking space category and the redundant area used for the parking space corner regression, determine the target parking space category and the target parking space corner point of the parking space to be tested.

Determine the target parking space category of the parking space to be tested based on the coordinates of the pixels in the redundant area for identifying the parking space category in the class target heat map or the corresponding parking space category for the pixels in the redundant area for identifying the parking space category.

Based on the coordinates of the pixels in the redundant area used for the regression of the parking space corner in the regression target heat map, the target parking space corner point of the parking space to be tested is determined.

During the training process of the present disclosure, the redundancy in the recognition of the parking space category and the regression calculation of the corner point of the parking space is considered, so that the trained target parking space detection model will consider the redundant area when recognizing the parking space, preventing the randomness of outputting a single result. It can more accurately detect the center position of the parking space, the category of the parking space and the corner position.

In some embodiments, a non-maximum suppression (Non-Maximum Suppression, NMS) algorithm is performed on the center position target heat map of the parking space to be tested output by the target parking space detection model, and the processed center position target heat map is greater than The pixel point of the preset threshold is used as the target center position of the parking space to be tested.

FIG. 10 is an implementation of a parking space detection method proposed in the present disclosure. Based on the above-mentioned embodiments, as shown in FIG. The redundant area is to determine the target parking space category and the target parking space corner of the parking space to be tested, including the following steps: S1001-S1002.

S1001. Determine the target parking space category of the parking space to be tested based on the redundant area used for identifying the category of the parking space.

In some embodiments, the coordinates of multiple pixel points in the redundant area used for identifying the parking space category are averaged to obtain a target pixel point. According to the parking space category corresponding to the parking space category value of the target pixel point, the target parking space category of the parking space to be tested is determined. For example, if the parking space category value of the target pixel is 1, and the parking space category corresponding to 1 is a horizontal parking space, then the target parking space category of the parking space to be tested is a horizontal parking space.

In some other embodiments, a plurality of parking space categories corresponding to a plurality of pixels in the redundant area for identification of the parking space category is determined, voting is performed on the multiple parking space categories, and a target parking space category of the parking space to be tested is determined. For example, if there are 10 pixels in the redundant area for recognition of the parking space category, among them, the parking space category value of 9 pixels is 1, and the corresponding parking space category of 1 is a horizontal parking space; the parking space category of 1 pixel point The value is 2, and the parking space category corresponding to 2 is a longitudinal parking space, then vote for these 10 parking space categories. Since 90% of the parking spaces indicate that the category of the parking spaces is a horizontal parking space, it is determined that the target parking space category of the parking spaces to be tested is a horizontal parking space.

S1002. Determine a target corner point of the parking space to be tested based on the redundant area used for the regression of the corner point of the parking space.

Determine the corresponding multiple corner regression values in the redundant area used for parking corner regression, wherein the multiple corner regression values are based on the regression based on the parking corner after obtaining the regression true value heat map of the parking corner The true value heat map obtains multiple corner regression values from the corner point to the parking space corner regression redundant area in multiple dimensions.

Multiple pixel points in the redundant area used for parking corner regression are calculated based on multiple corner regression values corresponding to the redundant area used for parking corner regression, and four corner points of the parking space to be tested are obtained. Corresponding coordinates of a plurality of predicted corner points respectively, and averaging the coordinates of the plurality of predicted corner points to determine a corner point of a target parking space corresponding to any corner point of the parking space to be measured. Taking the first corner point of the parking space to be tested as an example, if the multiple corner regression values corresponding to the redundant area used for the parking space corner regression are 21, 22, and 23 pixels respectively, then for the parking space corner Each pixel in the redundant area of point regression is offset by 21, 22, and 23 pixels in both X and Y dimensions, and multiple predicted corners corresponding to the first corner of the parking space to be tested are respectively obtained coordinates, and average the coordinates of multiple predicted corner points to determine the corner point of the target parking space corresponding to any corner point of the parking space to be tested.

The target parking space detection model of the present disclosure will consider the redundant area when identifying the parking space, based on the average pixel point or the way of voting on the parking space category corresponding to multiple pixel points, which reduces the randomness of a single result, reduces the error, and can More accurately detect the center position of the parking space, the category of the parking space and the corner position.

Fig. 11 is an embodiment of a parking space detection method proposed in the present disclosure. The target parking space detection model of the parking space detection method can be obtained by training the parking space detection model training method of any one of the above-mentioned Fig. 1-Fig. 7 embodiments, as shown in Fig. 11 As shown, the parking space detection method includes the following steps: S1101-S1107.

S1101. Acquire a target image to be predicted, wherein the target image includes at least one parking space to be tested.

S1102, input the target image into the target parking space detection model to obtain the target center position of the parking space to be tested, the category target heat map and the regression target heat map, wherein the category target heat map includes a redundant area for identifying the category of the parking space, The regression target heatmap includes redundant regions for the parking corner regression.

S1103, averaging multiple pixels in the redundant area used for identifying the category of the parking space to obtain a target pixel.

S1104. Based on the target pixel, determine the target parking space category of the parking space to be tested.

Regarding the implementation manner of steps S1103-S1104, the implementation manners in the embodiments of the present disclosure may be adopted, and details are not repeated here.

S1105. Determine a plurality of corresponding corner regression values in the redundant area used for parking corner regression.

S1106. Determine a plurality of predicted corner coordinates corresponding to any corner of the parking space to be tested based on the regression values of the plurality of corners.

S1107. Average the coordinates of multiple predicted corner points, and determine the corner point of the target parking space corresponding to any corner point of the parking space to be measured.

Regarding the implementation manner of steps S1105-S1107, the implementation manners in the embodiments of the present disclosure may be adopted, and details are not repeated here.

Figure 12 is a parking space detection model training device proposed in the present disclosure. As shown in Figure 12, the parking space detection model training device 1200 includes an acquisition module 1201, a determination module 1202, a generation module 1203 and a training module 1204, wherein:

The obtaining module 1201 is configured to obtain a sample image of a sample parking space, and obtain a center position of the sample parking space from the sample image.

The determination module 1202 is configured to determine the redundant area of parking space recognition and the redundant area of parking space corner regression in the sample image according to the center position.

The generation module 1203 is used to identify the redundant area according to the parking space, mark the sample image, generate the category truth heat map of the sample image, and return the redundant area according to the corner point of the parking space, mark the sample image, and generate the parking space of the sample image Regression ground-truth heatmap of corner points.

The training module 1204 is configured to train the parking space detection model based on the sample image, the category true value heat map and the regression true value heat map to generate a target parking space detection model.

Further, the acquisition module 1201 is also used to: identify the corner points of the sample parking space from the sample image and mark them; down-sample the sample image to obtain a sample sample image; determine the sample in the sample sample image based on the coordinates of the corner points The central location of the parking lot.

Further, the determination module 1202 is also used to: determine the first redundant offset for parking space recognition; expand the sample image with the center position as the center and the first redundant offset as the pixel offset radius, Generate the parking space recognition redundant area in the sample image; based on the category of the sample parking space, mark the values of the pixels in the parking space recognition redundant area and the first remaining area except the parking space recognition redundant area to generate the category True value heat map.

Further, the determination module 1202 is also used to: determine the second redundant offset of the parking space corner regression; take the center position as the center, and use the second redundant offset as the pixel offset radius to perform the calculation on the sample image Expand and generate the parking space corner regression redundant area in the sample image; based on the multiple dimensions of the sample parking space corner point, mark the parking space corner regression redundant area as a parking space corner regression labeling map; mark the parking space corner regression The map is processed to obtain the regression true value heat map of the corner point of the parking space.

Further, the generation module 1203 is also used for: based on the heat map of the true regression value of the corner of the parking space, obtain the regression value of the corner point in multiple dimensions from the corner point to the regression redundant area of the corner point of the parking space.

Further, the training module 1204 is also used to: input the sample image into the parking space detection model for multi-scale feature extraction; perform center position prediction, category prediction and regression prediction based on the extracted multi-scale feature map, so as to obtain the prediction center position, category Prediction heat map, regression prediction heat map; based on the prediction center position and center position, category prediction heat map and category true value heat map, and regression prediction heat map and regression true value heat map, determine the loss function of the parking space detection model; based on loss The function adjusts the parking space detection model and continues training until the training ends to generate the target parking space detection model.

Further, the training module 1204 is also used to: fuse the feature maps of the target scale in the multi-scale feature map to obtain the fused feature map; predict the center position based on the fused feature map to obtain the predicted center position; The category prediction is performed on the map, and the category prediction heat map is obtained; the regression prediction is performed based on the fused feature map, and the regression prediction heat map is obtained.

FIG. 13 is a parking space detection device proposed in the present disclosure, which operates on the basis of the above-mentioned parking space detection model training device 1200. As shown in FIG. 13 , the parking space detection device 1300 includes a first acquisition module 1301 and a second acquisition module 1302 and determination module 1303, wherein:

The first acquiring module 1301 is configured to acquire a target image to be predicted, wherein the target image includes at least one parking space to be tested.

The second acquisition module 1302 is used to input the target image into the target parking space detection model to obtain the target center position of the parking space to be tested, the category target heat map and the regression target heat map, wherein the category target heat map includes the parking space category The identified redundant area, the regression target heat map includes the redundant area used for the parking corner regression.

The determining module 1303 is configured to determine a target parking space category and a target parking space corner point of the parking space to be tested based on the redundant area used for identifying the parking space category and the redundant area used for the parking space corner regression.

Further, the second acquisition module 1302 is also used to: process the central position target thermal map output by the target parking space detection model; use the processed pixel points in the central position target thermal map greater than the preset threshold as the parking spaces to be tested Target center position.

Further, the determining module 1303 is also used to: average a plurality of pixels in the redundant area used for identifying the parking space category to obtain the target pixel, and based on the target pixel, determine the target parking space category of the parking space to be tested or determine the parking space category Multiple parking space categories corresponding to multiple pixels in the identified redundant area; vote for multiple parking space categories to determine the target parking space category of the parking space to be tested.

Further, the determination module 1303 is also used to: determine a plurality of corner regression values corresponding to the redundant area used for the corner regression of the parking space; coordinates of a predicted corner point; the coordinates of multiple predicted corner points are averaged to determine the corner point of the target parking space corresponding to any corner point of the parking space to be measured.

Further, the present disclosure also proposes an automatic driving vehicle, which includes the above-mentioned parking space detection model training device 1200 or the parking space detection device 1300 .

In order to realize the above-mentioned embodiments, an embodiment of the present disclosure also proposes an electronic device 1400. As shown in FIG. The instructions executed by the processor are executed by at least one processor 1401 to implement the parking space detection model training method and the parking space detection method as shown in the above embodiments.

In order to realize the above-mentioned embodiments, the embodiments of the present disclosure also propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to implement the parking space detection model training method and the parking space as shown in the above-mentioned embodiments. Detection method.

In order to implement the above embodiments, the embodiments of the present disclosure further provide a computer program product, including a computer program, which implements the parking space detection model training method and the parking space detection method as shown in the above embodiments when the computer program is executed by a processor.

In order to realize the above-mentioned embodiment, the embodiment of the present disclosure also proposes a computer program, wherein the computer program includes computer program code, when the computer program code is run on the computer, it makes the computer execute the parking space detection model shown in the above-mentioned embodiment Training method and parking space detection method.

It should be noted that the foregoing explanations of the embodiments of the data transmission method are also applicable to the devices, vehicles, electronic equipment, non-transitory computer-readable storage media, computer program products, and computer programs in the above embodiments, and will not be repeated here. .

In describing the present disclosure, it is to be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", " Back", "Left", "Right", "Vertical", "Horizontal", "Top", "Bottom", "Inner", "Outer", "Clockwise", "Counterclockwise", "Axial", The orientations or positional relationships indicated by "radial", "circumferential", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying the referred devices or elements Must be in a particular orientation, constructed, and operate in a particular orientation, and thus should not be construed as limiting on the present disclosure.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms are not necessarily referring to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present disclosure, and those skilled in the art can understand the above-mentioned embodiments within the scope of the present disclosure. The embodiments are subject to changes, modifications, substitutions and variations.

Claims

A parking space detection model training method, comprising:

Obtain a sample image of the sample parking space, and obtain the center position of the sample parking space from the sample image;

According to the center position, determine the parking space recognition redundant area and the parking space corner regression redundant area in the sample image;

According to the parking space identification redundant area, the sample image is marked to generate a category truth heat map of the sample image, and according to the parking space corner regression redundant area, the sample image is marked to generate The regression true value thermodynamic map of the parking space corner of the sample image;

Based on the sample image, the category true value heat map and the regression true value heat map, the parking space detection model is trained to generate a target parking space detection model.
The method according to claim 1, wherein said obtaining the center position of the sample parking space from the sample image comprises:

identifying the corner points of the sample parking spaces from the sample images, and marking them;

Downsampling the sample image to obtain a sample image;

Based on the coordinates of the corner points, determine the center position of the sample parking space in the sample image.
The method according to claim 1 or 2, wherein the step of marking the sample image according to the redundant area of the parking space identification, and generating the class truth heat map of the sample image includes:

Determining the first redundant offset for parking space identification;

Taking the center position as the center and taking the first redundant offset as the pixel offset radius, expanding on the sample image to generate a parking space recognition redundant area in the sample image;

Based on the category of the sample parking spaces, the values of the pixels in the redundant area for parking space identification and the first remaining area except for the redundant area for parking space identification are respectively marked to generate the true value of the category. picture.
The method according to any one of claims 1 to 3, wherein, according to the regression redundant area of the corner point of the parking space, the sample image is marked, and the regression true of the corner point of the parking space of the sample image is generated. Value heat map, including:

Determining the second redundant offset for the regression of the corner point of the parking space;

Taking the center position as the center and taking the second redundant offset as the pixel offset radius, expanding on the sample image to generate a parking space corner regression redundant area in the sample image;

Based on the plurality of dimensions of the corner points of the sample parking spaces, the regression redundant area of the corner points of the parking spaces is marked as a regression labeling map of the corner points of the parking spaces;

The regression annotation map of the parking space corner is processed to obtain the regression true value heat map of the parking space corner.
The method according to any one of claims 1 to 4, wherein said method further comprises:

Based on the regression true value thermodynamic map of the corner of the parking space, obtain the corner regression value from the corner to the redundant area of the corner of the parking space in multiple dimensions.
The method according to any one of claims 1 to 5, wherein, based on the sample image, the category true value heat map and the regression true value heat map, the parking space detection model is trained to generate a target Parking space detection model, including:

Inputting the sample image into the parking space detection model for multi-scale feature extraction;

Based on the extracted multi-scale feature map, the center position prediction, category prediction and regression prediction are respectively performed to obtain the prediction center position, category prediction heat map, and regression prediction heat map;

Based on the predicted center position and the center position, the category prediction heat map and the category true value heat map, and the regression prediction heat map and the regression true value heat map, determine the parking space detection model loss function;

Adjust the parking space detection model based on the loss function and continue training until the training ends to generate the target parking space detection model.
The method according to claim 6, wherein the center position prediction, category prediction and regression prediction are respectively performed based on the extracted multi-scale feature map, so as to obtain a prediction center position, a category prediction heat map, and a regression prediction heat map, including:

Fusing the feature maps of the target scale in the multi-scale feature map to obtain a fused feature map;

Predicting the central position based on the fused feature map to obtain the predicted central position;

performing category prediction based on the fused feature map to obtain the category prediction heat map;

Regression prediction is performed based on the fused feature map to obtain the regression prediction heat map.
A parking space detection method, comprising:

Acquiring a target image to be predicted, wherein the target image includes at least one parking space to be tested;

The target image is input into the target parking space detection model to obtain the target center position of the parking space to be tested, the category target heat map and the regression target heat map, wherein the category target heat map includes A redundant area, the regression target heat map includes a redundant area for the regression of parking space corners, and the target parking space detection model is generated by training the parking space detection model training method described in any one of claims 1-7;

The target parking space category and the target parking space corner point of the parking space to be tested are determined based on the redundant area used for identifying the category of the parking space and the redundant area used for the regression of the corner point of the parking space.
The method according to claim 8, wherein said obtaining the target center position of the parking space to be tested comprises:

Processing the center position target heat map output by the target parking space detection model;

A pixel point in the processed center position target heat map greater than a preset threshold is used as the target center position of the parking space to be tested.
The method according to claim 8 or 9, wherein said determining the target parking space category of said parking space to be tested comprises:

Averaging a plurality of pixels in the redundant area for identifying the parking space category to obtain a target pixel, and based on the target pixel, determining the target parking space category of the parking space to be tested; or

Determining a plurality of parking space categories corresponding to a plurality of pixels in the redundant area for identifying the parking space category, and voting on the plurality of parking space categories to determine the target parking space category of the parking space to be tested.
The method according to any one of claims 8 to 10, wherein determining the target corner point of the parking space to be tested comprises:

Determining a plurality of corresponding corner regression values in the redundancy zone used for parking corner regression;

determining a plurality of predicted corner coordinates corresponding to any one of the corner points of the parking space to be tested based on the plurality of corner point regression values;

The coordinates of the plurality of predicted corner points are averaged to determine the corner point of the target parking space corresponding to any one of the corner points of the parking space to be tested.
A parking space detection model training device, comprising:

An acquisition module, configured to acquire a sample image of a sample parking space, and obtain the center position of the sample parking space from the sample image;

A determining module, configured to determine a parking space recognition redundant area and a parking space corner regression redundant area in the sample image according to the center position;

The generation module is used to mark the sample image according to the parking space identification redundant area, generate the category true value heat map of the sample image, and return the redundant area according to the parking space corner point to the sample image The image is marked, and the regression true value heat map of the parking space corner of the sample image is generated;

The training module is configured to train the parking space detection model based on the sample image, the category true value thermodynamic map and the regression true value thermodynamic map to generate a target parking space detection model.
A parking space detection device, comprising:

A first acquisition module, configured to acquire a target image to be predicted, wherein the target image includes at least one parking space to be tested;

The second acquisition module is used to input the target image into the target parking space detection model, so as to obtain the target center position, category target heat map and regression target heat map of the parking space to be tested, wherein the category target heat map Including a redundant area for class recognition of parking spaces, the regression target heat map includes a redundant area for corner regression of parking spaces, and the target parking space detection model is generated by training the parking space detection model training device described in claim 12 ;

The determination module is configured to determine the target parking space category and the target parking space corner of the parking space to be tested based on the redundant area used for identifying the parking space category and the redundant area used for the regression of the parking space corner point.
A self-driving vehicle comprising the device according to claim 12 or 13.
An electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can perform any one of claims 1 to 7. The parking space detection model training method or the parking space detection method described in any one of claims 8 to 11.
A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the parking space detection model training method according to any one of claims 1 to 7 or claim 8 to the parking space detection method described in any one of 11.
A computer program product, comprising a computer program, wherein said computer program realizes the parking space detection model training method according to any one of claims 1 to 7 or any one of claims 8 to 11 when executed by a processor The parking space detection method.
A computer program, the computer program comprising computer program code, wherein when the computer program code is run on the computer, the computer is made to execute the parking space detection model training method or the right according to any one of claims 1 to 7 The parking space detection method described in any one of requirements 8 to 11.