CN113743151B - Method, device and storage medium for detecting road surface casting object - Google Patents
Method, device and storage medium for detecting road surface casting object Download PDFInfo
- Publication number
- CN113743151B CN113743151B CN202010458477.XA CN202010458477A CN113743151B CN 113743151 B CN113743151 B CN 113743151B CN 202010458477 A CN202010458477 A CN 202010458477A CN 113743151 B CN113743151 B CN 113743151B
- Authority
- CN
- China
- Prior art keywords
- image
- pavement
- connected domain
- suspected
- casting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000005266 casting Methods 0.000 title claims description 70
- 238000012545 processing Methods 0.000 claims abstract description 47
- 238000004891 communication Methods 0.000 claims abstract description 40
- 230000011218 segmentation Effects 0.000 claims description 52
- 238000001514 detection method Methods 0.000 claims description 39
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012216 screening Methods 0.000 description 15
- 238000013461 design Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 7
- 230000008878 coupling Effects 0.000 description 6
- 238000010168 coupling process Methods 0.000 description 6
- 238000005859 coupling reaction Methods 0.000 description 6
- 238000013523 data management Methods 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 5
- 241001465754 Metazoa Species 0.000 description 3
- 238000000638 solvent extraction Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The embodiment of the application provides a method, a device and a storage medium for detecting road surface casts, wherein the method comprises the following steps: acquiring video data, wherein the video data comprises a plurality of video frames with pavement mask information; respectively carrying out gray scale processing on each video frame to obtain a first image; determining at least one first communication domain in the first image; obtaining a suspected throwing object area according to the first image and the at least one first communicating area; determining a target connected domain from the suspected throwing object area; and determining the target connected domain as a throwing object area. According to the scheme, the real foreign matter state of the road surface can be analyzed with high precision, various types of throwing objects can be detected, and the method is not limited to the size, the spatial characteristics, the types and the motion state of the throwing objects.
Description
Technical Field
The embodiment of the application relates to the technical field of machine vision, in particular to a method, a device and a storage medium for detecting road surface casting objects.
Background
In the process of high-speed running, the throwing objects often appear due to artificial and non-artificial factors, so as to ensure the safety on the expressway, the existing throwing object detection system mainly based on vision detects the throwing objects in the expressway.
In the research and practice process of the prior art, the inventor of the embodiment of the application finds that the existing vision-based junk detection system is mostly a target detection scheme, and because the types of junk are too many and the sizes of the junk are uncertain, only a few types of junk can be detected, training data are difficult to obtain, and the robustness of the junk detection system is low.
Disclosure of Invention
The embodiment of the application provides a method, a device and a storage medium for detecting a road surface casting object, which can improve the high-precision analysis of the real foreign matter state of the road surface, and detect various casting object types, and are not limited to casting object sizes, spatial characteristics, types and motion states.
In a first aspect, an embodiment of the present application provides a method for detecting a road surface casting, the method including:
Acquiring video data, wherein the video data comprises a plurality of video frames with pavement mask information;
respectively carrying out gray scale processing on each video frame to obtain a first image;
Determining at least one first communication domain in the first image;
obtaining a suspected throwing object area according to the first image and the at least one first communicating area;
determining a target connected domain from the suspected throwing object area;
and determining the target connected domain as a throwing object area.
In one possible design, the determining the target connected domain from the suspected throwing object area includes:
detecting a vehicle region in the first image;
And deleting the first communication domain including the center point if the center point of the vehicle region is located in the first communication domain.
In one possible design, the determining the target connected domain from the suspected throwing object area includes:
respectively calculating a segmentation threshold value corresponding to each first communication domain;
And determining the target connected domain from the at least one first connected domain, wherein the target connected domain is the first connected domain of which the segmentation threshold value is out of a preset threshold value range.
In one possible design, the obtaining the suspected throwing object area according to the first image and the at least one first connected domain includes:
Setting the pixel point value in each first communication domain to be 1 to obtain a second image;
taking the area with the gray value of 1 in the second image as a pavement coverage area;
performing exclusive-or processing on the first image and the second image to obtain an exclusive-or processed image;
and taking the region with the gray value of 1 in the image as the suspected throwing object region.
In one possible design, the acquiring video data includes:
Acquiring initial video data;
Performing pavement semantic segmentation on the initial video data to obtain a pavement mask map, wherein the pavement mask map comprises pavement and ground mark lines;
And combining the pavement and the ground marking line into pavement mask information.
In one possible design, the performing pavement semantic segmentation on the initial video data to obtain a pavement mask map includes:
Scaling each video frame in the initial video data respectively to obtain a plurality of corresponding target video frames with preset sizes;
carrying out semantic segmentation on each target video frame to obtain a first characteristic map;
Shrinking the first characteristic map according to a preset size, and respectively endowing a plurality of weight factors to corresponding positions in the first characteristic map to obtain a second characteristic map;
And decoding the second characteristic map to obtain the pavement mask map.
In one possible design, after the determining the target connected domain as the casting area, the method further includes:
And generating a casting event, wherein the casting event records a video frame corresponding to the casting, position information of the casting and equipment identification of shooting equipment for shooting the casting.
In a second aspect, an embodiment of the present application provides a road surface casting detection apparatus having a function of implementing a method for detecting a road surface casting corresponding to the above first aspect. The functions may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above, which may be software and/or hardware.
In one possible design, the road surface casting detection device includes:
an acquisition module for acquiring video data, the video data comprising a plurality of video frames having pavement mask information;
The processing module is used for respectively carrying out gray processing on each video frame to obtain a first image; determining at least one first communication domain in the first image; obtaining a suspected throwing object area according to the first image and the at least one first communicating area; determining a target connected domain from the suspected throwing object area; and determining the target connected domain as a throwing object area.
In one possible design, the pavement casting detection device further includes a detection module, and the processing module is specifically configured to:
Detecting, by the detection module, a vehicle region in the first image;
And deleting the first communication domain including the center point if the center point of the vehicle region is located in the first communication domain.
In one possible design, the processing module is specifically configured to:
respectively calculating a segmentation threshold value corresponding to each first communication domain;
And determining the target connected domain from the at least one first connected domain, wherein the target connected domain is the first connected domain of which the segmentation threshold value is out of a preset threshold value range.
In one possible design, the processing module is specifically configured to:
Setting the pixel point value in each first communication domain to be 1 to obtain a second image;
taking the area with the gray value of 1 in the second image as a pavement coverage area;
performing exclusive-or processing on the first image and the second image to obtain an exclusive-or processed image;
and taking the region with the gray value of 1 in the image as the suspected throwing object region.
In one possible design, the obtaining module is specifically configured to:
Acquiring initial video data;
Performing pavement semantic segmentation on the initial video data to obtain a pavement mask map, wherein the pavement mask map comprises pavement and ground mark lines;
And combining the pavement and the ground marking line into pavement mask information.
In one possible design, the processing module is specifically configured to:
Scaling each video frame in the initial video data respectively to obtain a plurality of corresponding target video frames with preset sizes;
carrying out semantic segmentation on each target video frame to obtain a first characteristic map;
Shrinking the first characteristic map according to a preset size, and respectively endowing a plurality of weight factors to corresponding positions in the first characteristic map to obtain a second characteristic map;
And decoding the second characteristic map to obtain the pavement mask map.
In one possible design, after the processing module determines the target connected domain as the casting area, the processing module is further configured to:
And generating a casting event, wherein the casting event records a video frame corresponding to the casting, position information of the casting and equipment identification of shooting equipment for shooting the casting.
In a further aspect, an embodiment of the present application provides a computer device, which includes at least one connected processor and a memory, where the memory is configured to store a computer program, and the processor is configured to invoke the computer program in the memory to perform the method according to the first aspect.
A further aspect of an embodiment of the application provides a computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of the first aspect described above.
Compared with the prior art, in the scheme provided by the embodiment of the application, since the first image is obtained by respectively carrying out gray processing on each video frame, objects (such as vehicles, objects to be thrown, and the like) or other moving objects such as people and animals can be effectively separated from the road surface, so that at least one first communication domain in the first image can be conveniently determined later, a suspected object throwing area is obtained according to the first image and the at least one first communication domain, and then the target communication domain comprising the objects to be thrown is positioned according to the suspected object throwing area. Compared with the prior art, the method and the device can analyze the real foreign matter state of the road surface with high precision, detect various types of throwing objects, and are not limited to the size, the spatial characteristics, the types and the motion state of the throwing objects.
Drawings
FIG. 1 is a schematic diagram of a frame of a pavement casting detection apparatus according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of a method for detecting road surface sprinklers according to an embodiment of the present application;
FIG. 3 is a schematic diagram of pavement semantic segmentation of initial video data based on a segmentation model according to an embodiment of the present application;
FIG. 4a is a schematic diagram of a first image according to an embodiment of the present application;
FIG. 4b is a schematic diagram of the first connected domain found in FIG. 4a according to an embodiment of the present application;
FIG. 4c is a schematic diagram of a binarized second image obtained after setting the internal region of the first domain to true according to an embodiment of the present application;
FIG. 4d is a schematic illustration of a suspected projectile region according to an embodiment of the application;
FIG. 5 is a schematic diagram of a vehicle detection frame-based connected domain cnt k according to an embodiment of the present application;
FIG. 6 is a schematic view of a detected sprinkle-zone based on a rectangular label frame according to an embodiment of the present application;
FIG. 7a is a schematic diagram of a video frame in which no sprinklers are detected in an embodiment of the present application;
FIG. 7b is a schematic diagram of a video frame of an embodiment of the present application in which a casting is detected;
FIG. 8 is a schematic diagram of a road surface casting detection apparatus according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
The terms first, second and the like in the description and in the claims of embodiments of the application and in the above-described figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or modules is not necessarily limited to those explicitly listed but may include other steps or modules not expressly listed or inherent to such process, method, article, or apparatus, such that the partitioning of modules by embodiments of the application is only one logical partitioning, may be implemented with additional partitioning, such as a plurality of modules may be combined or integrated in another system, or some features may be omitted, or not implemented, and further, such that the coupling or direct coupling or communication connection between modules may be via some interfaces, indirect coupling or communication connection between modules may be electrical or otherwise similar, none of which are limited in embodiments of the application. The modules or sub-modules described as separate components may or may not be physically separate, may or may not be physical modules, or may be distributed in a plurality of circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purposes of the embodiment of the present application.
The embodiment of the application provides a method, a device and a storage medium for detecting road surface casting objects, which can be used for a server or a terminal side (only the server side is taken as an example in the embodiment of the application), wherein the server side can be used for detecting the foreign matters on the road surface, namely the road surface casting objects, such as objects which stay on the road surface for a long time under the scenes of people on vehicles, high altitude, ground and the like and have potential safety hazards for road traffic. In some embodiments, the method may be applied to a road surface casting detection apparatus as shown in fig. 1, which may be deployed in a server. The road surface casting detection device mainly comprises an acquisition module, a segmentation module, a detection module, a screening module and a data management module. The following is a detailed description.
The acquisition module is used for acquiring video data of the road surface shot by the shooting equipment in the road, and in some embodiments, the acquisition module can be integrated in the shooting equipment and can also be deployed in a cloud server. The embodiment of the application does not limit whether the modules in the pavement casting detection device are in separated deployment or centralized deployment.
The segmentation module is used for carrying out pavement semantic segmentation on the video data from the acquisition module so as to obtain a pavement mask map.
And the detection module is used for carrying out gray processing on the pavement mask map obtained by the segmentation module, and then detecting a connected domain region containing foreign matters from the image after gray processing.
And the screening module is used for screening the connected domain with obvious foreign matters with high probability from the plurality of connected domains detected by the detection module.
And the data management module is used for storing the video frames with foreign matters, the equipment identifiers of the shooting equipment and the positions of the shooting equipment screened by the screening module. And uploading the data to the cloud platform to update the database.
It should be specifically noted that, the server according to the embodiment of the present application may be an independent physical server, or may be a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms.
The embodiment of the application mainly provides the following technical scheme:
1. The method comprises the steps of carrying out semantic segmentation on a road surface based on a semantic segmentation algorithm of a convolutional neural network, merging the segmented road surface and a ground identification line into road surface mask information, determining a cavity in the road surface mask information as a suspected throwing object in a segmentation result, obtaining a closed operation binarization picture and a connected domain binarization picture based on the road surface mask information, determining a suspected throwing object area according to the closed operation binarization picture and the connected domain binarization picture, positioning boundary information of a vehicle according to the suspected throwing object area, and finally determining that a central point of the vehicle is in the connected domain, wherein if a threshold value corresponding to the connected domain is in a threshold value range, the connected domain is determined to be a road surface interference area. Compared with the prior art, the method and the device can analyze the real foreign matter state of the road surface with high precision, detect various types of throwing objects, and are not limited to the size, the spatial characteristics, the types and the motion state of the throwing objects.
2. The division size and the size number matched with the calculation force are selected for the multi-scale division module according to the calculation force of each processor, so that the algorithm engineering can carry out flexible and changeable deployment schemes according to different task demands. Therefore, the scheme has general applicability, is not only suitable for the deployment of the server GPU end with high computation power, but also can be deployed on the CPU of the mobile end.
Referring to fig. 2, a method for detecting a road surface casting according to an embodiment of the present application is described below, and the method may be performed by a road surface casting detection device, which may be disposed on a server or a terminal. In some embodiments, the terminal may be a road side unit, a vehicle-mounted unit or a mobile terminal, which is not limited by the embodiment of the present application. The server may be a cloud server. In some embodiments. The method may also be implemented by the terminal in conjunction with a server, e.g., the terminal is a slave node in a distributed system deployment and the server is a master node in the distributed system. The embodiment of the application does not limit the deployment scenario. The embodiment of the application comprises the following steps:
201. Video data is acquired.
The video data comprises a plurality of video frames with pavement mask information, and the pavement mask information indicates that detection objects exist in the video frames, namely the detection objects in the video frames are used for positioning or identifying, and the detection objects may be suspected throwing objects.
In some embodiments, the acquiring video data includes:
(1) Initial video data is acquired.
(2) And carrying out pavement semantic segmentation on the initial video data to obtain a pavement mask map.
The pavement mask includes pavement and ground marking lines.
In some embodiments, the pavement semantic segmentation may be performed on the initial video data based on a semantic segmentation algorithm of a convolutional neural network, and the algorithm adopted by the semantic segmentation is not limited in the embodiment of the present application.
In other embodiments, it is contemplated that the accuracy of the segmentation model directly affects the accuracy of the final projectile detection due to the numerous projectile categories and the varying shapes. To obtain a more robust segmentation result, a segmentation strategy of multi-size input and weighted voting may also be used. Specifically, the above-described pavement mask map can be obtained by:
Scaling each video frame in the initial video data respectively to obtain a plurality of corresponding target video frames with preset sizes;
carrying out semantic segmentation on each target video frame to obtain a first characteristic map;
Shrinking the first characteristic map according to a preset size, and respectively endowing a plurality of weight factors to corresponding positions in the first characteristic map to obtain a second characteristic map;
And decoding the second characteristic map to obtain the pavement mask map.
For example, a segmentation model as shown in fig. 3 may be used to perform pavement semantic segmentation on the initial video data. Firstly, scaling a video frame (i.e. a picture) to be detected into n (n is a positive integer) sizes, and sending all pictures with different sizes into a segmentation model to finally obtain n-size feature maps. And then re-scaling the n feature maps to uniform size, multiplying the n feature maps by corresponding weight factors, scale 1、scale 2、scale 3…scale n, and adding the corresponding positions to obtain a final feature map. And finally, obtaining a binary mask map of the pavement segmentation of the current frame through decoding the characteristic map, namely the pavement mask map.
(3) And combining the pavement and the ground marking line into pavement mask information.
The voids in the pavement mask information are highly probable as casting on the pavement.
202. And respectively carrying out gray scale processing on each video frame to obtain a first image.
In some embodiments, the gray scale processing may include binarization processing, penta-binarization processing, and the like, and the embodiment of the present application does not limit the manner of gray scale processing. As shown in fig. 4a, the first image is a binarized image obtained by performing binarization processing on each video frame.
203. At least one first connected domain in the first image is determined.
The first connected domain refers to a road surface area in the first image, and the first connected domain may be a maximum connected domain. In the embodiment of the application, the connected domain (Connected Component) refers to an image Region (Region, blob) formed by foreground pixel points which have the same gray value and are adjacent in position in the image. Visually, the pixel points in the first image that are connected to each other form one region, and the pixel points that are not connected form a different region. Then, a set of all the pixel points connected to each other may be referred to as a connected region. As shown in fig. 4b, the first connected domain shown in fig. 4b is a connected domain found from the first image shown in fig. 4 a.
204. And obtaining a suspected throwing object area according to the first image and the at least one first communicating area.
The suspected throwing object area refers to an area possibly comprising throwing objects in the first image.
Specifically, since each video frame is a video frame including road surface mask information, a closed-loop binary image (that is, described below) and a connected-domain binary image (that is, a first connected domain) can be obtained based on the road surface mask information, and a suspected throwing object area can be determined based on the closed-loop binary image and the first connected domain.
In some embodiments, the obtaining the suspected throwing object area according to the first image and the at least one first connected domain includes:
Setting the pixel point value in each first communication domain to be 1 to obtain a second image;
taking the area with the gray value of 1 in the second image as a pavement coverage area;
performing exclusive-or processing on the first image and the second image to obtain an exclusive-or processed image;
and taking the region with the gray value of 1 in the image after the exclusive or treatment as the suspected throwing object region.
The exclusive-or processing refers to that pixel values of pixel points in a first image and pixel values of pixel points corresponding to the pixel point positions in a second image are subjected to bit-wise exclusive-or operation, if the pixel values are the same, the exclusive-or operation result is 0, and if the pixel values are different, the exclusive-or result is 1, and then the image after the exclusive-or processing is obtained.
For example, the binarization processing may be performed on the first connected domain, or the fifth binarization processing may be performed on the first connected domain, which is not limited in the embodiment of the present application. Fig. 4c is a binarized second image obtained by setting the internal area of the first connected domain shown in fig. 4b to true, i.e., the black hatched portion in fig. 4c is the first connected domain, i.e., the road surface coverage area. After binarization processing, the road surface pixel points and the non-road surface pixel points (namely vehicles and foreign matters) in the first image can be separated, because the color blocks of the road surface are not communicated with the non-road surface color blocks.
Then, based on fig. 4c, the maximum connected domain shown in fig. 4b and the second image shown in fig. 4c are subjected to exclusive or, and after the exclusive or, the remaining True area is the suspected throwing object area, for example, after the exclusive or, the suspected throwing object area shown in fig. 4d is finally obtained. The unshaded area in fig. 4d may be referred to as a hole in the pavement mask information, where the hole is a region where the to-be-found casting is located with a high probability.
Therefore, the road surface coverage area is found from the second image through the embodiment, then the basis can be provided for the subsequent use of the road surface coverage area to reversely push the area where the throwing object is located, and the accuracy rate of identifying the existence of the throwing object in the second image can be improved on the premise that the type and the shape of the throwing object do not need to be concerned, and the robustness is high.
205. And determining a target connected domain from the suspected throwing object area.
The determination of the target connected domain from the suspected throwing object area is described below from two angles of a vehicle detection screening mode and a segmentation threshold screening mode, namely, throwing objects existing in the road surface are obtained through screening.
(1) Determining a target connected domain from the suspected throwing object area based on a vehicle detection screening mode
In some embodiments, since the obtained suspected throwing object area may be the vehicle mask information on the expressway, the vehicle category appearing on the expressway may be detected to obtain the contour information (bounding box) of the vehicle. Correspondingly, the communication domain can be searched again for the suspected throwing object region to find out the road surface interference region in the road, namely, the road surface interference region is screened out through vehicle detection, and finally the target communication domain is obtained through back-pushing. Specifically, the determining the target connected domain from the suspected throwing object area includes:
Detecting a vehicle region in the first image; if the center point of the vehicle region is located in the suspected throwing object region and the segmentation threshold is determined not to be located in the preset pavement pixel interval, determining that no throwing object exists in the suspected throwing object region, namely, the communicating region corresponding to the suspected throwing object region is an invalid communicating region, and the invalid communicating region belongs to a pavement interference region, so that the finally remaining communicating region is taken as a target communicating region.
Let M connected domains be present, and the ith is denoted as cnt i. If the center point of the vehicle detection frame is located in the kth connected domain cnt k, deleting the connected domain cnt k, where M, i and k are both positive integers. As shown in fig. 5, the rectangular frame in fig. 5 is a vehicle detection frame, and the center point of the vehicle detection frame is located in cnt k, so that the connected domain cnt k is deleted.
Therefore, the road surface interference area can be effectively screened out by screening the determined suspected throwing object areas based on the vehicle center point, and then the throwing object area truly comprising the throwing object is obtained, namely, the range of the area where the throwing object is located can be further reduced by screening, so that the subsequent analysis operation on the throwing object is simplified.
(2) Determining a target connected domain from the suspected throwing object area based on a segmentation threshold screening mode
In some embodiments, considering that there may be a region with abrupt pixel change in the suspected sprinkle region, road information such as a ground light spot may be misdetected as a sprinkle. In order to further improve the system detection accuracy and the robustness. The following segmentation threshold screening method may be further adopted to determine a target connected domain from the suspected throwing object area:
respectively calculating a segmentation threshold value corresponding to each first communication domain;
And determining the target connected domain from the at least one first connected domain, wherein the target connected domain is the first connected domain of which the segmentation threshold value is out of a preset threshold value range.
Specifically, if the segmentation threshold value is within a preset threshold value range, determining a first connected domain of which the segmentation threshold value is within the preset threshold value range as a road surface interference region; if the segmentation threshold value is out of the preset threshold value range, determining that a first connected domain of which the segmentation threshold value is out of the preset threshold value range is a casting object region, and taking the connected domain corresponding to each casting object region as the target connected domain.
In some embodiments, if the target connected domain determined by the segmentation model is used, a segmentation threshold (OTSU), abbreviated as T OTSU, may be calculated for the region corresponding to the connected domain. The T OTSU is used to separate the object and the background in the area where the first connection domain is located, specifically, a T OTSU is determined first, then the gray value of each pixel is compared with a threshold value, and the pixel is divided into the object and the background according to the comparison result. The algorithm used for the segmentation threshold is not limited in the embodiment of the application.
206. And determining the target connected domain as a throwing object area.
The throwing object area refers to an area where the throwing object is located in the first image, as shown in fig. 6, and the rectangular labeling frame is the detected throwing object area.
Screening out the remaining casting areas by the screening module shown in fig. 1 generates casting events by the cloud data management module, and records corresponding frames of casting time, corresponding GPS longitude and latitude information and camera numbers. For example, after the video frames are collected, the video frames are screened by a screening module, the video frames are divided into video frames in which the casting objects are detected and video frames in which the casting objects are not detected, and then the casting objects in the video frames in which the casting objects are detected are marked and then uploaded to a data management module of the cloud. FIG. 7a is a schematic illustration of a non-uploading example (i.e., a video frame in which no casting is detected); fig. 7b is a schematic illustration of an example of uploading (i.e., a video frame in which a casting is detected). The embedded equipment transmits the acquired video frames of the safety helmet which is not worn correctly, GPS longitude and latitude information and the accident vehicle camera number to the data management module through the 4G, and the data management module stores the information and updates the database.
In the embodiment of the application, since the first images are obtained by respectively carrying out gray processing on each video frame, objects (such as vehicles, objects to be thrown, and the like) or other moving objects such as people, animals, and the like existing on the road surface can be effectively separated from the road surface, so that at least one first communication domain in the first images can be conveniently and subsequently determined, a suspected object throwing area is obtained according to the first images and the at least one first communication domain, and then the target communication domain comprising the objects to be thrown is positioned according to the suspected object throwing area. Compared with the prior art, the method and the device can analyze the real foreign matter state of the road surface with high precision, detect various types of throwing objects, and are not limited to the size, the spatial characteristics, the types and the motion state of the throwing objects.
Any technical features mentioned in the embodiment corresponding to any one of fig. 1 to fig. 7b are also applicable to the embodiment corresponding to fig. 8 and fig. 9 in the embodiment of the present application, and the following similar parts will not be repeated.
The method for detecting the road surface casting object in the embodiment of the application is described above, and the following description is given to the device for executing the method for detecting the road surface casting object.
Referring to fig. 8, a schematic structural diagram of a road surface casting detection device 80 shown in fig. 8 may be applied to a server or a terminal, and the road surface casting detection device 80 may be used for detecting foreign matters on a road surface, such as objects that stay on the road surface for a long time and have potential safety hazards for road traffic in scenes of people, high altitudes, ground, etc. on a vehicle. The road surface casting detection apparatus 80 in the embodiment of the present application can implement the steps corresponding to the method for detecting a road surface casting performed in the embodiment corresponding to any one of fig. 1 to 7b described above. The road surface casting detection device 80 includes:
An obtaining module 801, configured to obtain video data, where the video data includes a plurality of video frames with pavement mask information;
A processing module 802, configured to perform gray processing on each video frame to obtain a first image; determining at least one first communication domain in the first image; obtaining a suspected throwing object area according to the first image and the at least one first communicating area; determining a target connected domain from the suspected throwing object area; and determining the target connected domain as a throwing object area.
In the embodiment of the present application, since the first image is obtained by respectively performing gray processing on each video frame, the processing module 802 can effectively separate an object (such as a vehicle, a throwing object, etc.) or other moving objects such as a person, an animal, etc. existing on a road surface from the road surface, so as to facilitate subsequent determination of at least one first communication domain in the first image, obtain a suspected throwing object region according to the first image and the at least one first communication domain, and then position a target communication domain including the throwing object according to the suspected throwing object region. Compared with the prior art, the method and the device can analyze the real foreign matter state of the road surface with high precision, detect various types of throwing objects, and are not limited to the size, the spatial characteristics, the types and the motion state of the throwing objects.
In some embodiments, the pavement projectile apparatus further includes a detection module 803, and the processing module 802 is specifically configured to:
detecting a vehicle region in the first image by the detection module 803;
And deleting the first communication domain including the center point if the center point of the vehicle region is located in the first communication domain.
In some embodiments, the processing module 802 is specifically configured to:
respectively calculating a segmentation threshold value corresponding to each first communication domain;
And determining the target connected domain from the at least one first connected domain, wherein the target connected domain is the first connected domain of which the segmentation threshold value is out of a preset threshold value range.
In some embodiments, the processing module 802 is specifically configured to:
Setting the pixel point value in each first communication domain to be 1 to obtain a second image;
taking the area with the gray value of 1 in the second image as a pavement coverage area;
performing exclusive-or processing on the first image and the second image to obtain an exclusive-or processed image;
and taking the region with the gray value of 1 in the image as the suspected throwing object region.
In some embodiments, the obtaining module 801 is specifically configured to:
Acquiring initial video data;
Performing pavement semantic segmentation on the initial video data to obtain a pavement mask map, wherein the pavement mask map comprises pavement and ground mark lines;
And combining the pavement and the ground marking line into pavement mask information.
In some embodiments, the processing module 802 is specifically configured to:
Scaling each video frame in the initial video data respectively to obtain a plurality of corresponding target video frames with preset sizes;
carrying out semantic segmentation on each target video frame to obtain a first characteristic map;
Shrinking the first characteristic map according to a preset size, and respectively endowing a plurality of weight factors to corresponding positions in the first characteristic map to obtain a second characteristic map;
And decoding the second characteristic map to obtain the pavement mask map.
In some embodiments, after the processing module 802 determines the target connected domain as the casting area, the processing module is further configured to:
And generating a casting event, wherein the casting event records a video frame corresponding to the casting, position information of the casting and equipment identification of shooting equipment for shooting the casting.
The network authentication server and the terminal device in the embodiment of the present application are described above from the point of view of the modularized functional entity, and the network authentication server and the terminal device in the embodiment of the present application are described below from the point of view of hardware processing, respectively.
It should be noted that, in the embodiment of the present application shown in fig. 8, the entity device corresponding to the acquisition module may be a transceiver or a processor, and the entity device corresponding to the detection module and the processing module may be a processor. Each of the devices shown in fig. 8 may have a structure as shown in fig. 9, and when one of the devices has a structure as shown in fig. 9, the processor, the transmitter and the receiver in fig. 9 implement the same or similar functions as the processing module, the acquiring module and the detecting module provided in the foregoing embodiment of the device corresponding to the device, and the memory in fig. 9 stores a computer program to be invoked when the processor executes the foregoing method for detecting a road surface casting.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the systems, apparatuses and modules described above may refer to the corresponding processes in the foregoing method embodiments, which are not repeated herein.
In the embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
The modules illustrated as separate components may or may not be physically separate, the components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the flow or functions according to the embodiments of the present application are fully or partially produced. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk Solid STATE DISK (SSD)), etc.
The above description has been made in detail on the technical solutions provided by the embodiments of the present application, and specific examples are applied in the embodiments of the present application to illustrate the principles and implementation manners of the embodiments of the present application, where the above description of the embodiments is only for helping to understand the methods and core ideas of the embodiments of the present application; meanwhile, as for those skilled in the art, according to the idea of the embodiment of the present application, there are various changes in the specific implementation and application scope, and in summary, the present disclosure should not be construed as limiting the embodiment of the present application.
Claims (9)
1. A method of detecting road surface sprinklers, the method comprising:
Acquiring video data, wherein the video data comprises a plurality of video frames with pavement mask information;
respectively carrying out gray scale processing on each video frame to obtain a first image;
Determining at least one first communication domain in the first image; the first connected domain is a road surface region in the first image;
obtaining a suspected throwing object area according to the first image and the at least one first communicating area;
determining a target connected domain from the suspected throwing object area;
Determining the target connected domain as a casting area;
The obtaining a suspected throwing object area according to the first image and the at least one first connected domain includes:
setting pixel point values in each first communication domain to obtain a second image;
performing bit exclusive OR operation on pixel values of pixel points in the first image and pixel values of pixel points corresponding to the pixel points in the second image to obtain an exclusive OR processed image;
And taking the areas with different exclusive or operation results in the exclusive or processed image as the suspected throwing object areas.
2. The method of claim 1, wherein the determining a target connected domain from the suspected projectile region comprises:
detecting a vehicle region in the first image;
And deleting the first communication domain including the center point if the center point of the vehicle region is located in the first communication domain.
3. The method according to claim 1 or 2, wherein said determining a target connected domain from said suspected projectile region comprises:
respectively calculating a segmentation threshold value corresponding to each first communication domain;
And determining the target connected domain from the at least one first connected domain, wherein the target connected domain is the first connected domain of which the segmentation threshold value is out of a preset threshold value range.
4. The method of claim 1, wherein the acquiring video data comprises:
Acquiring initial video data;
Performing pavement semantic segmentation on the initial video data to obtain a pavement mask map, wherein the pavement mask map comprises pavement and ground mark lines;
And combining the pavement and the ground marking line into pavement mask information.
5. The method of claim 4, wherein the performing the pavement semantic segmentation on the initial video data to obtain a pavement mask map comprises:
Scaling each video frame in the initial video data respectively to obtain a plurality of corresponding target video frames with preset sizes;
carrying out semantic segmentation on each target video frame to obtain a first characteristic map;
Shrinking the first characteristic map according to a preset size, and respectively endowing a plurality of weight factors to corresponding positions in the first characteristic map to obtain a second characteristic map;
And decoding the second characteristic map to obtain the pavement mask map.
6. The method of any one of claims 1, 4, 5, wherein after the determining the target connected domain as a casting area, the method further comprises:
And generating a casting event, wherein the casting event records a video frame corresponding to the casting, position information of the casting and equipment identification of shooting equipment for shooting the casting.
7. The utility model provides a road surface casting detection device which characterized in that, the road surface casting detection device includes:
an acquisition module for acquiring video data, the video data comprising a plurality of video frames having pavement mask information;
The processing module is used for respectively carrying out gray processing on each video frame to obtain a first image; determining at least one first connected domain in the first image, wherein the first connected domain is a pavement area in the first image; obtaining a suspected throwing object area according to the first image and the at least one first communicating area; determining a target connected domain from the suspected throwing object area; determining the target connected domain as a casting area;
the processing module is specifically configured to:
setting pixel point values in each first communication domain to obtain a second image;
performing bit exclusive OR operation on pixel values of pixel points in the first image and pixel values of pixel points corresponding to the pixel points in the second image to obtain an exclusive OR processed image;
And taking the areas with different exclusive or operation results in the exclusive or processed image as the suspected throwing object areas.
8. A computer device, the computer device comprising:
At least one processor and memory;
Wherein the memory is for storing a computer program and the processor is for invoking the computer program stored in the memory to perform the method of any of claims 1-6.
9. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010458477.XA CN113743151B (en) | 2020-05-27 | 2020-05-27 | Method, device and storage medium for detecting road surface casting object |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010458477.XA CN113743151B (en) | 2020-05-27 | 2020-05-27 | Method, device and storage medium for detecting road surface casting object |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113743151A CN113743151A (en) | 2021-12-03 |
| CN113743151B true CN113743151B (en) | 2024-08-02 |
Family
ID=78723558
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010458477.XA Active CN113743151B (en) | 2020-05-27 | 2020-05-27 | Method, device and storage medium for detecting road surface casting object |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113743151B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| BR102021019290A2 (en) * | 2020-10-27 | 2022-05-10 | Cnh Industrial America Llc | System and method for monitoring spray quality |
| CN114299002A (en) * | 2021-12-24 | 2022-04-08 | 中用科技有限公司 | Intelligent detection system and method for abnormal road surface throwing behavior |
| CN116311161B (en) * | 2023-03-08 | 2024-09-17 | 中国矿业大学 | Road surface casting object detection method based on joint multitasking model |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109886219A (en) * | 2019-02-26 | 2019-06-14 | 中兴飞流信息科技有限公司 | Shed object detecting method, device and computer readable storage medium |
| CN110619651A (en) * | 2019-09-09 | 2019-12-27 | 博云视觉（北京）科技有限公司 | Driving road segmentation method based on monitoring video |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105957077B (en) * | 2015-04-29 | 2019-01-15 | 国网河南省电力公司电力科学研究院 | The electric line foreign matter detection method of view-based access control model significance analysis |
| CN104809449B (en) * | 2015-05-14 | 2018-09-21 | 重庆大学 | Track dotted line line of demarcation automatic testing method suitable for highway video monitoring system |
| CN106297278B (en) * | 2015-05-18 | 2019-12-20 | 杭州海康威视数字技术股份有限公司 | Method and system for querying a projectile vehicle |
| US10872244B2 (en) * | 2015-08-31 | 2020-12-22 | Intel Corporation | Road marking extraction from in-vehicle video |
| CN105839505B (en) * | 2016-03-18 | 2017-09-29 | 交通运输部科学研究院 | The detection method and detection means of a kind of road surface breakage information of three-dimensional visualization |
| CN106022336B (en) * | 2016-05-17 | 2019-05-07 | 英飞拓(杭州)信息系统技术有限公司 | A kind of detection method of license plate for capturing image based on highway |
| CN107613244A (en) * | 2016-07-08 | 2018-01-19 | 杭州海康威视数字技术股份有限公司 | A kind of navigation channel monitoring objective acquisition methods and device |
| CN111127507A (en) * | 2019-12-18 | 2020-05-08 | 成都通甲优博科技有限责任公司 | Method and system for determining throwing object |
-
2020
- 2020-05-27 CN CN202010458477.XA patent/CN113743151B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109886219A (en) * | 2019-02-26 | 2019-06-14 | 中兴飞流信息科技有限公司 | Shed object detecting method, device and computer readable storage medium |
| CN110619651A (en) * | 2019-09-09 | 2019-12-27 | 博云视觉（北京）科技有限公司 | Driving road segmentation method based on monitoring video |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113743151A (en) | 2021-12-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113743151B (en) | Method, device and storage medium for detecting road surface casting object | |
| CN111723654B (en) | High-altitude parabolic detection method and device based on background modeling, YOLOv3 and self-optimization | |
| JP2023523243A (en) | Obstacle detection method and apparatus, computer device, and computer program | |
| CN110785719A (en) | Method and system for instant object tagging via cross temporal verification in autonomous vehicles | |
| CN105279484A (en) | Method and device for object detection | |
| CN110869559A (en) | Method and system for integrated global and distributed learning in autonomous vehicles | |
| CN110799982A (en) | Method and system for object-centric stereo vision in an autonomous vehicle | |
| CN112417953A (en) | Road condition detection and map data updating method, device, system and equipment | |
| CN108216252A (en) | A kind of subway driver vehicle carried driving behavior analysis method, car-mounted terminal and system | |
| CN110163109A (en) | A kind of lane line mask method and device | |
| Yaghoobi Ershadi et al. | Vehicle tracking and counting system in dusty weather with vibrating camera conditions | |
| Chang et al. | Locating waterfowl farms from satellite images with parallel residual u-net architecture | |
| CN111402301B (en) | Water accumulation detection method and device, storage medium and electronic device | |
| CN108229273B (en) | Method and device for training multilayer neural network model and recognizing road characteristics | |
| US20220366683A1 (en) | Artificial intuition based visual data extraction for distributed systems | |
| CN112183434A (en) | Building change detection method and device | |
| CN110969645A (en) | Unsupervised abnormal track detection method and unsupervised abnormal track detection device for crowded scenes | |
| Ballinas-Hernández et al. | Marked and unmarked speed bump detection for autonomous vehicles using stereo vision | |
| US20220172295A1 (en) | Systems, methods, and devices for aggregating and quantifying telematics data | |
| CN115719362B (en) | High-altitude parabolic detection method, system, equipment and storage medium | |
| CN114782496A (en) | Object tracking method and device, storage medium and electronic device | |
| JP7327355B2 (en) | Map update device and map update method | |
| CN116311205A (en) | License plate recognition method, license plate recognition device, electronic equipment and storage medium | |
| CN113850111A (en) | Road condition identification method and system based on semantic segmentation and neural network technology | |
| CN112732446A (en) | Task processing method and device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |