CN114387571A - Unsupervised vehicle re-identification method and unsupervised vehicle re-identification device based on hierarchical matching - Google Patents
Unsupervised vehicle re-identification method and unsupervised vehicle re-identification device based on hierarchical matching Download PDFInfo
- Publication number
- CN114387571A CN114387571A CN202111550495.1A CN202111550495A CN114387571A CN 114387571 A CN114387571 A CN 114387571A CN 202111550495 A CN202111550495 A CN 202111550495A CN 114387571 A CN114387571 A CN 114387571A
- Authority
- CN
- China
- Prior art keywords
- vehicle
- local
- detected
- distance value
- similarity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 239000013598 vector Substances 0.000 claims abstract description 68
- 238000012545 processing Methods 0.000 claims abstract description 30
- 230000004927 fusion Effects 0.000 claims abstract description 14
- 238000001514 detection method Methods 0.000 claims description 48
- 230000000007 visual effect Effects 0.000 claims description 31
- 238000004364 calculation method Methods 0.000 claims description 20
- 238000011176 pooling Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 206010061274 Malocclusion Diseases 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an unsupervised vehicle re-identification method and a device based on hierarchical matching, wherein the method comprises the following steps: acquiring a picture of a vehicle to be detected from a video source; extracting overall characteristics, vehicle local component information, vehicle body color information and vehicle type information of a vehicle from a picture of the vehicle to be detected; carrying out first processing on the overall characteristics of the vehicle to obtain the overall similarity of the vehicle; performing second processing on the vehicle local component information to obtain a vehicle local characteristic distance value; performing third processing on the vehicle body color information and the vehicle type information to obtain a color characteristic vector and a vehicle type characteristic vector; carrying out feature fusion on the overall vehicle similarity, the local vehicle feature distance value, the color feature vector and the vehicle type feature vector to obtain the vehicle similarity; and determining a target vehicle corresponding to the vehicle to be detected from all vehicles in the candidate vehicle set according to the vehicle similarity. The invention improves the accuracy of vehicle weight recognition and can be widely applied to the technical field of vehicle recognition.
Description
Technical Field
The invention relates to the technical field of vehicle identification, in particular to an unsupervised vehicle re-identification method and device based on hierarchy matching.
Background
The vehicle weight recognition problem is a retrieval problem of judging whether vehicles belong to the same vehicle in images shot by cameras installed in different areas under a specific traffic monitoring scene. The vehicle weight recognition algorithm can be applied to a plurality of difficult problems such as vehicle recognition, tracking and positioning, and the safety and reliability of the monitoring system are further improved.
Vehicle re-identification remains a very challenging problem due to the effects of constraints such as complex traffic scenes, limited camera viewpoints, illumination changes, etc. At present, the license plate recognition technology is the simplest and most direct method for judging the identity of a vehicle, but the method only depends on the license plate recognition technology, so that obvious disadvantages still exist. For example, in traffic monitoring, the shooting visual angle is relatively fixed due to the limitation of the position of the camera, and the difficulty of license plate detection is increased or the license plate detection is not applicable in complex scenes such as congestion queuing, large-scale vehicle shielding or the situation that a driver intentionally shields a fake license plate. To remedy the shortcomings of license plate recognition, more and more researchers are beginning to use visual information of vehicles other than license plates to perform vehicle retrieval tasks. In the past research in the visual field, vehicle retrieval mainly involves retrieval and identification of content information such as vehicle color or model, but this often does not make it possible to accurately retrieve a specific vehicle, but rather to obtain a vehicle having the same content (e.g., color, model) as the target vehicle. In recent years, research on vehicle re-recognition, a sub-field of vehicle search, has become an important research point. The disadvantages of the existing vehicle weight recognition technology are mainly reflected in two aspects: firstly, under different camera visual angles, the difference of the characteristics presented by the same vehicle is larger, so that the intra-class difference of the data of the vehicle re-identification task is larger; and the difference of different vehicles with similar global characteristics (such as colors, vehicle types and the like) under multiple visual angles is further reduced, so that the difference between classes of the vehicle re-identification task is smaller.
Disclosure of Invention
In view of this, the embodiment of the invention provides an unsupervised vehicle re-identification method and device based on hierarchical matching with high accuracy.
The invention provides an unsupervised vehicle re-identification method based on hierarchical matching, which comprises the following steps:
acquiring a picture of a vehicle to be detected from a video source;
extracting overall characteristics, local component information, body color information and vehicle type information of the vehicle from the picture of the vehicle to be detected;
carrying out first processing on the overall vehicle characteristics to obtain overall vehicle similarity;
performing second processing on the vehicle local component information to obtain a vehicle local characteristic distance value;
performing third processing on the vehicle body color information and the vehicle type information to obtain a color feature vector and a vehicle type feature vector;
performing feature fusion on the overall vehicle similarity, the local vehicle feature distance value, the color feature vector and the vehicle type feature vector to obtain vehicle similarity;
and determining a target vehicle corresponding to the vehicle to be detected from all vehicles in the candidate vehicle set according to the vehicle similarity.
Optionally, the acquiring a picture of a vehicle to be detected from a video source includes:
acquiring image frames from the video source according to a preset time interval;
carrying out target detection on the extracted image frames to find out vehicle targets in the image frames;
and intercepting the image frames with the vehicle targets to obtain a vehicle set, and dividing the vehicle set into a search image set and a candidate image set.
Optionally, the step of obtaining the picture of the vehicle to be detected from the video source further includes:
calculating the intersection ratio between the detection frame of any current image frame and the detection frame of the corresponding previous image frame;
when the intersection ratio is larger than a preset threshold value, removing a detection result corresponding to the current image frame;
wherein, the calculation formula of the intersection ratio is as follows:
wherein IoU represents the cross-over ratio; r1A detection frame for a previous image frame; r2A detection frame for a current image frame; area (R)1)∩area(R2) Solving the area intersection between the two detection frames; area (R)1)∪area(R2) And (5) solving the union set of the areas of the two detection frames.
Optionally, the extracting, from the picture of the vehicle to be detected, the overall characteristic of the vehicle, the local component information of the vehicle, the color information of the vehicle body, and the vehicle type information includes:
dividing the vehicle into vehicle overall characteristics and vehicle local component information through a ResNet50 characteristic extractor;
processing the picture of the vehicle to be detected through global average pooling to obtain the overall characteristics of the vehicle;
outputting the overall vehicle characteristics through a CNN network;
comparing the characteristics of each visual angle of the vehicle to obtain visual angle visibility scores of each visual angle of the vehicle;
after the weight of the visual angle visibility score is calculated, calculating by combining an Euclidean distance calculation method to obtain a local characteristic distance value, and determining vehicle local component information;
and performing feature extraction through a Resnet50 network to obtain color feature vectors and vehicle type feature vectors of all vehicles, and determining the vehicle body color information and the vehicle type information.
Optionally, the extracting, from the picture of the vehicle to be detected, the overall characteristic of the vehicle, the local component information of the vehicle, the color information of the vehicle body, and the vehicle type information further includes:
dividing the vehicles in the picture of the vehicle to be detected into four directions, namely a front side, a top side, a back side and a side;
and calculating area values of the four directions of the vehicle as visual angle visibility scores, and using the visual angle visibility scores as confidence values of local features corresponding to the vehicle, wherein the confidence values are the confidence values of the four local features of the vehicle and are used for representing the weight of the feature distance of each part between the vehicles in the subsequent calculation in the local feature distance.
Optionally, the calculation formula for calculating the weight of the visibility score of the viewing angle is as follows:
wherein,
a weight representing the perspective visibility score;
a visibility score representing a vehicle to be detected;
a visibility score representing a corresponding vehicle in the image library; m and n represent the vehicle to be detected and the corresponding vehicle in the image library; i represents a region corresponding to the visibility score and the weight; n represents the number of vehicles in the image library to be searched;
the calculation formula of the local characteristic distance value is as follows:
wherein D ism,nRepresenting the local feature distance value; d represents the Euclidean distance of each direction characteristic of the vehicle; f. ofi mCharacteristic information representing a vehicle to be detected; f. ofi nRepresenting corresponding vehicles in the image libraryCharacteristic information of the vehicle.
Optionally, the performing feature fusion on the overall vehicle similarity, the local vehicle feature distance value, the color feature vector, and the vehicle type feature vector to obtain the vehicle similarity includes:
performing L2 norm normalization processing on the vehicle overall similarity, the vehicle local characteristic distance value, the color characteristic vector and the vehicle type characteristic vector to obtain normalized characteristics;
calculating an initial re-identification distance value between the vehicle to be detected and the corresponding vehicle in the image library according to the normalized features;
carrying out weighted summation calculation on the initial re-identification distance value of the overall similarity of the vehicle, the initial re-identification distance value of the local characteristic distance value of the vehicle, the initial re-identification distance value of the color characteristic vector and the initial re-identification distance value of the vehicle type characteristic vector to obtain a total weight identification distance value;
and determining the similarity of the vehicle according to the total weight identification distance value.
In another aspect, an embodiment of the present invention further provides an unsupervised vehicle re-identification apparatus based on hierarchical matching, including:
the first module is used for acquiring a picture of the vehicle to be detected from a video source;
the second module is used for extracting the overall characteristics, the local component information, the color information and the model information of the vehicle from the picture of the vehicle to be detected;
the third module is used for carrying out first processing on the overall characteristics of the vehicle to obtain the overall similarity of the vehicle;
the fourth module is used for carrying out second processing on the vehicle local component information to obtain a vehicle local characteristic distance value;
the fifth module is used for performing third processing on the vehicle body color information and the vehicle type information to obtain a color feature vector and a vehicle type feature vector;
a sixth module, configured to perform feature fusion on the vehicle overall similarity, the vehicle local feature distance value, the color feature vector, and the vehicle type feature vector to obtain a vehicle similarity;
and the seventh module is used for determining a target vehicle corresponding to the vehicle to be detected from all vehicles in the candidate vehicle set according to the vehicle similarity.
Another aspect of the embodiments of the present invention further provides an electronic device, including a processor and a memory;
the memory is used for storing programs;
the processor executes the program to implement the method as described above.
Yet another aspect of the embodiments of the present invention provides a computer-readable storage medium, which stores a program, which is executed by a processor to implement the method as described above.
The embodiment of the invention also discloses a computer program product or a computer program, which comprises computer instructions, and the computer instructions are stored in a computer readable storage medium. The computer instructions may be read by a processor of a computer device from a computer-readable storage medium, and the computer instructions executed by the processor cause the computer device to perform the foregoing method.
The embodiment of the invention obtains the picture of the vehicle to be detected from the video source; extracting overall characteristics, local component information, body color information and vehicle type information of the vehicle from the picture of the vehicle to be detected; carrying out first processing on the overall vehicle characteristics to obtain overall vehicle similarity; performing second processing on the vehicle local component information to obtain a vehicle local characteristic distance value; performing third processing on the vehicle body color information and the vehicle type information to obtain a color feature vector and a vehicle type feature vector; performing feature fusion on the overall vehicle similarity, the local vehicle feature distance value, the color feature vector and the vehicle type feature vector to obtain vehicle similarity; and determining a target vehicle corresponding to the vehicle to be detected from all vehicles in the candidate vehicle set according to the vehicle similarity. The invention improves the accuracy of vehicle weight identification.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flowchart illustrating the overall steps provided by an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Aiming at the problems in the prior art, the embodiment of the invention provides an unsupervised vehicle re-identification method based on hierarchical matching, as shown in fig. 1, the method of the invention comprises the following steps:
acquiring a picture of a vehicle to be detected from a video source;
extracting overall characteristics, local component information, body color information and vehicle type information of the vehicle from the picture of the vehicle to be detected;
carrying out first processing on the overall vehicle characteristics to obtain overall vehicle similarity;
performing second processing on the vehicle local component information to obtain a vehicle local characteristic distance value;
performing third processing on the vehicle body color information and the vehicle type information to obtain a color feature vector and a vehicle type feature vector;
performing feature fusion on the overall vehicle similarity, the local vehicle feature distance value, the color feature vector and the vehicle type feature vector to obtain vehicle similarity;
and determining a target vehicle corresponding to the vehicle to be detected from all vehicles in the candidate vehicle set according to the vehicle similarity.
Optionally, the acquiring a picture of a vehicle to be detected from a video source includes:
acquiring image frames from the video source according to a preset time interval;
carrying out target detection on the extracted image frames to find out vehicle targets in the image frames;
and intercepting the image frames with the vehicle targets to obtain a vehicle set, and dividing the vehicle set into a search image set and a candidate image set.
Optionally, the step of obtaining the picture of the vehicle to be detected from the video source further includes:
calculating the intersection ratio between the detection frame of any current image frame and the detection frame of the corresponding previous image frame;
when the intersection ratio is larger than a preset threshold value, removing a detection result corresponding to the current image frame;
wherein, the calculation formula of the intersection ratio is as follows:
wherein IoU represents the cross-over ratio; r1A detection frame for a previous image frame; r2A detection frame for a current image frame; area (R)1)∩area(R2) Solving the area intersection between the two detection frames; area (R)1)∪area(R2) And (5) solving the union set of the areas of the two detection frames.
Optionally, the extracting, from the picture of the vehicle to be detected, the overall characteristic of the vehicle, the local component information of the vehicle, the color information of the vehicle body, and the vehicle type information includes:
dividing the vehicle into vehicle overall characteristics and vehicle local component information through a ResNet50 characteristic extractor;
processing the picture of the vehicle to be detected through global average pooling to obtain the overall characteristics of the vehicle;
outputting the overall vehicle characteristics through a CNN network;
comparing the characteristics of each visual angle of the vehicle to obtain visual angle visibility scores of each visual angle of the vehicle;
after the weight of the visual angle visibility score is calculated, calculating by combining an Euclidean distance calculation method to obtain a local characteristic distance value, and determining vehicle local component information;
and performing feature extraction through a Resnet50 network to obtain color feature vectors and vehicle type feature vectors of all vehicles, and determining the vehicle body color information and the vehicle type information.
Optionally, the extracting, from the picture of the vehicle to be detected, the overall characteristic of the vehicle, the local component information of the vehicle, the color information of the vehicle body, and the vehicle type information further includes:
dividing the vehicles in the picture of the vehicle to be detected into four directions, namely a front side, a top side, a back side and a side;
and calculating area values of the four directions of the vehicle as visual angle visibility scores, and using the visual angle visibility scores as confidence values of local features corresponding to the vehicle, wherein the confidence values are the confidence values of the four local features of the vehicle and are used for representing the weight of the feature distance of each part between the vehicles in the subsequent calculation in the local feature distance.
Optionally, the calculation formula for calculating the weight of the visibility score of the viewing angle is as follows:
wherein,
a weight representing the perspective visibility score;
a visibility score representing a vehicle to be detected;
a visibility score representing a corresponding vehicle in the image library; m, n represents the vehicle to be detected and the pair in the image libraryA corresponding vehicle; i represents a region corresponding to the visibility score and the weight; n represents the number of vehicles in the image library to be searched;
the calculation formula of the local characteristic distance value is as follows:
wherein D ism,nRepresenting the local feature distance value; d represents the Euclidean distance of each direction characteristic of the vehicle; f. ofi mCharacteristic information representing a vehicle to be detected; f. ofi nCharacteristic information representing the corresponding vehicle in the image library.
Optionally, the performing feature fusion on the overall vehicle similarity, the local vehicle feature distance value, the color feature vector, and the vehicle type feature vector to obtain the vehicle similarity includes:
performing L2 norm normalization processing on the vehicle overall similarity, the vehicle local characteristic distance value, the color characteristic vector and the vehicle type characteristic vector to obtain normalized characteristics;
calculating an initial re-identification distance value between the vehicle to be detected and the corresponding vehicle in the image library according to the normalized features;
carrying out weighted summation calculation on the initial re-identification distance value of the overall similarity of the vehicle, the initial re-identification distance value of the local characteristic distance value of the vehicle, the initial re-identification distance value of the color characteristic vector and the initial re-identification distance value of the vehicle type characteristic vector to obtain a total weight identification distance value;
and determining the similarity of the vehicle according to the total weight identification distance value.
In another aspect, an embodiment of the present invention further provides an unsupervised vehicle re-identification apparatus based on hierarchical matching, including:
the first module is used for acquiring a picture of the vehicle to be detected from a video source;
the second module is used for extracting the overall characteristics, the local component information, the color information and the model information of the vehicle from the picture of the vehicle to be detected;
the third module is used for carrying out first processing on the overall characteristics of the vehicle to obtain the overall similarity of the vehicle;
the fourth module is used for carrying out second processing on the vehicle local component information to obtain a vehicle local characteristic distance value;
the fifth module is used for performing third processing on the vehicle body color information and the vehicle type information to obtain a color feature vector and a vehicle type feature vector;
a sixth module, configured to perform feature fusion on the vehicle overall similarity, the vehicle local feature distance value, the color feature vector, and the vehicle type feature vector to obtain a vehicle similarity;
and the seventh module is used for determining a target vehicle corresponding to the vehicle to be detected from all vehicles in the candidate vehicle set according to the vehicle similarity.
Another aspect of the embodiments of the present invention further provides an electronic device, including a processor and a memory;
the memory is used for storing programs;
the processor executes the program to implement the method as described above.
Yet another aspect of the embodiments of the present invention provides a computer-readable storage medium, which stores a program, which is executed by a processor to implement the method as described above.
The embodiment of the invention also discloses a computer program product or a computer program, which comprises computer instructions, and the computer instructions are stored in a computer readable storage medium. The computer instructions may be read by a processor of a computer device from a computer-readable storage medium, and the computer instructions executed by the processor cause the computer device to perform the foregoing method.
The following detailed description of the specific implementation principles of the present invention is made with reference to the accompanying drawings:
aiming at the problems in the prior art, the invention provides a vehicle re-identification method based on multi-model fusion, which comprises a complete process of detecting, extracting and comparing video vehicles. The method comprises the steps of firstly, detecting and extracting vehicles from a video through a YOLO detection algorithm and storing the vehicles, comparing the license plates preferentially when the license plates of the vehicles are clear and can be identified, and carrying out subsequent re-identification on the remaining vehicle pictures with low license plate definition. The heavy identification part is used for comparing the vehicle into four parts, namely vehicle overall characteristics, vehicle local components, vehicle body color and vehicle type. After the overall characteristics of the vehicle are input into a network, the overall similarity is output by using the CNN; the vehicle local component characteristics are compared with the vehicle characteristics under different visual angles by introducing visual angle visibility scores, the vehicle is divided into four directions of a front side, a back side, a side surface and a top by the visual angle visibility scores, the areas of the vehicle in the four directions are counted and weighted as weights to obtain the vehicle local characteristic distance values of the subsequent Euclidean distance values of all components. And classifying the color and the type of the vehicle through a ResNet50 network to obtain a color characteristic vector and a vehicle type characteristic vector S of the vehicle. And finally, fusing the four vehicle characteristics to obtain the final vehicle similarity. The method aims to expand the inter-class difference of data and reduce the intra-class difference through the fusion of the global features and the features after vehicle analysis, so that the effect of improving the accuracy of vehicle weight identification is achieved. And finally, sequencing the similarity of the target vehicle and all vehicles in the candidate vehicle set, and screening the front 5 (or 10) vehicles with the highest similarity as output by utilizing the camera space and the vehicle occurrence time information.
The invention realizes a vehicle re-identification method based on multi-feature fusion, which comprises the steps of detecting and extracting video vehicles, classifying the colors and the types of the vehicles, comparing the component features of the vehicles at different visual angles, comparing the occurrence positions of the vehicles, sequencing the similarity values of the whole vehicles by using weighted feature values, and selecting the vehicle with the highest similarity as an output vehicle.
The vehicle detection and license plate recognition part:
the vehicle detection part firstly takes frames of a given video at certain intervals, and judges whether vehicles (including three types of car, bus and truck) appear in a scene or not by carrying out target detection on frame images, wherein the detection result is related to the set frame taking intervals, the video frame rate and the speed of a moving object. The target detection adopts a YOLO v4 algorithm, the target detection is converted into a regression problem, classification and frame regression are carried out simultaneously in a model, an input image is subjected to extraction of effective features through a trunk network (Backbone) CSPDarknet-53, a Neck (Neck) pyramid pooling (SPP), a Path Aggregation Network (PAN) and a Head (Head) YOLO v3, and then the effective features are mapped to a tensor, so that a target detection result is obtained. The method inputs a video frame image and then detects whether a vehicle target exists in the image.
For the repeated detection problem of a static vehicle or a low-speed vehicle possibly existing in an input video, the Intersection over Unit (IoU) of the current detection frame and the last detection frame is calculated, and the detection result of which the number of IoU is larger than 0.5 is removed, so that the repeated detection problem is improved.
Wherein R is1Detection frame for last video frame, R2A detection box for a current video frame; area (R)1)∩area(R2) Find the area intersection of the two detection boxes, area (R)1)∪area(R2) And (5) solving the union set of the areas of the two detection frames.
After the vehicle detection is finished, a vehicle detection part in the frame image is intercepted and stored into a vehicle set, and the vehicle set and a candidate picture set can be divided into a search picture set (query set) and a candidate picture set (galery set) according to user requirements and used for a vehicle weight identification part.
The vehicle set obtained by vehicle detection is directly used in a vehicle weight recognition part and is also used for license plate recognition to serve as auxiliary reference information for vehicle weight recognition. For the License Plate Recognition part, a License Plate Recognition network (LPRNet) is used for inputting a vehicle picture, and the License Plate part is recognized through a CNN model built by a SqueezeNet Fire module and an acceptance module. And then, storing the license plate recognition result into a csv result file for license plate information comparison in the re-recognition process.
In the vehicle weight identification comparison part: the vehicle is first split into three main part predictions. The vehicle component feature comparison part divides the vehicle into four directions of a front side, a top side, a back side and a side by using a view angle classification network, counts the area values of the four directions of the vehicle as view angle visibility scores (vis _ scores), and regards the scores as confidence values of the corresponding local features of the vehicle.
The vehicle is first divided into a global feature and a vehicle component (local) feature by a ResNet50 feature extractor. The global features of the vehicle are obtained through the global average pooling for the feature map, the global features are output through the CNN network, and when the vehicles with the same visual angle are matched, the global features can bring higher detection accuracy.
For the feature comparison of each view angle of the vehicle, after the view angle visibility scores of each view angle of the vehicle are obtained, the vehicle feature information of different view angles can be decoupled into corresponding local features on the basis, and the feature comparison under each view angle is realized. Since the visibility score is the area of the region in this case, it is necessary to further weight:
fi mand fi nFor the two vehicles corresponding to the features of the comparison,
the weights obtained by the above calculation.
The two parts are used for solving the similarity of the overall characteristics and the local characteristics of the vehicle, and then the re-identification accuracy is further improved through the classification of the vehicle type and the color characteristics of the vehicle. The vehicle type and the color are subjected to feature extraction through a Resnet50 network to obtain color feature vectors and vehicle type feature vectors S of all vehicles.
At this time, four vehicle feature values, that is, a vehicle global feature vector, a vehicle local feature vector, a vehicle type feature vector, and a vehicle body color feature vector, are already obtained in this embodiment, and after each feature vector is normalized by a norm of L2, the distance between the query vehicle and the galery vehicle is calculated, where the calculation methods of the local distance and the global distance are as shown in the foregoing formulas (1) and (2), the color feature distance and the vehicle type feature distance are obtained by calculating the euclidean distance between the features, and finally, the total vehicle weight identification distance value D is obtained by weighting the euclidean distances through linear combination.
Wherein, F'iRepresenting normalized features, FiFeature vectors output for the network.
Wherein, Fm,FnRepresenting the feature vectors of different vehicles, and N represents the dimension of the feature vectors.
D＝mDglobal+βDlocal+γDcolor+λDtype#(5)
In summary, the present invention first selects a vehicle in a photo or video stream by using a vehicle recognition technology, and then predicts the selected vehicle photo in three parts. Scoring four visual angles of the vehicle by utilizing a visual angle classification network in the vehicle component characteristics to serve as confidence values of local characteristics corresponding to the vehicle; dividing the vehicle into a global feature and a local feature through a feature extractor, and obtaining the global feature of the vehicle through global average pooling for a feature map; and finally, identifying through the vehicle type and the color characteristics. And finally, calculating characteristic distances of the four characteristics, carrying out linear combination weighting to obtain a final total distance value, and outputting the result with the highest score as a result of vehicle re-identification, so that a complete re-identification process from video to vehicle comparison is realized, and the method has high practicability and feasibility and high final detection accuracy.
In alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flow charts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.
Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and/or features may be integrated in a single physical device and/or software module, or one or more functions and/or features may be implemented in a separate physical device or software module. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary for an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be understood within the ordinary skill of an engineer, given the nature, function, and internal relationship of the modules. Accordingly, those skilled in the art can, using ordinary skill, practice the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative of and not intended to limit the scope of the invention, which is defined by the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. The unsupervised vehicle re-identification method based on the hierarchical matching is characterized by comprising the following steps:
acquiring a picture of a vehicle to be detected from a video source;
extracting overall characteristics, local component information, body color information and vehicle type information of the vehicle from the picture of the vehicle to be detected;
carrying out first processing on the overall vehicle characteristics to obtain overall vehicle similarity;
performing second processing on the vehicle local component information to obtain a vehicle local characteristic distance value;
performing third processing on the vehicle body color information and the vehicle type information to obtain a color feature vector and a vehicle type feature vector;
performing feature fusion on the overall vehicle similarity, the local vehicle feature distance value, the color feature vector and the vehicle type feature vector to obtain vehicle similarity;
and determining a target vehicle corresponding to the vehicle to be detected from all vehicles in the candidate vehicle set according to the vehicle similarity.
2. The unsupervised vehicle re-identification method based on hierarchical matching according to claim 1, wherein the obtaining of the to-be-detected vehicle picture from the video source comprises:
acquiring image frames from the video source according to a preset time interval;
carrying out target detection on the extracted image frames to find out vehicle targets in the image frames;
and intercepting the image frames with the vehicle targets to obtain a vehicle set, and dividing the vehicle set into a search image set and a candidate image set.
3. The unsupervised vehicle re-identification method based on hierarchical matching according to claim 2, wherein the step of obtaining the picture of the vehicle to be detected from the video source further comprises:
calculating the intersection ratio between the detection frame of any current image frame and the detection frame of the corresponding previous image frame;
when the intersection ratio is larger than a preset threshold value, removing a detection result corresponding to the current image frame;
wherein, the calculation formula of the intersection ratio is as follows:
wherein IoU represents the cross-over ratio; r1A detection frame for a previous image frame; r2A detection frame for a current image frame; area (R)1)∩area(R2) Solving the area intersection between the two detection frames; area (R)1)∪area(R2) And (5) solving the union set of the areas of the two detection frames.
4. The unsupervised vehicle re-identification method based on hierarchical matching according to claim 1, wherein the extracting of the overall vehicle feature, the local vehicle component information, the body color information and the vehicle type information from the picture of the vehicle to be detected comprises:
dividing the vehicle into vehicle overall characteristics and vehicle local component information through a ResNet50 characteristic extractor;
processing the picture of the vehicle to be detected through global average pooling to obtain the overall characteristics of the vehicle;
outputting the overall vehicle characteristics through a CNN network;
comparing the characteristics of each visual angle of the vehicle to obtain visual angle visibility scores of each visual angle of the vehicle;
after the weight of the visual angle visibility score is calculated, calculating by combining an Euclidean distance calculation method to obtain a local characteristic distance value, and determining vehicle local component information;
and performing feature extraction through a Resnet50 network to obtain color feature vectors and vehicle type feature vectors of all vehicles, and determining the vehicle body color information and the vehicle type information.
5. The unsupervised vehicle re-identification method based on hierarchical matching according to claim 4, wherein the extracting of the overall vehicle feature, the local vehicle component information, the body color information and the vehicle type information from the picture of the vehicle to be detected further comprises:
dividing the vehicles in the picture of the vehicle to be detected into four directions, namely a front side, a top side, a back side and a side;
and calculating area values of the four directions of the vehicle as visual angle visibility scores, and using the visual angle visibility scores as confidence values of local features corresponding to the vehicle, wherein the confidence values are the confidence values of the four local features of the vehicle and are used for representing the weight of the feature distance of each part between the vehicles in the subsequent calculation in the local feature distance.
6. The unsupervised vehicle re-identification method based on hierarchical matching according to claim 4,
the calculation formula for calculating the weight of the visual angle visibility score is as follows:
wherein,
a weight representing the perspective visibility score;
a visibility score representing a vehicle to be detected;
a visibility score representing a corresponding vehicle in the image library; m and n represent the vehicle to be detected and the corresponding vehicle in the image library; i represents a region corresponding to the visibility score and the weight; n represents the number of vehicles in the image library to be searched;
the calculation formula of the local characteristic distance value is as follows:
wherein D ism,nRepresenting the local feature distance value; d represents the Euclidean distance of each direction characteristic of the vehicle; f. ofi mCharacteristic information representing a vehicle to be detected; f. ofi nCharacteristic information representing the corresponding vehicle in the image library.
7. The unsupervised vehicle re-identification method based on hierarchical matching according to claim 1, wherein the performing feature fusion on the overall vehicle similarity, the local vehicle feature distance value, the color feature vector and the vehicle type feature vector to obtain vehicle similarity comprises:
performing L2 norm normalization processing on the vehicle overall similarity, the vehicle local characteristic distance value, the color characteristic vector and the vehicle type characteristic vector to obtain normalized characteristics;
calculating an initial re-identification distance value between the vehicle to be detected and the corresponding vehicle in the image library according to the normalized features;
carrying out weighted summation calculation on the initial re-identification distance value of the overall similarity of the vehicle, the initial re-identification distance value of the local characteristic distance value of the vehicle, the initial re-identification distance value of the color characteristic vector and the initial re-identification distance value of the vehicle type characteristic vector to obtain a total weight identification distance value;
and determining the similarity of the vehicle according to the total weight identification distance value.
8. The unsupervised vehicle re-identification device based on the hierarchical matching is characterized by comprising the following components:
the first module is used for acquiring a picture of the vehicle to be detected from a video source;
the second module is used for extracting the overall characteristics, the local component information, the color information and the model information of the vehicle from the picture of the vehicle to be detected;
the third module is used for carrying out first processing on the overall characteristics of the vehicle to obtain the overall similarity of the vehicle;
the fourth module is used for carrying out second processing on the vehicle local component information to obtain a vehicle local characteristic distance value;
the fifth module is used for performing third processing on the vehicle body color information and the vehicle type information to obtain a color feature vector and a vehicle type feature vector;
a sixth module, configured to perform feature fusion on the vehicle overall similarity, the vehicle local feature distance value, the color feature vector, and the vehicle type feature vector to obtain a vehicle similarity;
and the seventh module is used for determining a target vehicle corresponding to the vehicle to be detected from all vehicles in the candidate vehicle set according to the vehicle similarity.
9. An electronic device comprising a processor and a memory;
the memory is used for storing programs;
the processor executing the program realizes the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the storage medium stores a program, which is executed by a processor to implement the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111550495.1A CN114387571A (en) | 2021-12-17 | 2021-12-17 | Unsupervised vehicle re-identification method and unsupervised vehicle re-identification device based on hierarchical matching |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111550495.1A CN114387571A (en) | 2021-12-17 | 2021-12-17 | Unsupervised vehicle re-identification method and unsupervised vehicle re-identification device based on hierarchical matching |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114387571A true CN114387571A (en) | 2022-04-22 |
Family
ID=81198108
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111550495.1A Pending CN114387571A (en) | 2021-12-17 | 2021-12-17 | Unsupervised vehicle re-identification method and unsupervised vehicle re-identification device based on hierarchical matching |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114387571A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115762172A (en) * | 2022-11-02 | 2023-03-07 | 济南博观智能科技有限公司 | Method, device, equipment and medium for identifying vehicles entering and exiting parking places |
-
2021
- 2021-12-17 CN CN202111550495.1A patent/CN114387571A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115762172A (en) * | 2022-11-02 | 2023-03-07 | 济南博观智能科技有限公司 | Method, device, equipment and medium for identifying vehicles entering and exiting parking places |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8509478B2 (en) | Detection of objects in digital images | |
| Marszalek et al. | Accurate object localization with shape masks | |
| Wu et al. | A practical system for road marking detection and recognition | |
| CN111914664A (en) | Vehicle multi-target detection and track tracking method based on re-identification | |
| Zhang et al. | A PCA-based vehicle classification framework | |
| CN107273832B (en) | License plate recognition method and system based on integral channel characteristics and convolutional neural network | |
| Derpanis et al. | Classification of traffic video based on a spatiotemporal orientation analysis | |
| Van Pham et al. | Front-view car detection and counting with occlusion in dense traffic flow | |
| Saran et al. | Traffic video surveillance: Vehicle detection and classification | |
| Asgarian Dehkordi et al. | Vehicle type recognition based on dimension estimation and bag of word classification | |
| Yabo et al. | Vehicle classification and speed estimation using computer vision techniques | |
| Suryakala et al. | Vision based vehicle/pedestrian detection in traffic surveillance system | |
| Huynh et al. | Convolutional neural network for motorbike detection in dense traffic | |
| JP5759124B2 (en) | Computerized method and system for analyzing objects in images obtained from a camera system | |
| CN114387571A (en) | Unsupervised vehicle re-identification method and unsupervised vehicle re-identification device based on hierarchical matching | |
| Jain et al. | Number plate detection using drone surveillance | |
| Hasan et al. | Comparative analysis of vehicle detection in urban traffic environment using Haar cascaded classifiers and blob statistics | |
| Sotheeswaran et al. | A coarse-to-fine strategy for vehicle logo recognition from frontal-view car images | |
| Romdhane et al. | Combined 2d/3d traffic signs recognition and distance estimation | |
| Prates et al. | An adaptive vehicle license plate detection at higher matching degree | |
| Das et al. | Bag of feature approach for vehicle classification in heterogeneous traffic | |
| Prahara et al. | Vehicle pose estimation for vehicle detection and tracking based on road direction | |
| Azi et al. | Car tracking technique for DLES project | |
| Deotale et al. | Analysis of Human Activity Recognition Algorithms Using Trimmed Video Datasets | |
| Lin et al. | The studies on Automated Road Safety Monitoring System and Real-Time Traffic Reporting System of an Automatic Vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |