CN116246139A - Target identification method based on multi-sensor fusion for unmanned ship navigation environment - Google Patents
Target identification method based on multi-sensor fusion for unmanned ship navigation environment Download PDFInfo
- Publication number
- CN116246139A CN116246139A CN202310167630.7A CN202310167630A CN116246139A CN 116246139 A CN116246139 A CN 116246139A CN 202310167630 A CN202310167630 A CN 202310167630A CN 116246139 A CN116246139 A CN 116246139A
- Authority
- CN
- China
- Prior art keywords
- target
- recognition
- image
- model
- unmanned ship
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 230000004927 fusion Effects 0.000 title claims abstract description 48
- 230000003287 optical effect Effects 0.000 claims abstract description 23
- 238000012549 training Methods 0.000 claims description 28
- 238000001514 detection method Methods 0.000 claims description 21
- 238000012545 processing Methods 0.000 claims description 17
- 238000005286 illumination Methods 0.000 claims description 16
- 230000004044 response Effects 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 11
- 238000000605 extraction Methods 0.000 claims description 9
- 238000012937 correction Methods 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 5
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 5
- 238000000354 decomposition reaction Methods 0.000 claims description 4
- 238000003672 processing method Methods 0.000 claims description 4
- 230000001629 suppression Effects 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 3
- 238000004088 simulation Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 18
- 238000007781 pre-processing Methods 0.000 abstract description 3
- 238000003384 imaging method Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000008447 perception Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 3
- 239000002245 particle Substances 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 239000013535 sea water Substances 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 239000000443 aerosol Substances 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000002238 attenuated effect Effects 0.000 description 1
- 239000011324 bead Substances 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004941 influx Effects 0.000 description 1
- 238000011068 loading method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000004083 survival effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Life Sciences & Earth Sciences (AREA)
- Electromagnetism (AREA)
- Computer Networks & Wireless Communication (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a target recognition method based on multi-sensor fusion for unmanned ship navigation environment, which aims at target recognition under the interference of huge marine environment information at sea, provides fusion of an optical image and an optical radar, combines a first recognition model with a second recognition model, uses a time-space fusion algorithm for preprocessing a shooting image, breaks through the difficult problems of sensing the unmanned ship navigation environment and recognizing the target under complex sea conditions, and adopts the combination of the first model and the second model to ensure that the model has better effect of extracting the recognition target from the marine environment. And the recognition target extracted from the image by the first recognition model and the natural factor recognition result are input into the second recognition model, so that the second recognition model is focused on the recognition classification of the target object to be recognized, the target recognition effect is improved, the interference of various factors on the sea on the target recognition can be considered, and the influence of complex sea conditions on the target recognition is reduced.
Description
Technical Field
The invention belongs to the technical field of unmanned boats, and particularly relates to a target identification method based on multi-sensor fusion for an unmanned boat navigation environment.
Background
The unmanned ship can replace or assist some ships to perform tasks in some high-risk areas, so that risks are reduced to the greatest extent, cost is reduced, safety is guaranteed, and more importantly, future combat patterns are greatly influenced or even thoroughly overturned. According to different carried task modules, the unmanned ship can cooperate with the manned ship or independently complete various military tasks. Unmanned boats are becoming a medium-hard force for offshore equipment, and unmanned systems and key technologies related to the field are also becoming open beads in offshore intelligent manufacturing, which have a decisive influence on the development of the whole unmanned boat field.
Autonomous navigation is well known to be a requisite feature capability and core value of unmanned boats, and its prerequisites include achieving rapid perception of surrounding environmental targets. However, due to complex and even severe offshore environments, the change is always apparent, and the mutual influx of a great amount of environmental information, the mutual influence of various environmental targets and the mutual alternation of changeable task scenes greatly influence and reduce, even severely restrict the autonomous control capability of the unmanned ship.
The existing identification technology needs to keep the unmanned ship platform as stable as possible, and particularly, under the conditions that a huge amount of marine environment information elements are gathered, the marine environment target characteristics are similar or are difficult to distinguish in a long distance, a relatively stable loading platform is needed to be used as a support, but the marine environment is certainly not an ideal condition. The wind wave has obvious shape under the 4-level sea condition, the 5-level sea condition has high and big wave crest, and the wind starts to cut off the wave crest, which has great influence on the autonomous navigation of the small and medium unmanned ships; even in the 3-level sea condition, although the stormy waves are not large, the stormy waves are very touching and the wave peaks are broken, so that the stability of the base of the target identification equipment is difficult to ensure, and if the influence factors such as heavy fog, low illumination and the like interact and are mutually overlapped, the perception, decision making and autonomous navigation control capability of the unmanned ship are extremely restricted.
In order to realize target identification under typical sea conditions, unmanned ships need to autonomously process the obtained video images and can rapidly and accurately identify target objects in the video images. Determining the target object by recognition is the primary task of the unmanned ship vision system.
However, more target recognition currently used for unmanned vessels is to transplant or migrate recognition technology from land to unmanned vessels at sea, and many problems of offshore scenes and vessels themselves are not taken into consideration, such as huge amounts of marine environment information collection and interference, interference and influence of marine natural environments (such as sea waves, sea fog, light reflection, attack, etc.), influence of typical sea conditions on movements and target movement states of unmanned vessels themselves, mutual influence of natural objects and artificial targets on the sea surface and in the sea water, and the like.
Disclosure of Invention
The invention aims to solve the problems and provides a target identification method for an unmanned ship navigation environment based on multi-sensor fusion.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
an unmanned ship navigation environment target recognition method based on multi-sensor fusion, the method comprises the following steps:
s1, receiving an image shot by an unmanned ship optical camera and a radar signal captured by an optical radar;
s2, performing image processing on the received image through a space-time fusion algorithm, and then inputting the image into a first recognition model by combining radar signals;
s3, classifying marine environment information of the input image by the first recognition model to extract recognition targets and recognize natural factors;
s4, inputting the output of the first recognition model into a second recognition model;
s5, classifying each recognition target by the second recognition model to obtain a target recognition result;
s6, judging the motion state of each recognition target by combining the current state of the unmanned ship through the optical radar signals of the previous image and the optical radar signals of the subsequent image;
s7, outputting a target identification result with a classification mark and a motion state according to the identification and judgment result, and completing the multi-target identification process.
In the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment, in step S3, the marine environment information includes: any combination of waves, tides, ocean currents, sea fog, reefs, islands, dawn, dusk, noon, vessels, fishing vessels, lighthouses, buoys, sunken vessels;
the target object to be identified comprises any one or a combination of a plurality of reefs, islands, ships, fishing vessels, lighthouses, buoys and sunken ships.
In the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment, in step S5, the recognition targets are judged to belong to fixed targets, moving targets or relatively fixed targets according to the types of the recognition targets;
in step S6, the category recognition results of the recognition targets are checked according to the motion state judgment results of the recognition targets, if all the recognition results of the recognition targets are checked to be correct, the process proceeds to step S7, otherwise, the recognition targets with incorrect checks are marked and output, and the rest recognition targets are output according to step S7.
For example, islands and lighthouses are fixed targets, buoys are relatively fixed targets within a certain range, and fishing vessels and ships are moving targets. The staff carries out manual identification, and the manual identification result can be used as a label of an identification target to be input into a second identification model for reclassifying so as to continuously improve the accurate classification capability of the model.
In the above-mentioned target recognition method based on multi-sensor fusion for unmanned ship navigation environment, in step S7, the country to which the ship belongs is further recognized by using the ship-belonging recognition model for the recognition target recognized as the ship.
In the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment, when sea fog or other marine environment information causing low-illumination problems are recognized in the marine environment information, defogging and low-illumination correction processing is performed on the input image.
In the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment, the defogging processing method comprises the following steps:
s111, extracting the first 0.01% of pixels from the input image as candidate areas;
s112, reading the number of connected components of the candidate region, and selecting the connected component with the largest number of pixels of the connected component as the candidate region for atmospheric light value estimation;
s113, selecting the maximum brightness value in the candidate area as the total atmosphere light value A of the area.
The number of pixels in the extracted dark channel is 0.01%, and the method can have a good atmospheric light value estimation effect in a sea surface scene with thicker fog.
In the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment, the low-illumination correction processing method comprises the following steps:
s121, improving the brightness of an image from a multi-scale Ret index to obtain a reflection component;
s122, carrying out detail enhancement on reflection components of the image by adopting guide filtering and high-frequency lifting;
s123, removing sparse noise by using a global low-rank decomposition algorithm, so that noise in a low-illumination image can be effectively eliminated, and noise generated in a high-frequency lifting process can be eliminated.
In the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment, in step S2, the process of performing image processing on the received image by the space-time fusion algorithm is as follows:
A. performing image detection through an SSD detection algorithm to obtain required spatial information;
B. tracking a water surface target by using a KCF tracking algorithm to obtain required time information;
C. and fusing the space information and the time information to obtain a final processed image.
In the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment, the extraction method of the space information in the A specifically comprises the following steps:
A1. the input image is subjected to iterative convolution by a basic network to fully extract characteristics;
A2. further extracting and analyzing the characteristics of the image through an additional network, and predicting and classifying the target position offset by utilizing convolution prediction;
A3. removing redundancy by using a non-maximum suppression method to obtain the required spatial information;
in B, the time information extraction method is as follows:
B1. initializing a current frame image, extracting features of the current frame image, and then performing model training to obtain a filter template;
B2. performing target tracking, extracting features of the current frame, convoluting the current frame with a filter template obtained by training of the previous frame to obtain a response chart, and obtaining the most relevant target position;
B3. taking the target position as the center, selecting rectangular areas with different scales as samples, respectively extracting the respective direction gradient histogram characteristics of the rectangular areas, obtaining corresponding sample responses through a tracking classifier, obtaining the strongest response after comparison, and taking the rectangular area corresponding to the sample with the strongest response as the target scale;
in the step C, the method for fusing the space information and the time information is as follows:
C1. determining target space position information in the first frame image through a detection algorithm;
C2. taking the first frame target space position as the input of a tracking algorithm, and adopting the tracking algorithm to realize target tracking on a plurality of frames of pictures;
C3. after tracking the set frame number, running a re-detection mechanism to re-determine the target space position information;
C4. taking the redetermined target space position information as the input of a tracking algorithm, and realizing target tracking by adopting the tracking algorithm for the subsequent frames of pictures;
repeating steps C3 and C4.
In the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment, training data for training the first recognition model and the second recognition model are real atlas and virtual atlas mixed training;
the real atlas is obtained by shooting the sea surface on site, and the virtual atlas is obtained by sea condition simulation of a virtual twin environment;
the real atlas is input into a virtual twin environment, the two-class model and the multi-class model for training are integrated into the twin environment, the model is trained in the twin environment, and the trained model parameters are output. The twinning environment itself is simulated based on the physical environment and complies with physical rules, and the twinning hull data is also based on real entities.
The invention has the advantages that:
1. aiming at target recognition under the interference of a huge amount of marine environment information on the sea, an optical image and an optical radar are fused, a first recognition model and a second recognition model are combined, and the modes of time-space fusion algorithm preprocessing and the like are used for shooting the image, so that the problems of complex sea-state unmanned ship navigation environment perception and target recognition are broken through;
2. the mode of combining the first model and the second model is adopted, so that the model is guaranteed to have a good effect of extracting the identification target from the marine environment. And the recognition target extracted from the image by the first recognition model and the natural factor recognition result are input into the second recognition model, so that the second recognition model is focused on the recognition classification of the target object to be recognized, the target recognition effect is improved, the interference of various factors on the sea on the target recognition can be considered, and the influence of complex sea conditions on the target recognition is reduced.
3. The optical image and the optical radar signal are combined and input into a first model, and the model can be combined with the characteristics of the optical radar signal to carry out target extraction and natural factor recognition on the optical image, so that the extraction and recognition capability is effectively improved;
4. aiming at the problem of difficult acquisition of unmanned ship training data, a virtual twin environment is used for generating a virtual atlas, and then a model is trained in a mode of mixing a real atlas and the virtual atlas, so that the performance of the model can be improved;
5. aiming at the particularity of sea fog, an improved atmospheric light value optimization method is provided, and the removal effect of the sea fog is improved, so that a clearer video image is provided for target identification, and the target identification effect is improved;
6. for the low illumination problem, the reflection model Ret index is further improved, so that the problems of color distortion and detail loss can be effectively improved, and the image quality of a shot image is further improved;
7. the virtual twin environment for generating the virtual atlas is utilized, the real atlas and the model are both input into the twin environment, virtual training is carried out in the twin environment, and as the twin environment and the twin environment simulate based on the physical environment and follow physical rules, the twin hull data are also based on the real entity, so that the effect similar to the training of the real environment can be obtained, the testing efficiency can be effectively improved, and the cost of experiments and training can be reduced.
Drawings
FIG. 1 is a method flow diagram of a method for target identification based on multi-sensor fusion for an unmanned ship navigation environment of the present invention;
FIG. 2 is a flow chart of two recognition models in the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment;
FIG. 3 is an atmospheric scattering physical model diagram of a fog mask in the target recognition method based on multi-sensor fusion in the unmanned ship navigation environment of the invention;
FIG. 4 is a flow chart of an improved atmospheric light value estimation in the target recognition method based on multi-sensor fusion for an unmanned ship navigation environment of the present invention;
FIG. 5 is a framework diagram of a spatial and temporal information fusion algorithm in a target recognition method based on multi-sensor fusion in an unmanned ship navigation environment;
FIG. 6 is a block diagram of an implementation of a hybrid training model using a virtual twin environment in a multi-sensor fusion-based target recognition method for an unmanned ship navigation environment of the present invention;
fig. 7 is a flowchart of a method for identifying a target in an unmanned ship navigation environment based on multi-sensor fusion according to a second embodiment of the present invention.
Detailed Description
The invention will be described in further detail with reference to the drawings and the detailed description. As shown in fig. 1, the present solution proposes a target recognition method based on multi-sensor fusion for an unmanned ship navigation environment, where the method includes:
s1, receiving an image shot by an unmanned ship optical camera and a radar signal captured by an optical radar;
s2, removing signal noise captured by the optical radar through a signal noise reduction technology;
carrying out image processing on the received image through a space-time fusion algorithm;
subsequently inputting the image into the first recognition model in combination with the radar signal; the image received by the system is a video image and the radar signal is input to the first recognition model corresponding to each frame or successive frames of the video.
S3, classifying marine environment information of the input image by the first recognition model to extract recognition targets and recognize natural factors; the marine environment information includes: any combination of ocean waves, tides, ocean currents, sea fog, dawn, dusk, noon, light, reefs, islands, vessels, fishing vessels, lighthouses, buoys, sunken vessels, and the like;
the target object to be identified comprises any one or a combination of a plurality of reefs, islands, ships, fishing vessels, lighthouses, buoys and sunken ships.
When the first recognition model is trained, the training data gives out a label which uniformly represents the target object to be recognized as the target object to be recognized, and the further recognition of the target object is completed by a subsequent second recognition model. In addition, training data of the first recognition model are respectively marked on natural factors such as sea waves, tides, ocean currents, sea fog, dawn, dusk, noon and the like, and can be used for recognizing influences such as water fog shielding and sunlight reflection.
S4, inputting the output of the first recognition model into a second recognition model;
s5, classifying each recognition target through a second recognition model to obtain a target recognition result;
when the second recognition model is trained, natural factors such as sea waves, tides, ocean currents, sea fog, dawn, dusk, noon and the like are respectively marked by training data, targets to be recognized are respectively marked, and marking results of part of training data on natural environments can be realized by means of the trained first recognition model. As shown in fig. 2, after training, the processed video image with the radar signal is input into a first recognition model and a second recognition model, the first recognition model extracts the target object to be recognized and outputs the target object to the second recognition model, and meanwhile, the recognition result of the target object to the natural environment is output to the second recognition model, so that the second recognition model can be helped to learn to recognize the category of each target object more accurately under the scene of the offshore complex environment.
S6, judging the motion state of each recognition target by combining the current state of the unmanned ship through the optical radar signals of the previous image and the optical radar signals of the subsequent image; the unmanned ship has a speed, the motion state of the unmanned ship relative to the recognition target is calculated according to the running direction of the unmanned ship and the position of the recognition target, for example, the unmanned ship approaches the recognition target at 20km/h, then the distance change of the unmanned ship relative to the recognition target is calculated according to the optical radar signal at the previous moment and the optical radar signal at the current moment, and then the motion state of the recognition target can be obtained according to the actual distance change condition and the motion state of the unmanned ship relative to the recognition target, for example, the unmanned ship is far away from/approaches at 10 km/h.
S7, outputting the identification result with the classification mark and the motion state according to the identification and judgment result. If two targets of the fishing boat and the lighthouse are identified, and the motion state is respectively a speed approaching state and a static state of 5km/h, outputting: (1) fishing boat: the distance is 5km/h close; (2) a lighthouse.
Thereby completing multi-target identification under complex sea conditions. According to the scheme, the target to be identified is extracted from the marine environment through the first identification model, various targets are not required to be identified in the first identification model, the targets are only required to be uniformly extracted, and the trained model is guaranteed to have a good extraction effect. The method has the advantages that the method inputs the recognition target extraction result of the first recognition model on the image into the second recognition model, and inputs the natural factor recognition result into the second recognition model, so that on one hand, the second recognition model is focused on the recognition classification of the target object to be recognized, on the other hand, the interference of various factors on the sea on the target recognition can be considered, and the influence of complex sea conditions on the target recognition is reduced.
Specifically, when it is recognized that there is sea fog or other marine environment information causing a problem of low illuminance in the marine environment information, defogging and low illuminance correction processing is performed on the input image, and when there is sea fog in the marine environment, a problem of low illuminance is necessarily caused, so that the defogging and low illuminance correction processing needs to be performed simultaneously. When there is no sea fog in the marine environment, but there is a problem of low illuminance, defogging processing is not required, and only low illuminance correction processing is required. Consider the case where the dusk, dawn or sunlight illuminance is below a set value as a low illuminance problem.
For the problem of defogging, more researches at home and abroad are based on simple monitoring scenes and scenes with slow background transformation, but the defogging research quantity under the condition of high-speed movement of an unmanned ship on the sea surface is not large, the scheme uses an improved dark channel prior sea fog removal algorithm based on a physical model, combines the characteristics of good guide filtering edge protection effect and operation speed block, achieves the effects of good sea fog removal effect and high speed, and can meet the requirement on algorithm instantaneity; the improved sea fog removal algorithm can realize better and faster defogging effect on images and video information, and the specific scheme is as follows:
after the sea fog in the input image is identified, the sea fog in the image is effectively removed by using a sea fog removing algorithm based on a single image of a physical model. In haze weather, a large number of suspended micro water drops, aerosol and other particles exist in the atmosphere, and the particles in the atmosphere have the effects of absorbing light and scattering light, so that the final effect of the absorbed light on imaging is small (can be ignored), and the image imaging is blurred mainly due to the scattering effect of the light. From physical reasons, the scheme is to build a physical model of foggy weather imaging, carry out inverse solution on the physical model, and then recover clear images from degraded images. McCartney proposed a well-known atmospheric scattering physical model based on the michaelis scattering theory in 1975, from which a desired foggy imaging physical model can be designed, as shown in fig. 3. Due to the scattering effect of atmospheric particles on light in a foggy environment, a part of light reflected by an object is scattered or attenuated and cannot reach the imaging device, and the rest of unscattered reflected light reaches the imaging device to form an image of the object (part of light cannot accurately reach and thus the image quality is reduced).
The image restoration in the atmospheric physical model by using the traditional dark channel theory has the defect of insufficient estimated atmospheric light value, and is more obvious in a special scene of sea fog. Therefore, as shown in fig. 4, for the special scene of sea fog, the scheme uses an improved atmospheric light value optimization estimation method: extracting the first 0.01% pixels in the dark channel as a candidate region, then reading the number of 8 field connected components of the candidate region, selecting the connected component with the largest number of the connected component pixels as a candidate region of the atmospheric light value estimation, and finally selecting the maximum brightness value in the region as the total atmospheric light value estimation A of the region. The method is suitable for sea surface scenes with generally thicker fog, and can ensure that images containing the fog can be clearly imaged after the images are subjected to an algorithm.
For the low-illumination problem, the scheme adopts guided filtering and low-rank decomposition to improve and promote the illumination reflection model Ret index theory based on the illumination reflection model Ret index theory proposed by the American physicist Edwin.H.Land. The images shot in the low-illumination environment are often underexposed and are accompanied with uneven illumination, so that the whole image can be dark in brightness, little in image detail information and fuzzy in visual effect, and visual effect of human eyes on the image and information extraction, analysis and processing of a machine vision system on the image are greatly influenced.
The method adopted by the scheme is as follows: after the brightness of the image is improved from the multi-scale Ret index and the reflection component is obtained, carrying out detail enhancement on the reflection component of the image by adopting guide filtering and high-frequency improvement; then, the sparse noise is removed by using a global low-rank decomposition algorithm, so that the noise in the low-illumination image can be effectively eliminated, and the noise generated in the high-frequency lifting process can be eliminated. Due to the isotropy characteristic of the Gaussian function, the problems of detail loss, noise amplification and contrast reduction caused by over-enhancement can occur when the image processed by the Ret inex algorithm is subjected to illumination compensation.
After the defogging and low-illumination processing, the image quality of the shot image can be obviously improved. In the case of sea fog, the defogging and low illumination processing are performed respectively, and the final processing image is output after the results are combined.
Specifically, the space-time fusion algorithm performs object recognition after image processing on the received images, can solve the problems of object shielding and camera shake, overcomes the influence caused by the motion of the ship and the object in the offshore environment, and overcomes the mutual influence of natural objects and artificial objects on the sea surface and in the sea water.
The process of image processing on the received image based on the space-time fusion algorithm in the scheme is approximately as shown in fig. 5:
firstly, performing image detection by a single-shot multi-core detection algorithm (Single Shot mult ibox Detector, SSD), then tracking a water surface target by a nucleated correlation filter tracking algorithm (Kernel i zed Correlat ion Fi lter, KCF), detecting by an SSD mechanism after tracking a certain number of frames, and comparing the confidence coefficient of a newly built side frame with that of an old tracking frame; and obtaining a proper detection frame by combining the new information and the old information through a space-time fusion strategy, and then continuing tracking detection, and if a new target appears in the visual field, simultaneously tracking the new target.
The spatial information is extracted as follows:
firstly, fully extracting features of an input image through iterative convolution by a basic network; further extracting and analyzing the characteristics of the image through an additional network, and predicting and classifying the target position offset by utilizing convolution prediction; finally, redundancy is removed by Non-maximum suppression (Non-Maximum Suppress ion, NMS) and then the desired spatial information is obtained.
The solution employs an SSD algorithm to achieve target detection, using a small convolution filter applied to feature mapping to predict class scores and block offsets for a set of fixed default bounding boxes. During prediction, the network generates a score for the existence of each object category in each default frame, and adjusts the frame to better match the shape of the object, so that the precision and the detection speed are obviously improved, and the detection precision of the target space information is improved.
The final two full connection layers in the SSD algorithm network are changed into volume layers, and then 4 convolution layers are added to construct an integral network structure to obtain an additional network, wherein the truncated SSD algorithm is a basic network, and the added volume layers are the additional network. In addition, these convolutional feature layers added at the end of the truncated SSD base network are progressively smaller in size and allow predictive detection on multiple scales, further improving detection accuracy.
The time information is extracted as follows:
firstly, initializing a first frame image, extracting features of the first frame image, then performing model training, and updating a filter in real time by using the state of a target in each frame to obtain a filter template; and then, performing target tracking, extracting features of the current frame, and convolving with a filter template obtained by training of the previous frame to obtain a response chart, wherein the most relevant (the sample with the strongest response is taken as the position of the current frame) is the target position. In order to cope with the target scale change, a scale self-adaptive strategy is formulated so as to ensure the tracking stability: taking the target position as the center, selecting rectangular areas with different scales as samples, respectively extracting the characteristics of the respective direction gradient histograms (Hi stogram of Ori ented Gradi ent, HOG), obtaining corresponding sample responses after tracking the classifier, obtaining the strongest response after comparison, and taking the frame corresponding to the sample with the strongest response as the target scale.
After the space information and the time information are obtained by the method, the information is fused, and the advantages of the space information and the time information are utilized to improve the overall performance and give consideration to robustness and instantaneity. The fusion steps are as follows: determining target space position information in a first frame image through an SSD detection algorithm, taking the position of a first frame target as the input of a KCF algorithm, and adopting a tracking algorithm to realize target tracking on a plurality of subsequent frames of pictures; and (3) running a re-detection mechanism after tracking the fixed frame number, taking the target position as the input of a KCF algorithm, obtaining a proper detection frame by combining new and old information through a space-time fusion strategy, and continuing to track and detect, so as to ensure the accuracy of continuous tracking (the frame number for tracking between the two re-detection is determined by a person skilled in the art according to the need). After a certain number of frames are tracked by using a KCF algorithm, detecting by using an SSD mechanism to obtain new and old tracking frames, ensuring the tracking stability and obtaining the strongest sample response.
For training of the second recognition model, different feature points (such as color of life jackets for offshore personnel, sea surface floats, ship mast flags and the like, and specific feature points can be drawn up again according to requirements) can be used for training the model. For example, for target positioning of life jackets, ship flags and the like, a method based on RGB color features can be adopted to divide a target area in an image and position the target area. If a plurality of targets have the same color characteristics in the identification process, the accuracy of identifying the targets can be improved.
Unmanned boats want to accurately identify targets under high sea conditions and target tracking is difficult to achieve satisfactory results according to current research conditions. In addition, the number of actual scenes of the domestic complex sea conditions is not large, the randomness of the complex sea conditions is high, and the feasibility of reaching the site at the first time is not very high when the conditions are satisfied under the high sea conditions. The deep learning is stable in performance in the field of image recognition, and the recognition accuracy of the algorithm on the target object after training can reach quite high accuracy under the general condition. However, based on the previous explanation, the difficulty of acquiring the real pictures under the high sea condition is high, and the number of effective training pictures is small, which results in poor final effect of algorithm learning.
In this regard, as shown in fig. 6, the present solution proposes to use a virtual twin environment, obtain some real atlas by shooting the sea surface on site, input these real atlas into the virtual twin environment, perform sea condition simulation on the virtual twin environment to obtain a large number of virtual atlas, use the foregoing real atlas and virtual atlas hybrid training model in the virtual twin environment, and finally output the trained model parameters for target recognition of the unmanned ship. The twinning environment itself is simulated based on the physical environment and complies with physical rules, and the twinning hull data is also based on real entities.
Aiming at the problems of autonomous navigation environment cognition difficulty and the like caused by multiple marine environment information elements and similar sea and battlefield environment target types, the scheme breaks through the difficult problems of complex sea condition unmanned ship navigation environment perception and target identification based on technologies such as twin environment, multi-sensor fusion, multi-model matching, shooting image preprocessing and the like, greatly strengthens the agility, accuracy, stability and reliability of the unmanned ship on marine target identification under typical sea conditions, and provides strong support for improving autonomous perception capability of the unmanned ship under complex sea and battlefield environments, and further improving self-contained survival and autonomous battlefield efficiency of the unmanned ship. The unmanned ship can be controlled autonomously according to the target recognition result, or can be controlled remotely by a person according to the target recognition result.
Example two
The difference between this embodiment and the embodiment is that, as shown in fig. 7, in step S5, each recognition target is determined to belong to a fixed target, a moving target or a relatively fixed target according to its category;
in step S6, the category recognition results of the recognition targets are checked according to the motion state judgment results of the recognition targets, if all the recognition results of the recognition targets are checked to be correct, the process proceeds to step S7, otherwise, the recognition targets with incorrect checks are marked and output, and the rest recognition targets are output according to step S8.
For example, islands and lighthouses are fixed targets, buoys are relatively fixed targets within a certain range, and fishing vessels and ships are moving targets. The staff can carry out manual identification, and the manual identification result can be used as a label of an identification target to be input into a second identification model for reclassifying so as to continuously improve the accurate classification capability of the model.
Further, the identification model which belongs to the ship can be used for further identifying the country to which the ship belongs aiming at the identification target which is identified as the ship, so that the identification of the enemy warship is realized.
The specific embodiments described herein are offered by way of example only to illustrate the spirit of the invention. Those skilled in the art may make various modifications or additions to the described embodiments or substitutions thereof without departing from the spirit of the invention or exceeding the scope of the invention as defined in the accompanying claims.
Claims (10)
1. The target identification method based on multi-sensor fusion for the unmanned ship navigation environment is characterized by comprising the following steps of:
s1, receiving an image shot by an unmanned ship optical camera and a radar signal captured by an optical radar;
s2, performing image processing on the received image through a space-time fusion algorithm, and then inputting the image into a first recognition model by combining radar signals;
s3, classifying marine environment information of the input image by the first recognition model to extract recognition targets and recognize natural factors;
s4, inputting the output of the first recognition model into a second recognition model;
s5, classifying each recognition target by the second recognition model to obtain a target recognition result;
s6, judging the motion state of each recognition target by combining the current state of the unmanned ship through the optical radar signals of the previous image and the optical radar signals of the subsequent image;
s7, outputting a target identification result with a classification mark and a motion state according to the identification and judgment result.
2. The method for identifying the target of the unmanned ship navigation environment based on the multi-sensor fusion according to claim 1, wherein in the step S3, the marine environment information comprises: any combination of waves, tides, ocean currents, sea fog, reefs, islands, dawn, dusk, noon, vessels, fishing vessels, lighthouses, buoys, sunken vessels;
the target object to be identified comprises any one or a combination of a plurality of reefs, islands, ships, fishing vessels, lighthouses, buoys and sunken ships.
3. The unmanned ship navigation environment multi-sensor fusion-based target recognition method according to claim 1, wherein in step S5, each recognition target is judged to belong to a fixed target, a moving target or a relatively fixed target according to its category;
in step S6, the category recognition results of the recognition targets are checked according to the motion state judgment results of the recognition targets, if all the recognition results of the recognition targets are checked to be correct, the process proceeds to step S7, otherwise, the recognition targets with incorrect checks are marked and output, and the rest recognition targets are output according to step S7.
4. The method for identifying the target in the unmanned ship navigation environment based on the multi-sensor fusion according to claim 2 or 3, wherein in the step S7, the country to which the ship belongs is further identified by using the ship belonging identification model for the identified target identified as the ship.
5. The method for identifying the target of the unmanned ship navigation environment based on the multi-sensor fusion according to claim 2, wherein defogging and low-illuminance correction processing is performed on the input image when it is identified that there is sea fog or other marine environment information causing a low-illuminance problem in the marine environment information.
6. The target recognition method based on multi-sensor fusion for the unmanned ship navigation environment according to claim 5, wherein the defogging processing method is as follows:
s111, extracting the first 0.01% of pixels from the input image as candidate areas;
s112, reading the number of connected components of the candidate region, and selecting the connected component with the largest number of pixels of the connected component as the candidate region for atmospheric light value estimation;
s113, selecting the maximum brightness value in the candidate area as the total atmosphere light value A of the area.
7. The target recognition method based on multi-sensor fusion for the unmanned ship navigation environment according to claim 6, wherein the low-illumination correction processing method is as follows:
s121, improving the brightness of an image from multi-scale Retinex to obtain a reflection component;
s122, carrying out detail enhancement on reflection components of the image by adopting guide filtering and high-frequency lifting;
s123, removing sparse noise by using a global low-rank decomposition algorithm.
8. The method for identifying the target of the unmanned ship navigation environment based on the multi-sensor fusion according to claim 1, wherein in the step S2, the process of performing the image processing on the received image by the space-time fusion algorithm is as follows:
A. performing image detection through an SSD detection algorithm to obtain required spatial information;
B. tracking a water surface target by using a KCF tracking algorithm to obtain required time information;
C. and fusing the space information and the time information to obtain a final processed image.
9. The target recognition method based on multi-sensor fusion for the unmanned ship navigation environment according to claim 8, wherein in the step A, the extraction method of the spatial information is specifically as follows:
A1. the input image is subjected to iterative convolution by a basic network to fully extract characteristics;
A2. further extracting and analyzing the characteristics of the image through an additional network, and predicting and classifying the target position offset by utilizing convolution prediction;
A3. removing redundancy by using a non-maximum suppression method to obtain the required spatial information;
in B, the time information extraction method is as follows:
B1. initializing a current frame image, extracting features of the current frame image, and then performing model training to obtain a filter template;
B2. performing target tracking, extracting features of the current frame, convoluting the current frame with a filter template obtained by training of the previous frame to obtain a response chart, and obtaining the most relevant target position;
B3. taking the target position as the center, selecting rectangular areas with different scales as samples, respectively extracting the respective direction gradient histogram characteristics of the rectangular areas, obtaining corresponding sample responses through a tracking classifier, obtaining the strongest response after comparison, and taking the rectangular area corresponding to the sample with the strongest response as the target scale;
in the step C, the method for fusing the space information and the time information is as follows:
C1. determining target space position information in the first frame image through a detection algorithm;
C2. taking the first frame target space position as the input of a tracking algorithm, and adopting the tracking algorithm to realize target tracking on a plurality of frames of pictures;
C3. after tracking the set frame number, running a re-detection mechanism to re-determine the target space position information;
C4. taking the redetermined target space position information as the input of a tracking algorithm, and realizing target tracking by adopting the tracking algorithm for the subsequent frames of pictures;
repeating steps C3 and C4.
10. The unmanned ship navigation environment target recognition method based on multi-sensor fusion according to claim 1, wherein,
training data for training the first recognition model and the second recognition model adopts real atlas and virtual atlas mixed training;
the real atlas is obtained by shooting the sea surface on site, and the virtual atlas is obtained by sea condition simulation of a virtual twin environment;
the real atlas is input into a virtual twin environment, the two-class model and the multi-class model for training are integrated into the twin environment, the model is trained in the twin environment, and the trained model parameters are output.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310167630.7A CN116246139A (en) | 2023-02-27 | 2023-02-27 | Target identification method based on multi-sensor fusion for unmanned ship navigation environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310167630.7A CN116246139A (en) | 2023-02-27 | 2023-02-27 | Target identification method based on multi-sensor fusion for unmanned ship navigation environment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116246139A true CN116246139A (en) | 2023-06-09 |
Family
ID=86634652
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310167630.7A Withdrawn CN116246139A (en) | 2023-02-27 | 2023-02-27 | Target identification method based on multi-sensor fusion for unmanned ship navigation environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116246139A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117033910A (en) * | 2023-09-27 | 2023-11-10 | 宁波麦思捷科技有限公司武汉分公司 | Sea surface high-precision signal processing method and system |
| CN118135420A (en) * | 2024-03-21 | 2024-06-04 | 中国科学院空天信息创新研究院 | Ship identification device, method and electronic equipment based on multi-system data fusion |
-
2023
- 2023-02-27 CN CN202310167630.7A patent/CN116246139A/en not_active Withdrawn
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117033910A (en) * | 2023-09-27 | 2023-11-10 | 宁波麦思捷科技有限公司武汉分公司 | Sea surface high-precision signal processing method and system |
| CN117033910B (en) * | 2023-09-27 | 2023-12-19 | 宁波麦思捷科技有限公司武汉分公司 | Sea surface high-precision signal processing method and system |
| CN118135420A (en) * | 2024-03-21 | 2024-06-04 | 中国科学院空天信息创新研究院 | Ship identification device, method and electronic equipment based on multi-system data fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111626290B (en) | Infrared ship target detection and identification method under complex sea surface environment | |
| CN104933680B (en) | A kind of intelligent quick sea fog minimizing technology of unmanned boat vision system video | |
| CN116246139A (en) | Target identification method based on multi-sensor fusion for unmanned ship navigation environment | |
| CN109919026B (en) | Surface unmanned ship local path planning method | |
| Zhang et al. | Research on unmanned surface vehicles environment perception based on the fusion of vision and lidar | |
| CN109859202B (en) | A deep learning detection method based on USV water surface optical target tracking | |
| CN107808161A (en) | A kind of Underwater targets recognition based on light vision | |
| CN106372590A (en) | Sea surface ship intelligent tracking system and method based on machine vision | |
| Duan et al. | Unmanned aerial vehicle recognition of maritime small-target based on biological eagle-eye vision adaptation mechanism | |
| CN116486243A (en) | DP-ViT-based target detection method in sonar images | |
| CN115641767A (en) | Unmanned ship perception experiment platform device | |
| CN112417948A (en) | Method for accurately guiding lead-in ring of underwater vehicle based on monocular vision | |
| Li et al. | Vision-based target detection and positioning approach for underwater robots | |
| Zhang et al. | Nearshore vessel detection based on Scene-mask R-CNN in remote sensing image | |
| CN116486102A (en) | Infrared dim target detection method based on mixed spatial modulation characteristic convolutional neural network | |
| CN113850166B (en) | A ship image recognition method and system based on convolutional neural network | |
| Qiu et al. | Underwater sea cucumbers detection based on pruned SSD | |
| Yang et al. | Knowledge distillation for feature extraction in underwater VSLAM | |
| CN112307943B (en) | Water area man-boat target detection method, system, terminal and medium | |
| CN116664545B (en) | Offshore benthos quantitative detection method and system based on deep learning | |
| CN116185049A (en) | Unmanned helicopter autonomous landing method based on visual guidance | |
| Gentili et al. | Detection and classification of man-made objects for the autonomy of underwater robots | |
| CN113920300A (en) | Visible light remote sensing image ship target identification method and system | |
| Roy et al. | Under water objects detection and classification using deep learning technique | |
| Cheng et al. | A robust visual servo scheme for underwater pipeline following |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20230609 |