CN113705503B - An abnormal behavior detection system and method based on multimodal information fusion - Google Patents
An abnormal behavior detection system and method based on multimodal information fusion Download PDFInfo
- Publication number
- CN113705503B CN113705503B CN202111025975.6A CN202111025975A CN113705503B CN 113705503 B CN113705503 B CN 113705503B CN 202111025975 A CN202111025975 A CN 202111025975A CN 113705503 B CN113705503 B CN 113705503B
- Authority
- CN
- China
- Prior art keywords
- ship
- track
- target
- information
- behavior detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 94
- 206010000117 Abnormal behaviour Diseases 0.000 title claims abstract description 48
- 230000004927 fusion Effects 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 title claims abstract description 13
- 238000012544 monitoring process Methods 0.000 claims abstract description 55
- 230000002159 abnormal effect Effects 0.000 claims abstract description 29
- 238000012545 processing Methods 0.000 claims description 34
- 238000004458 analytical method Methods 0.000 claims description 15
- 238000007781 pre-processing Methods 0.000 claims description 6
- 230000009429 distress Effects 0.000 claims description 4
- 230000005856 abnormality Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Radar Systems Or Details Thereof (AREA)
- Traffic Control Systems (AREA)
Abstract
本发明提供一种基于多模态信息融合的异常行为检测系统及方法，其中该方法包括如下步骤：S1采集目标海域上船舶的AIS报文信息和雷达系统的雷达监测信息；S2基于获取的AIS报文信息和雷达监测信息对船舶进行一次异常行为检测处理，得到船舶的一次行为检测结果；S3针对一次行为检测结果为异常的目标船舶，根据该目标船舶的定位信息，进一步采集该目标船舶的视频图像信息；S4根据获取的目标船舶的视频图像信息进行二次行为检测处理，得到该目标船舶的二次行为检测结果。本发明有助于提高船舶监管系统对目标海域船舶监管的可靠性。
The present invention provides an abnormal behavior detection system and method based on multimodal information fusion, wherein the method comprises the following steps: S1 collects AIS message information of ships in the target sea area and radar monitoring information of the radar system; S2 performs a primary abnormal behavior detection process on the ship based on the acquired AIS message information and radar monitoring information to obtain a primary behavior detection result of the ship; S3 further collects video image information of the target ship according to the positioning information of the target ship for the target ship whose primary behavior detection result is abnormal; S4 performs a secondary behavior detection process according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship. The present invention is helpful to improve the reliability of the ship supervision system for the supervision of ships in the target sea area.
Description
Technical Field
The invention relates to the technical field of sea area supervision, in particular to an abnormal behavior detection system and method based on multi-mode information fusion.
Background
At present, for the supervision system of the ship on the sea, detection of the behavior of the ship on the sea is mostly realized by means of AIS data transmitted by an application ship automatic identification system (AIS, automatic Identification System) on the ship, but in the technical scheme of the prior art, the behavior of the ship on the sea cannot be accurately analyzed by solely relying on the AIS data to perform ship analysis on the sea, and the reliability of the ship supervision system is affected when the ship-borne AIS equipment fails and cannot normally run, the ship-borne AIS is manually closed, the AIS data is inaccurate, or the ship is not loaded with the AIS equipment.
Disclosure of Invention
Aiming at the technical problem that the reliability is insufficient in the technical scheme for carrying out marine ship analysis only by means of AIS data, the invention aims to provide an abnormal behavior detection system and method based on multi-mode information fusion.
The aim of the invention is realized by adopting the following technical scheme:
In a first aspect, the present invention provides a method for detecting abnormal behavior based on multimodal information fusion, including:
S1, acquiring AIS message information of a ship on a target sea area and radar monitoring information of a radar system;
s2, performing primary abnormal behavior detection processing on the ship based on the acquired AIS message information and radar monitoring information to obtain a primary behavior detection result of the ship;
s3, aiming at a target ship with abnormal primary behavior detection results, further acquiring video image information of the target ship according to positioning information of the target ship;
and S4, performing secondary behavior detection processing according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship.
In one embodiment, step S1 specifically includes:
receiving AIS message information transmitted by a ship on a target sea area, wherein the AIS message information comprises a ship ID and ship positioning data;
And receiving radar monitoring information transmitted by a radar system for monitoring the target sea area, wherein the radar monitoring information comprises positioning information of a ship on the target sea area.
In one embodiment, step S2 specifically includes:
S21, carrying out positioning consistency judgment according to ship positioning data in AIS message information at a designated moment and ship positioning information in radar monitoring information to obtain a positioning consistency judgment result;
S22, acquiring first ship track data according to ship positioning data in AIS message information in a set time period, acquiring second ship track data according to positioning data of a corresponding ship in radar monitoring information in the set time period, and performing track consistency judgment according to the first ship track data and the second ship track data to obtain a track consistency judgment result;
S23, outputting a primary behavior detection result of the ship as abnormal when the positioning consistency judgment result or the track consistency judgment result of the ship is abnormal.
In one embodiment, step S3 specifically includes:
S31, aiming at a target ship with an abnormal primary behavior detection result, acquiring positioning data of the target ship according to radar monitoring information;
s32, transmitting the positioning data of the target ship to an unmanned aerial vehicle control system, and assigning the unmanned aerial vehicle to arrive at a corresponding positioning site by the unmanned aerial vehicle control system to acquire video image information of the target ship;
s33 receives video image information of a target ship transmitted by the unmanned aerial vehicle.
In one embodiment, step S4 specifically includes:
S41, according to the acquired video image information of the target ship, performing abnormal behavior recognition processing based on the trained abnormal behavior detection model to obtain an abnormal behavior recognition result of the target ship.
In one embodiment, step S41 includes:
According to the acquired video image information of the target ship, a trained ship classification and identification model is adopted to identify the type of the ship;
Based on the acquired ship type information of the target ship, analyzing whether the target ship is positioned in a warning area corresponding to the type of the target ship according to the positioning data of the target ship, and outputting a secondary behavior detection result of the target ship.
In one embodiment, step S41 includes:
based on the obtained video image information of the target ship, detecting whether a distress identification is sent out in the ship by adopting a trained monitoring model, and outputting a secondary behavior detection result of the target ship.
In a second aspect, the present invention provides an abnormal behavior detection system based on multi-modal information fusion, including:
the first acquisition module is used for acquiring AIS message information of the ship on the target sea area and radar monitoring information of a radar system;
the first analysis module is used for carrying out primary abnormal behavior detection processing on the ship based on the acquired AIS message information and radar monitoring information to obtain a primary behavior detection result of the ship;
the second acquisition module is used for further acquiring video image information of the target ship according to the positioning information of the target ship aiming at the target ship with the abnormal primary behavior detection result;
and the second analysis module is used for carrying out secondary behavior detection processing according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship.
The ship behavior detection method has the beneficial effects that firstly, the AIS data of the ship is judged, the target ship with abnormality in the AIS data can be rapidly screened, and the behavior detection of the target ship is further carried out by adopting a mode based on video image data aiming at the target ship which cannot be subjected to behavior detection according to the AIS data, so that the reliability of the ship supervision system on the ship supervision of the target sea area can be improved.
Drawings
The invention will be further described with reference to the accompanying drawings, in which embodiments do not constitute any limitation of the invention, and other drawings can be obtained by one of ordinary skill in the art without inventive effort from the following drawings.
FIG. 1 is a flowchart of an abnormal behavior detection method based on multimodal information fusion according to an exemplary embodiment of the present invention;
FIG. 2 is a block diagram of an exemplary embodiment of a system for detecting abnormal behavior based on multimodal information fusion in accordance with the present invention.
Detailed Description
The invention is further described in connection with the following application scenario.
The invention, which is shown in the embodiment of fig. 1, shows an abnormal behavior detection method based on multi-modal information fusion, which comprises the following steps:
S1, acquiring AIS message information of a ship on a target sea area and radar monitoring information of a radar system;
s2, performing primary abnormal behavior detection processing on the ship based on the acquired AIS message information and radar monitoring information to obtain a primary behavior detection result of the ship;
s3, aiming at a target ship with abnormal primary behavior detection results, further acquiring video image information of the target ship according to positioning information of the target ship;
and S4, performing secondary behavior detection processing according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship.
In the embodiment, the abnormal behavior detection system firstly respectively acquires AIS message information transmitted by a ship and radar monitoring information transmitted by a radar system for monitoring a target sea area range, performs primary abnormal behavior detection processing on the ship on the target sea area according to the acquired AIS message information and radar monitoring information (for example, detecting whether the ship detected according to the radar monitoring information has corresponding AIS message information or not), detects whether the AIS message information transmitted by the ship is accurate or not, and obtains a primary behavior detection result of the ship, and further acquires positioning information of the target ship according to the radar monitoring information and video image information of the target ship aiming at the target ship with abnormal AIS message information, and performs secondary behavior detection on the target ship based on the video image information of the target ship to obtain a behavior detection result of the target ship. According to the technical scheme, firstly, the AIS data of the ship is judged, the target ship with abnormality in the AIS data can be screened out quickly, and the behavior detection of the target ship is further carried out in a mode based on video image data aiming at the target ship which cannot be detected according to the AIS data, so that the reliability of the ship supervision system on the supervision of the target sea area ship can be improved.
In a scene, the abnormal behavior detection method and system based on multi-mode information fusion can be applied to a ship supervision system for supervising ships on a target sea area.
In one embodiment, step S1 specifically includes:
receiving AIS message information transmitted by a ship on a target sea area, wherein the AIS message information comprises a ship ID and ship positioning data;
And receiving radar monitoring information transmitted by a radar system for monitoring the target sea area, wherein the radar monitoring information comprises positioning information of a ship on the target sea area.
In one embodiment, step S2 specifically includes:
S21, carrying out positioning consistency judgment according to ship positioning data in AIS message information at a designated moment and ship positioning information in radar monitoring information to obtain a positioning consistency judgment result;
In a scene, extracting positioning data of a ship according to the acquired AIS message information at a certain moment, detecting whether the positioning data detect the occurrence of the corresponding ship according to the radar monitoring information at the same moment, if so, obtaining a consistency judgment result to be normal, and carrying out identity identification on the ship in the radar monitoring information, otherwise, judging that the positioning consistency judgment result is abnormal when the radar monitoring information at the certain moment detects that the ship occurs in a certain positioning, but does not acquire the AIS message information corresponding to the positioning at the same moment, and marking the consistency judgment result to be an abnormal target ship according to the radar monitoring information.
In a scenario, when a ship appears in the radar monitoring information for the first time and the positioning consistency determination result of the ship is normal, the process jumps to step S22, and the positioning consistency determination of step S21 is not performed on the ship.
S22, acquiring first ship track data according to ship positioning data in AIS message information in a set time period, acquiring second ship track data according to positioning data of a corresponding ship in radar monitoring information in the set time period, and performing track consistency judgment according to the first ship track data and the second ship track data to obtain a track consistency judgment result;
In a scene, in view of continuously judging and judging the positioning consistency through AIS message information and radar monitoring information corresponding to the same moment, the situation that the robustness is low due to the influence of data transmission or other factors is easy to occur. Therefore, when the ship determined to be normal by the positioning consistency is detected in step S21, the track data of the ship is further tracked, that is, the track consistency determination is performed based on the first ship track data acquired aiming at the AIS message information and the second ship track data acquired based on the radar monitoring information, so as to help to improve the robustness of the primary behavior detection result.
In one embodiment, the track consistency determination specifically includes:
Acquiring first ship track data GA= { D a(1),Da(2),…,Da(t),…Da (T) } according to ship positioning data in AIS message information, wherein GA represents a first ship track in a set time period, D a (T) represents positioning data corresponding to the T-th moment in the first ship track, wherein t=1, 2, & gt, T and T represent the total number of the first ship track positioning data in the time period, and acquiring corresponding speed change data VA= { V a(1),Va(2),…,Va(t),…Va (T) } and heading data theta A= { theta a(1),θa(2),…,θa(t),…θa (T) } according to the first ship track data, wherein V a (T) and theta a (T) respectively represent the speed and heading corresponding to the T-th moment in the first ship track;
Acquiring second ship track data according to positioning data of a corresponding ship in radar monitoring information, wherein GL= { D l(1),Dl(2),…,Dl(t),…Dl (T) }, GL represents a second ship track in a set time period, D l (T) represents positioning data corresponding to a T-th moment in the second ship track, and acquiring corresponding speed change data VL= { V l(1),Vl(2),…,Vl(t),…Vl (T) } and heading data thetal= { theta l(1),θl(2),…,θl(t),…θl (T) } according to the second ship track data, wherein V l (T) and theta l (T) respectively represent corresponding speed and heading of the T-th moment in the second ship track;
acquiring speed deviation between the first ship track and the second ship track Deviation of distanceAnd heading deviation
And carrying out track consistency judgment based on the acquired speed deviation, distance deviation and heading deviation, wherein the adopted judgment condition function is as follows: Wherein v ' represents a set speed deviation threshold, d ' represents a set distance deviation threshold, and θ ' represents a set heading deviation threshold;
if the judging condition function is met, judging that the track consistency judging result is normal, otherwise, judging that the track consistency judging result is abnormal.
S23, outputting a primary behavior detection result of the ship as abnormal when the positioning consistency judgment result or the track consistency judgment result of the ship is abnormal, and outputting the primary behavior detection result as normal when the positioning consistency judgment result and the track consistency judgment result are normal.
In the above embodiment, the abnormal behavior detection system performs comparison and consistency analysis according to the obtained AIS message information and the radar monitoring information monitored by the radar system, performs one-time behavior judgment on the ship on the target sea area, and based on the comparison, marks the target ship when there is no corresponding AIS message data of the ship or the AIS message data does not correspond to the ship positioning data monitored by the radar monitoring information (for example, the situation that the ship-borne AIS equipment fails and cannot normally operate, the ship-borne AIS is manually closed, the AIS data is inaccurate, the AIS data is tampered or the ship is not loaded with the AIS equipment, etc.), and performs further behavior detection processing on the target ship.
Under the condition that the primary behavior detection result is normal, the ship in the target sea area normally transmits AIS data to an abnormal behavior detection system, and the abnormal behavior detection system can detect the behavior of the ship according to the acquired AIS data:
In one embodiment, the AIS message information further includes ship destination path data, record data, speed data, heading data, and the like.
Step S2 further includes:
S20, judging whether the ship is abnormal in navigational speed or not (the speed is too high, the speed is too low, the ship is wandering and the like) according to navigational speed data in the AIS message information, judging whether the ship is abnormal in navigational route or not (the ship illegally enters a warning area, does not travel to a destination and the like) according to destination path data, traveling direction data and ship positioning data in the AIS message information, judging whether the ship is legal or not according to record data in the AIS message information and the like;
Step S23 also comprises outputting a primary behavior detection result of the ship as abnormal when the speed of the ship is abnormal, the route is abnormal or the ship is illegal.
In one embodiment, step S3 specifically includes:
S31, aiming at a target ship with an abnormal primary behavior detection result, acquiring positioning data of the target ship according to radar monitoring information;
s32, transmitting the positioning data of the target ship to an unmanned aerial vehicle control system, and assigning the unmanned aerial vehicle to arrive at a corresponding positioning site by the unmanned aerial vehicle control system to acquire video image information of the target ship;
s33 receives video image information of a target ship transmitted by the unmanned aerial vehicle.
The system comprises an unmanned aerial vehicle control system, an anomaly analysis system, an unmanned aerial vehicle control system, a target ship, an anomaly detection result, an anomaly analysis system and an anomaly analysis system, wherein the anomaly analysis system is connected with the unmanned aerial vehicle control system, the unmanned aerial vehicle is assigned to the appointed target location for target location video image acquisition, further video image information acquisition can be carried out on the target ship with the anomaly detection result of one time, and a foundation is laid for further anomaly behavior detection of the target ship according to the acquired video image data of the target ship by the subsequent anomaly analysis system.
The unmanned aerial vehicle is provided with a positioning device and an image acquisition device, and can shoot the target ship to a designated place according to real-time positioning information (acquired by a radar monitoring system) of the target ship.
In one embodiment, step S4 specifically includes:
S41, according to the acquired video image information of the target ship, performing abnormal behavior recognition processing based on the trained abnormal behavior detection model to obtain an abnormal behavior recognition result of the target ship.
In one embodiment, step S41 includes:
According to the acquired video image information of the target ship, a trained ship classification and identification model is adopted to identify the type of the ship;
Based on the acquired ship type information of the target ship, analyzing whether the target ship is positioned in a warning area corresponding to the type of the target ship according to the positioning data of the target ship, and outputting a secondary behavior detection result of the target ship.
In one embodiment, step S41 includes:
based on the obtained video image information of the target ship, detecting whether a distress identification is sent out in the ship by adopting a trained monitoring model, and outputting a secondary behavior detection result of the target ship.
In a scene, whether a flag for asking for help is hung on a ship or whether a light signal for asking for help is sent out at night or not can be monitored through a video image aiming at a ship which fails in the sea area and causes an AIS system to normally operate and send out a distress signal, timely discovery of trapped ships and the marine ships is facilitated through an abnormal behavior detection system, and rescue work is timely arranged.
In one embodiment, step S4 further includes:
s40, preprocessing the acquired video image of the target ship to obtain a preprocessed video image;
S41, according to the preprocessed target image, performing abnormal behavior recognition processing based on the trained abnormal behavior detection model to obtain an abnormal behavior recognition result of the target ship.
In one embodiment, step S40 performs preprocessing on the acquired video image of the target ship, specifically includes:
converting the acquired video image from an RGB color space to an LAB color space, and acquiring a brightness component L of the video image;
For the obtained luminance component L, luminance enhancement processing is performed for each pixel point in the luminance component L, where:
when the luminance component value L (x, y) of the pixel point (x, y) is equal to the average luminance component value When the absolute value of the difference is smaller than the set brightness interval L T, that isThe pixel (x, y) is luminance enhanced using the following first luminance enhancement function:
Wherein, Alpha represents a set scale factor, wherein alpha epsilon (0, 1) and L 'represents a set standard brightness value, and L' epsilon [60,70];
when the luminance component value L (x, y) of the pixel point (x, y) is equal to the average luminance component value When the absolute value of the difference is greater than or equal to the set brightness interval L T, that isThe pixel (x, y) is luminance enhanced using the following second luminance enhancement function:
Where beta denotes a set scale factor, where beta E (0, 1), Representing the set compensation factor, where
According to the brightness component value of each pixel point after brightness enhancement processing, brightness component after brightness enhancement processing
Based on luminance component after luminance enhancement processingAnd converting from the LAB color space to the RGB color space again to obtain the video image with enhanced brightness as a preprocessed video image.
According to the embodiment, aiming at the condition that the brightness of the collected video image of the target ship is uneven (such as a reflection point or overall brightness is higher) and the definition of the ship on the sea is influenced under the condition that the direct sunlight, the reflection of the sea surface, the low night brightness and the reflection of the water surface are easily received, the technical scheme for carrying out brightness enhancement processing on the obtained brightness image can adapt to the condition of the reflection of the strong light on the water surface at daytime or night, the overall brightness level and the local brightness level of the image are enhanced and adjusted, the definition of the video image and the contrast of the target ship are improved, and a foundation is laid for further behavior detection of the target ship according to the video image.
Meanwhile, referring to fig. 2, there is shown an abnormal behavior detection system based on multi-modal information fusion according to the present invention, including:
the first acquisition module is used for acquiring AIS message information of the ship on the target sea area and radar monitoring information of a radar system;
the first analysis module is used for carrying out primary abnormal behavior detection processing on the ship based on the acquired AIS message information and radar monitoring information to obtain a primary behavior detection result of the ship;
the second acquisition module is used for further acquiring video image information of the target ship according to the positioning information of the target ship aiming at the target ship with the abnormal primary behavior detection result;
and the second analysis module is used for carrying out secondary behavior detection processing according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship.
The modules in the abnormal behavior detection system are further configured to correspondingly implement method steps corresponding to embodiments in the abnormal behavior detection method based on multi-mode information fusion, which are not repeated herein.
It should be noted that, in each embodiment of the present invention, each functional unit/module may be integrated in one processing unit/module, or each unit/module may exist alone physically, or two or more units/modules may be integrated in one unit/module. The integrated units/modules described above may be implemented either in hardware or in software functional units/modules.
From the description of the embodiments above, it will be apparent to those skilled in the art that the embodiments described herein may be implemented in hardware, software, firmware, middleware, code, or any suitable combination thereof. For a hardware implementation, the processors may be implemented within one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof. For a software implementation, some or all of the flow of an embodiment may be accomplished by a computer program to instruct the associated hardware. When implemented, the above-described programs may be stored in or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. Computer-readable media can include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
Finally, it should be noted that the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the scope of the present invention, and although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.
Claims (6)
1. The abnormal behavior detection method based on multi-mode information fusion is characterized by comprising the following steps of:
S1, acquiring AIS message information of a ship on a target sea area and radar monitoring information of a radar system;
s2, carrying out primary abnormal behavior detection processing on the ship based on the acquired AIS message information and radar monitoring information to obtain a primary behavior detection result of the ship, wherein the method specifically comprises the following steps:
S21, carrying out positioning consistency judgment according to ship positioning data in AIS message information at a designated moment and ship positioning information in radar monitoring information to obtain a positioning consistency judgment result;
S22, acquiring first ship track data according to ship positioning data in AIS message information in a set time period, acquiring second ship track data according to positioning data of a corresponding ship in radar monitoring information in the set time period, and performing track consistency judgment according to the first ship track data and the second ship track data to obtain a track consistency judgment result;
the track consistency judgment specifically comprises the following steps:
Acquiring first ship track data GA= { D a(1),Da(2),…,Da(t),…Da (T) } according to ship positioning data in AIS message information, wherein GA represents a first ship track in a set time period, D a (T) represents positioning data corresponding to the T-th moment in the first ship track, wherein t=1, 2, & gt, T and T represent the total number of the first ship track positioning data in the time period, and acquiring corresponding speed change data VA= { V a(1),Va(2),…,Va(t),…Va (T) } and heading data theta A= { theta a(1),θa(2),…,θa(t),…θa (T) } according to the first ship track data, wherein V a (T) and theta a (T) respectively represent the speed and heading corresponding to the T-th moment in the first ship track;
Acquiring second ship track data according to positioning data of a corresponding ship in radar monitoring information, wherein GL= { D l(1),Dl(2),…,Dl(t),…Dl (T) }, GL represents a second ship track in a set time period, D l (T) represents positioning data corresponding to a T-th moment in the second ship track, and acquiring corresponding speed change data VL= { V l(1),Vl(2),…,Vl(t),…Vl (T) } and heading data thetal= { theta l(1),θl(2),…,θl(t),…θl (T) } according to the second ship track data, wherein V l (T) and theta l (T) respectively represent corresponding speed and heading of the T-th moment in the second ship track;
acquiring speed deviation between the first ship track and the second ship track Deviation of distanceAnd heading deviation
And carrying out track consistency judgment based on the acquired speed deviation, distance deviation and heading deviation, wherein the adopted judgment condition function is as follows: Wherein v ' represents a set speed deviation threshold, d ' represents a set distance deviation threshold, and θ ' represents a set heading deviation threshold;
If the judging condition function is met, judging that the track consistency judging result is normal, otherwise, judging that the track consistency judging result is abnormal;
s23, outputting a primary behavior detection result of the ship as abnormal when the positioning consistency judgment result or the track consistency judgment result of the ship is abnormal;
s3, aiming at a target ship with abnormal primary behavior detection results, further acquiring video image information of the target ship according to positioning information of the target ship;
s4, performing secondary behavior detection processing according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship;
Wherein, step S4 includes:
s40, preprocessing the acquired video image of the target ship to obtain a preprocessed video image;
s41, carrying out abnormal behavior recognition processing based on a trained abnormal behavior detection model according to the preprocessed target image to obtain an abnormal behavior recognition result of the target ship;
the step S40 of preprocessing the acquired video image of the target ship specifically includes:
converting the acquired video image from an RGB color space to an LAB color space, and acquiring a brightness component L of the video image;
For the obtained luminance component L, luminance enhancement processing is performed for each pixel point in the luminance component L, where:
when the luminance component value L (x, y) of the pixel point (x, y) is equal to the average luminance component value When the absolute value of the difference is smaller than the set brightness interval L T, that isThe pixel (x, y) is luminance enhanced using the following first luminance enhancement function:
Wherein, Alpha represents a set scale factor, wherein alpha epsilon (0, 1) and L 'represents a set standard brightness value, and L' epsilon [60,70];
when the luminance component value L (x, y) of the pixel point (x, y) is equal to the average luminance component value When the absolute value of the difference is greater than or equal to the set brightness interval L T, that isThe pixel (x, y) is luminance enhanced using the following second luminance enhancement function:
Where beta denotes a set scale factor, where beta E (0, 1), Representing the set compensation factor, where
According to the brightness component value of each pixel point after brightness enhancement processing, brightness component after brightness enhancement processing
Based on luminance component after luminance enhancement processingAnd converting from the LAB color space to the RGB color space again to obtain the video image with enhanced brightness as a preprocessed video image.
2. The abnormal behavior detection method based on multi-modal information fusion according to claim 1, wherein step S1 specifically includes:
receiving AIS message information transmitted by a ship on a target sea area, wherein the AIS message information comprises a ship ID and ship positioning data;
And receiving radar monitoring information transmitted by a radar system for monitoring the target sea area, wherein the radar monitoring information comprises positioning information of a ship on the target sea area.
3. The abnormal behavior detection method based on multi-modal information fusion according to claim 1, wherein step S3 specifically includes:
S31, aiming at a target ship with an abnormal primary behavior detection result, acquiring positioning data of the target ship according to radar monitoring information;
s32, transmitting the positioning data of the target ship to an unmanned aerial vehicle control system, and assigning the unmanned aerial vehicle to arrive at a corresponding positioning site by the unmanned aerial vehicle control system to acquire video image information of the target ship;
s33 receives video image information of a target ship transmitted by the unmanned aerial vehicle.
4. The abnormal behavior detection method based on multi-modal information fusion according to claim 1, wherein step S41 includes:
According to the acquired video image information of the target ship, a trained ship classification and identification model is adopted to identify the type of the ship;
Based on the acquired ship type information of the target ship, analyzing whether the target ship is positioned in a warning area corresponding to the type of the target ship according to the positioning data of the target ship, and outputting a secondary behavior detection result of the target ship.
5. The abnormal behavior detection method based on multi-modal information fusion according to claim 1, wherein step S41 includes:
based on the obtained video image information of the target ship, detecting whether a distress identification is sent out in the ship by adopting a trained monitoring model, and outputting a secondary behavior detection result of the target ship.
6. An abnormal behavior detection system based on multi-modal information fusion, comprising:
the first acquisition module is used for acquiring AIS message information of the ship on the target sea area and radar monitoring information of a radar system;
The first analysis module is used for carrying out primary abnormal behavior detection processing on the ship based on the acquired AIS message information and radar monitoring information to obtain a primary behavior detection result of the ship, and specifically comprises the following steps:
Carrying out positioning consistency judgment according to the ship positioning data in the AIS message information at the appointed moment and the ship positioning information in the radar monitoring information to obtain a positioning consistency judgment result;
Acquiring first ship track data according to ship positioning data in AIS message information in a set time period, acquiring second ship track data according to positioning data of a corresponding ship in radar monitoring information in the set time period, and performing track consistency judgment according to the first ship track data and the second ship track data to obtain a track consistency judgment result;
the track consistency judgment specifically comprises the following steps:
Acquiring first ship track data GA= { D a(1),Da(2),…,Da(t),…Da (T) } according to ship positioning data in AIS message information, wherein GA represents a first ship track in a set time period, D a (T) represents positioning data corresponding to the T-th moment in the first ship track, wherein t=1, 2, & gt, T and T represent the total number of the first ship track positioning data in the time period, and acquiring corresponding speed change data VA= { V a(1),Va(2),…,Va(t),…Va (T) } and heading data theta A= { theta a(1),θa(2),…,θa(t),…θa (T) } according to the first ship track data, wherein V a (T) and theta a (T) respectively represent the speed and heading corresponding to the T-th moment in the first ship track;
Acquiring second ship track data according to positioning data of a corresponding ship in radar monitoring information, wherein GL= { D l(1),Dl(2),…,Dl(t),…Dl (T) }, GL represents a second ship track in a set time period, D l (T) represents positioning data corresponding to a T-th moment in the second ship track, and acquiring corresponding speed change data VL= { V l(1),Vl(2),…,Vl(t),…Vl (T) } and heading data thetal= { theta l(1),θl(2),…,θl(t),…θl (T) } according to the second ship track data, wherein V l (T) and theta l (T) respectively represent corresponding speed and heading of the T-th moment in the second ship track;
acquiring speed deviation between the first ship track and the second ship track Deviation of distanceAnd heading deviation
And carrying out track consistency judgment based on the acquired speed deviation, distance deviation and heading deviation, wherein the adopted judgment condition function is as follows: Wherein v ' represents a set speed deviation threshold, d ' represents a set distance deviation threshold, and θ ' represents a set heading deviation threshold;
If the judging condition function is met, judging that the track consistency judging result is normal, otherwise, judging that the track consistency judging result is abnormal;
outputting a primary behavior detection result of the ship as abnormal when the positioning consistency judgment result or the track consistency judgment result of the ship is abnormal;
the second acquisition module is used for further acquiring video image information of the target ship according to the positioning information of the target ship aiming at the target ship with the abnormal primary behavior detection result;
the second analysis module performs secondary behavior detection processing according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship, and the second analysis module comprises the following steps:
preprocessing the obtained video image of the target ship to obtain a preprocessed video image;
according to the preprocessed target image, carrying out abnormal behavior recognition processing based on the trained abnormal behavior detection model to obtain an abnormal behavior recognition result of the target ship;
the method for preprocessing the acquired video image of the target ship specifically comprises the following steps:
converting the acquired video image from an RGB color space to an LAB color space, and acquiring a brightness component L of the video image;
For the obtained luminance component L, luminance enhancement processing is performed for each pixel point in the luminance component L, where:
when the luminance component value L (x, y) of the pixel point (x, y) is equal to the average luminance component value When the absolute value of the difference is smaller than the set brightness interval L T, that isThe pixel (x, y) is luminance enhanced using the following first luminance enhancement function:
Wherein, Alpha represents a set scale factor, wherein alpha epsilon (0, 1) and L 'represents a set standard brightness value, and L' epsilon [60,70];
when the luminance component value L (x, y) of the pixel point (x, y) is equal to the average luminance component value When the absolute value of the difference is greater than or equal to the set brightness interval L T, that isThe pixel (x, y) is luminance enhanced using the following second luminance enhancement function:
Where beta denotes a set scale factor, where beta E (0, 1), Representing the set compensation factor, where
According to the brightness component value of each pixel point after brightness enhancement processing, brightness component after brightness enhancement processing
Based on luminance component after luminance enhancement processingAnd converting from the LAB color space to the RGB color space again to obtain the video image with enhanced brightness as a preprocessed video image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111025975.6A CN113705503B (en) | 2021-09-02 | 2021-09-02 | An abnormal behavior detection system and method based on multimodal information fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111025975.6A CN113705503B (en) | 2021-09-02 | 2021-09-02 | An abnormal behavior detection system and method based on multimodal information fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113705503A CN113705503A (en) | 2021-11-26 |
| CN113705503B true CN113705503B (en) | 2025-01-10 |
Family
ID=78657418
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111025975.6A Active CN113705503B (en) | 2021-09-02 | 2021-09-02 | An abnormal behavior detection system and method based on multimodal information fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113705503B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116047986B (en) * | 2023-03-31 | 2023-06-09 | 山东科技大学 | Ship consistency control method under low communication coupling strength |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110928967A (en) * | 2019-11-05 | 2020-03-27 | 厦门北斗通信息技术股份有限公司 | Ship behavior analysis method and system based on big data |
| CN112857360A (en) * | 2021-03-22 | 2021-05-28 | 哈尔滨工程大学 | Ship navigation multi-information fusion method |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7817079B1 (en) * | 2008-04-17 | 2010-10-19 | The United States Of America As Represented By The Secretary Of The Navy | System and method for collection of global AIS and radar track information |
| US8180107B2 (en) * | 2009-02-13 | 2012-05-15 | Sri International | Active coordinated tracking for multi-camera systems |
| GB2501542A (en) * | 2012-04-28 | 2013-10-30 | Bae Systems Plc | Abnormal behaviour detection in video or image surveillance data |
| CN104660993B (en) * | 2015-02-09 | 2018-06-08 | 武汉理工大学 | Maritime affairs intelligent control method and system based on AIS and CCTV |
| CN106970387B (en) * | 2017-04-19 | 2019-07-23 | 武汉理工大学 | A kind of cruiseway Traffic flow detection method based on AIS and Radar Data Fusion |
| CN107818571B (en) * | 2017-12-11 | 2018-07-20 | 珠海大横琴科技发展有限公司 | Ship automatic tracking method and system based on deep learning network and average drifting |
| CN109460740B (en) * | 2018-11-15 | 2020-08-11 | 上海埃威航空电子有限公司 | Ship identity recognition method based on AIS and video data fusion |
| CN110363094A (en) * | 2019-06-20 | 2019-10-22 | 珠海云航智能技术有限公司 | A kind of ship abnormal behaviour recognition methods, device and terminal device |
| CN111899450B (en) * | 2020-07-24 | 2022-04-29 | 宁波盛洋电子科技有限公司 | Method and system for monitoring ships entering and exiting port and finding dangerous ships |
-
2021
- 2021-09-02 CN CN202111025975.6A patent/CN113705503B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110928967A (en) * | 2019-11-05 | 2020-03-27 | 厦门北斗通信息技术股份有限公司 | Ship behavior analysis method and system based on big data |
| CN112857360A (en) * | 2021-03-22 | 2021-05-28 | 哈尔滨工程大学 | Ship navigation multi-information fusion method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113705503A (en) | 2021-11-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109725310B (en) | Ship positioning supervision system based on YOLO algorithm and shore-based radar system | |
| CN105427342B (en) | A kind of underwater Small object sonar image target detection tracking method and system | |
| CN113112540B (en) | Method for positioning ship image target by using AIS (automatic identification system) Calibration CCTV (CCTV) camera in VTS (video tape server) system | |
| CN111460917B (en) | Airport abnormal behavior detection system and method based on multi-mode information fusion | |
| CN107909601A (en) | A kind of shipping anti-collision early warning video detection system and detection method suitable for navigation mark | |
| CN112862821A (en) | Water leakage detection method and device based on image processing, computing equipment and medium | |
| US20190103020A1 (en) | Vehicle search system, vehicle search method, and vehicle used therefor | |
| CN113705503B (en) | An abnormal behavior detection system and method based on multimodal information fusion | |
| KR102558387B1 (en) | System and method for providing motion information and size information of ship based on real-time radar image | |
| CN115034324A (en) | Multi-sensor fusion perception efficiency enhancement method | |
| GB2608378A (en) | Methods and systems for detecting vessels | |
| CN110120073B (en) | A method for guiding the recovery of unmanned boats based on visual signals of beacons | |
| Tang et al. | Fooling perception via location: A case of region-of-interest attacks on traffic light detection in autonomous driving | |
| CN113705502A (en) | Ship target behavior understanding system integrating target detection and target tracking | |
| KR20220065384A (en) | Ship entry/departure monitoring system using real-time port images and ship identification device signals | |
| CN112883856A (en) | Monitoring method, monitoring device, electronic equipment and storage medium | |
| CN116469276A (en) | Water area safety early warning method, device, equipment and storage medium | |
| KR102479959B1 (en) | Artificial intelligence based integrated alert method and object monitoring device | |
| CN108803651B (en) | A UAV highway inspection method based on image features | |
| CN205142399U (en) | Automatic snapshot system that checies of name of vessel | |
| CN112287808B (en) | Motion trajectory analysis warning method, device, system and storage medium | |
| CN113689646A (en) | Safety monitoring and warning system for travel dangerous area | |
| CN110817674B (en) | A method, device, equipment and storage medium for missing step detection of escalator | |
| CN113419075B (en) | Ship speed measuring method, system, device and medium based on binocular vision | |
| CN115461756A (en) | Device and method for decision support for artificial cognitive systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |