KR102355431B1 - AI based emergencies detection method and system - Google Patents

Abstract

Disclosed are an AI-based emergency detection method and system. The AI-based unexpected situation detection method includes the steps of: (a) acquiring an image reflecting traffic situation and environment information for a specific road point in real time through a camera installed at the specific road point; and (b) monitoring whether an unexpected situation has occurred by applying the image reflecting the traffic situation and environmental information to the learned deep learning-based emergency situation analysis model.

Description

AI-based emergency detection method and system {AI based　emergencies detection method and system}

The present invention relates to an AI-based unexpected situation detection method and system.

Due to the development of driving assistance devices and autonomous driving technology, drivers who use automobiles are increasingly less manipulated during the driving process, and automobiles are gradually becoming automated and unmanned.

Unmanned driving or autonomous driving technology is a technology in which a vehicle recognizes the driving environment by itself and drives to a target point without the driver's manipulation. It can be divided into a control device that gives a command, an actuator that performs a necessary operation according to a command, and the like.

In autonomous driving technology, mechanical automatic driving technology that controls steering, acceleration, deceleration, etc., which are driving-related devices of a car, has reached a certain level and is becoming more complete. Recognizing and judging technologies to respond to unexpected situations are relatively insufficient. To compensate for this, a technology for collecting dangerous situation information from road infrastructure and providing it to drivers is being developed.

(01) Republic of Korea Patent 10-2199252B1 (2020.12.30.)

The present invention is to provide an AI-based unexpected situation detection method and system.

In addition, the present invention enables the acquisition of high-quality learning data on the emergency situation based on the images collected according to the environment and traffic characteristics of the location installed on the road, and through this, AI capable of collecting and processing information on the emergency situation after accurate object recognition An object of the present invention is to provide a method and a system for detecting an emergency based on the situation.

In addition, the present invention is an AI-based system that can provide reliable emergency situation information to road users and drivers by improving the detection accuracy as time goes by through data learning using real-world traffic conditions and applying the results. An object of the present invention is to provide a method for detecting an unexpected situation and a system for the same.

According to one aspect of the present invention, an AI-based emergency detection method is provided.

According to an embodiment of the present invention, the method comprising: (a) acquiring an image in which traffic conditions and environmental information for a specific road point are reflected in real time through a camera installed at the specific road point; and (b) applying the image reflecting the traffic situation and environment information to the learned deep learning-based emergency situation analysis model to monitor whether an unexpected situation has occurred.

The abrupt situation may be at least one of a vehicle stop in a situation while driving at the specific road point, reverse driving of the vehicle, and occurrence of a pedestrian and a falling object.

prior to the step (a), designating common attribute information for the image based on traffic conditions and environment information for the image installed at each road point; designating object attribute information by analyzing the image to extract each object and reflecting characteristic information on the object; generating a training data set using the image to which the common attribute information is specified and the object to which the object attribute information is specified; The method may further include reinforcing learning of the deep learning-based contingency analysis model with images and objects matching the conditions in the training data set through condition specification according to a specific environment and a specific traffic situation.

The environment information includes surrounding environment information and spatial information, wherein the surrounding environment information is at least one of time information, illuminance information, and weather information at which an image was captured, and the spatial information includes area information at which the image was captured, and a road type. , at least one of a shooting distance and a height.

The traffic situation may be at least one of unexpected situation information, traffic accident related information, and a normal situation.

In the extracted object, common attribute information of an image including the object may be further designated.

According to another embodiment of the present invention, real-time images in which traffic conditions and environment information are reflected are respectively obtained from respective cameras installed at a plurality of road points, and a common property of each image by using the traffic condition and environment information specifying information; designating object attribute information by analyzing each image to extract an object, reflecting characteristic information on the object; generating a training data set using the image to which the common attribute information is specified and the object to which the object attribute information is specified; and selecting an image and object matching a specific environment and traffic conditions from among the training data set to reinforce-learning a deep learning-based emergency situation analysis model of a specific point may be provided.

According to another aspect of the present invention, a system and apparatus for AI-based unexpected situation detection are provided.

According to an embodiment of the present invention, a memory for storing at least one instruction; and a processor executing the instructions stored in the memory, wherein the instructions executed by the processor (a) use a camera installed at the specific road point to display an image reflecting the traffic situation and environment information for the specific road point in real time obtaining; and (b) applying the image reflecting the traffic situation and environmental information to the learned deep learning-based emergency situation analysis model to monitor whether an emergency has occurred. An AI-based emergency detection device may be provided.

According to another embodiment of the present invention, a plurality of cameras installed at a plurality of road points, respectively, for acquiring a real-time image in which traffic conditions and environment information of each road point are reflected; Designating common attribute information for each image by using traffic situation and environment information for each image obtained from the plurality of cameras, analyzing each image to extract an object, and then extracting an object a server for designating object attribute information by reflecting the information, and generating a training data set using an image to which the common attribute information is specified and an object to which the object attribute information is specified; and a plurality of sudden situation detection that is interlocked with the plurality of cameras, respectively, and selects images and objects that match the environment and traffic conditions of the installation point from among the training data sets to reinforce-learning a deep learning-based emergency situation analysis model of the installation point A system comprising an apparatus may be provided.

By providing an AI-based emergency detection method and system according to an embodiment of the present invention, it is possible to acquire high-quality learning data for an emergency situation based on images collected according to the environment and traffic characteristics of the installed location, and There is an advantage in that object recognition is possible, and through this, accurate unexpected situation information collection and processing is possible.

In addition, the present invention has the advantage of being able to improve detection accuracy as time goes by through data learning and application of the results using traffic conditions occurring in the actual field, and to provide reliable emergency situation information to road users and drivers. .

1 schematically shows a system according to an embodiment of the present invention;
2 is a diagram illustrating a photographed image according to an embodiment of the present invention.
3 is a flowchart illustrating a learning method of an apparatus for detecting a sudden situation according to an embodiment of the present invention.
4 is a diagram illustrating environment and situation information according to an embodiment of the present invention.
5 is a diagram illustrating an image and designated attribute information according to an embodiment of the present invention.
6 is a diagram illustrating an object type according to an embodiment of the present invention;
7 is a diagram illustrating object and attribute information according to an embodiment of the present invention.
8 is a flowchart illustrating a method for detecting a sudden situation according to an embodiment of the present invention.
9 is a block diagram schematically illustrating an internal configuration of a device for detecting a sudden situation according to an embodiment of the present invention.

As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are It should be construed that it may not include, or may further include additional components or steps. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram schematically illustrating a system according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating a captured image according to an embodiment of the present invention.

Referring to FIG. 1 , a system 100 according to an embodiment of the present invention is configured to include a plurality of cameras 110 , a plurality of unexpected situation detection devices 120 , and a server 130 .

The plurality of cameras 110 are installed in each road section, and are means for capturing a real-time image reflecting traffic conditions and environmental information of the corresponding road section.

The plurality of cameras 110 may have different installation heights according to their respective installation positions, and may also have different distances to a road section to be photographed.

For example, an image obtained by photographing a road condition at a specific point is, for example, as shown in FIG. 2 .

The contingency detection device 120 each has a deep learning-based contingency analysis model, and the deep learning-based contingency analysis model is strengthened based on a training data set suitable for the environment and traffic conditions of the analysis point (ie, installation point). can learn In addition, the sudden situation detection device 120 analyzes the video collected in real time through the camera 110 through the learned deep learning-based emergency analysis model to monitor whether the emergency situation occurs, and then notify the server 130. may be

The sudden situation detection device 120 may learn a deep learning-based emergency situation analysis model using only the images collected through the camera 110 interlocked with the sudden situation detection device 120 , and through the server 130 . After receiving the training data set, images and objects suitable for specific conditions (eg, environmental conditions and traffic conditions) of the installation point are selected and then a deep learning-based contingency analysis model can be reinforced using them. This will be described in more detail below with reference to the related drawings.

The server 130 is interlocked with a plurality of cameras 110 and the unexpected situation detection device 120, respectively, and the emergency detection device 120 supports learning a deep learning-based emergency situation analysis model to match the conditions of the installation point. is a means to

For example, after collecting images from a plurality of cameras 110 installed in various regions, the server 130 may designate common attribute information using traffic situation and environment information of each image. In addition, the server 130 may analyze each image to extract each object and then designate object attribute information by reflecting characteristic information on each object. Next, the server 130 may generate a training data set using an image to which common attribute information is specified and an object to which object attribute information is specified, and transmit it to each emergency detection device 120 .

In addition, the server 130 may receive an event according to the occurrence of the unexpected situation from the emergency detection device 120 that controls each specific point.

3 is a flowchart illustrating a learning method of an apparatus for detecting a sudden situation according to an embodiment of the present invention, FIG. 4 is a diagram illustrating environment and traffic situation information according to an embodiment of the present invention, and FIG. 5 is the present invention is a diagram illustrating an image and specified attribute information according to an embodiment of the present invention, FIG. 6 is a diagram illustrating an object type according to an embodiment of the present invention, and FIG. 7 is an object and attribute information according to an embodiment of the present invention It is a diagram illustrating information.

In step 310 , the sudden situation detection device 120 acquires an image reflecting the traffic situation and environment information of the installation point through the camera 110 .

In step 315, the abrupt situation detection device 120 designates common attribute information for the corresponding image based on the obtained traffic situation and environment information of each image.

Here, the common attribute information includes environment information and traffic situation information as shown in FIG. 4 , but the environment information may include surrounding environment information and spatial information.

That is, the surrounding environment information includes information on the time the image was taken, illuminance information (sunrise, daytime, sunset, night, etc.) at the time of capturing the image, and weather (climate) information (sunny, rain, snow, fog, ice, etc.) It may include at least one. In addition, spatial information includes at least one of regional information (metropolitan area, Chungcheong area, Gyeongsang area, Honam area, Gangwon area, etc.), shooting distance, shooting height, and road type (highway, general road, underground road, etc.) for a specific point where the image was shot. may include

In addition, the traffic situation information may include information related to an unexpected situation or a traffic accident, and a normal situation. The sudden situation may be a stop, reverse running, a pedestrian, an obstacle, a falling object, and the like. Also, the traffic accident-related information may include a single accident due to a collision or collision, a vehicle-to-vehicle accident such as overturning or overturning, and a vehicle-to-person accident including a fire.

In this way, the sudden situation detection device 120 determines the environmental factors that differentially affect the general commercial image for learning, such as the surrounding environmental conditions of the image taken at the installation point, shooting time, weather, illuminance, etc., as attribute information of the image. It can be specified in detail so that it can be used for learning.

5 shows an example in which common attribute information is designated in a photographed image.

In step 320, the sudden situation detection device 120 analyzes each image to extract an object, and then designates the characteristic information of the object as object attribute information. For example, based on a deep learning model such as a convolutional neural network (CNN), the abrupt situation detection device 120 may extract each object from the image. In this case, an object type to be extracted (refer to FIG. 6 ) may be defined in advance, and objects according to the predefined object type may be extracted.

That is, since the image is a photograph of a road, road facilities such as lanes are also photographed, and depending on the photographing location, tunnel facilities and the like may be photographed together. In an embodiment of the present invention, instead of extracting all objects from an image, object types necessary for detecting unexpected situations such as automobiles (eg, passenger cars, buses, trucks, etc.), people, animals, facilities, falling objects, temporary facilities, etc. After predefined object type, only objects suitable for the predefined object type can be extracted.

After each object is extracted, it can be designated as object property information by reflecting object characteristic information.

For example, it is assumed that the extracted object is "car". In correspondence with the automobile object, object attribute information can be specified by subdividing the object into "car", "bus", "lorry", "two-wheeled vehicle", and the like. In addition, the accident type for the car object may also be designated as object attribute information. The accident type may be, for example, front, rear, side (left), side (right), bottom, rear, and the like.

In addition, when the extracted object is a “person”, age attributes such as the elderly, adults, children, etc. may be stored so that the age can be known along with attribute information that subdivides the object into gender into men and women.

Also, it is assumed that the extracted object is an "animal". The properties of objects are subdivided into “live animals” and “dead animals” depending on whether they are alive or dead, and can be subdivided into “dogs”, “cats,” “cows”, and “elk” to specify the type of object. can

Also, when the extracted object is a "falling object", the type of the falling object (eg, box, tire, fruit, etc.) may be designated as attribute information.

In addition, when the extracted object is a "facility", the type of the corresponding facility (eg, an application mark, a sign, a median strip, a street tree, etc.) may be designated.

Also, when the extracted object is a "temporary facility", an object type such as "banner", "construction guide", etc. may be designated as an attribute.

In this way, the sudden situation detection device 120 may extract each object through the deep learning model and then designate a unique object property according to the object. In this way, object attribute information may be designated for each extracted object, and common attribute information designated corresponding to the corresponding image may be identically designated to the corresponding object.

7 shows an example in which common attribute information and object attribute information are designated for each extracted object. That is, the object may be designated as attribute information in addition to the object attribute information, as well as common attribute information for an image from which the object is extracted.

In step 325, the sudden situation detection device 120 generates a training data set using an image to which common attribute information is specified and an object to which common attribute information and object attribute information are specified.

The training data set may be generated by including the entire image of the road point and information about each object extracted from the image, respectively.

In step 330, the emergency detection device 120 learns a deep learning-based emergency analysis model using the training data set.

In this case, the sudden situation detection device 120 may reinforce-learning a deep learning-based emergency situation analysis model to be suitable for a specific situation or a specific environment by using only some data matching a specific condition among the training data sets.

In addition, in FIG. 3 , it is assumed that the training data set is generated for the image captured by the sudden situation detection device 120 only by one camera, but the training data set is the image captured by the plurality of cameras by the server. may be created based on

That is, the server 130 may designate common attribute information after acquiring images reflecting respective traffic conditions or environment information through cameras installed at different road points. Also, after extracting an object from each image, object property information may be specified. In this way, a training data set may be generated based on images obtained from a plurality of road points. The server 130 generates a training data set based on images acquired from a plurality of road points and then applies it to each deep learning-based contingency analysis model, rather than applying it to each installation point condition (environmental conditions, traffic conditions, etc.) It can also support reinforcement learning of a deep learning-based contingency analysis model suitable for a given condition by selecting some data sets suitable for

8 is a flowchart illustrating a method for detecting a sudden situation according to an embodiment of the present invention.

In step 810 , the sudden situation detection device 120 acquires a real-time image through the camera 110 .

In step 815, the sudden situation detection device 120 performs object recognition by applying the real-time acquired image to the learned deep learning-based emergency situation analysis model.

In step 820, the sudden situation detection device 120 determines whether there is a sudden situation by detecting the movement characteristics or motion characteristics of the object based on the object recognition result.

For example, when a vehicle is recognized as a result of object recognition, the sudden situation detecting device 120 may generate and transmit the unexpected situation information to the server 130 when the vehicle is stopped on the road according to whether the vehicle is moving.

Also, the sudden situation detecting device 120 may determine whether the unexpected situation occurs by determining whether the object recognition result is a vehicle and the vehicle is traveling in the reverse direction.

As another example, when a person or an animal is located on the road as a result of object recognition, the emergency detection device 120 may determine whether an unexpected situation occurs according to the location of the person or the appearance of the animal and transmit it to the server 130 . .

As another example, when a falling object is located on the road as a result of object recognition, the sudden situation detection device 120 may determine whether an unexpected situation occurs according to the occurrence of the falling object and transmit it to the server 130 .

Of course, according to the implementation method, the unexpected situation detecting device 120 may broadcast the event of occurrence of an unexpected situation in the corresponding road section to a vehicle entering the corresponding road section.

9 is a block diagram schematically illustrating an internal configuration of an apparatus for detecting a sudden situation according to an embodiment of the present invention.

Referring to FIG. 9 , the device for detecting a sudden situation according to an embodiment of the present invention 120 includes a communication unit 910 , a memory 920 , and a processor 930 . The sudden situation detection device 120 may include a camera 110 , and may be interlocked with the camera 110 installed at a specific point.

The communication unit 910 is a means for transmitting and receiving data with other devices (eg, the camera 110 , the server 130 , etc.) through a communication network.

The memory 920 is a means for storing instructions necessary to perform the AI-based unexpected situation detection method according to an embodiment of the present invention.

The processor 930 is a means for controlling internal components (eg, the communication unit 910 , the memory 920 , etc.) of the unexpected situation detection device 120 according to an embodiment of the present invention.

In addition, the processor 930 controls to execute the instructions stored in the memory 920 , and the instructions executed by the processor 730 may execute respective steps described with reference to FIGS. 3 to 8 .

The apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - Includes magneto-optical media and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been focused on the embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

100: system
110: camera
120: sudden situation detection device
130: server

Claims (12)

(a) acquiring an image in which traffic conditions and environmental information for the specific road point are reflected in real time through a camera installed at the specific road point; and
(b) applying the image reflecting the traffic situation and environmental information to the learned deep learning-based emergency situation analysis model to monitor whether an emergency situation occurs,
Before step (a),
designating common attribute information for the image based on traffic conditions and environment information for the image installed at each road point;
extracting each object by analyzing the image, and then designating object attribute information by reflecting characteristic information on the extracted object;
generating a training data set using the image to which the common attribute information is specified and the object to which the object attribute information is specified;
AI-based emergency situation comprising the step of reinforcing learning of the deep learning-based emergency situation analysis model with images and objects matching the conditions in the training data set through condition specification according to a specific environment and a specific traffic situation detection method.
According to claim 1,
The emergency situation is
AI-based unexpected situation detection method, characterized in that at least one of vehicle stopping, vehicle reverse running, pedestrians and falling objects in a driving situation at the specific road point.
delete According to claim 1,
The environment information includes surrounding environment information and space information,
The surrounding environment information is at least one of time information at which an image was taken, illuminance information, and weather information,
The spatial information is an AI-based unexpected situation detection method, characterized in that at least one of information about the area where the image was captured, a road type, a shooting distance, and a height.
According to claim 1,
The traffic situation is an AI-based unexpected situation detection method, characterized in that at least one of emergency situation information, traffic accident related information, and a normal situation.
According to claim 1,
The extracted object is an AI-based unexpected situation detection method, characterized in that the common attribute information of the image including the object is further designated.
A computer-readable recording medium having recorded thereon a program code for performing the method according to any one of claims 1 to 2 and 4 to 6 .
a memory storing at least one instruction; and
A processor for executing instructions stored in the memory,
The instructions executed by the processor are
(a) acquiring an image in which traffic conditions and environmental information for the specific road point are reflected in real time through a camera installed at the specific road point; and
(b) applying the image reflecting the traffic situation and environment information to the learned deep learning-based emergency situation analysis model to monitor whether an emergency situation occurs,
Before step (a),
designating common attribute information for the image based on traffic conditions and environment information for the image installed at each road point;
designating object attribute information by analyzing the image to extract each object and reflecting characteristic information on the object;
generating a training data set using the image to which the common attribute information is specified and the object to which the object attribute information is specified; and
AI-based emergency situation, characterized in that by performing the step of reinforcement learning the deep learning-based emergency situation analysis model with images and objects matching the conditions in the training data set through condition specification according to a specific environment and a specific traffic situation detection device.
9. The method of claim 8,
The emergency situation is
AI-based unexpected situation detection device, characterized in that at least one of vehicle stopping, vehicle reverse running, pedestrians and falling objects at the specific road point.
delete delete a plurality of cameras installed at a plurality of road points, respectively, and acquiring real-time images in which traffic conditions and environmental information of each road point are reflected;
Designating common attribute information for each image by using traffic situation and environment information for each image obtained from the plurality of cameras, analyzing each image to extract an object, and then extracting an object a server for designating object attribute information by reflecting the information, and generating a training data set using an image to which the common attribute information is specified and an object to which the object attribute information is specified; and
Each of the plurality of cameras is interlocked, and images and objects matching the environment and traffic conditions of the installation point are selected from the training data set to reinforce-learning a deep learning-based contingency analysis model of the installation point, and the reinforcement learning is completed. A system including a plurality of emergency situation detection devices that monitor whether an emergency has occurred based on an image reflecting real-time traffic situation and environmental information through a deep learning-based emergency analysis model.

Images