KR102661363B1 - Device and method for monitoring a berthing - Google Patents

Abstract

The present invention relates to a berthing monitoring method performed by computing means, and includes the steps of acquiring a port image including the sea and a plurality of ships using a camera installed in the port according to an embodiment to capture images, an input image, and Using an artificial neural network learned using a learning set that labels pixels corresponding to objects including the sea, ships, and features included in the input image with class values indicating the sea, ships, and features, respectively, Generating a segmentation image for the objects from a harbor image, tracking the plurality of ships based on first pixels assigned a class value indicating a ship in the segmentation image, and creating a representative representing each of the plurality of ships. Obtaining tracking information including location information of points, determining a target vessel to be guided to berth among the tracked vessels based on the tracking information, and determining a distance between the bow of the target vessel and the quay wall based on the segmentation image. Disclosed is a berthing monitoring method including the step of acquiring berthing guide information including the bow distance, which is the distance, and the stern distance, which is the distance between the stern of the target ship and the quay wall.

Description

Eyepiece monitoring device and method {DEVICE AND METHOD FOR MONITORING A BERTHING}

The present invention relates to an eyepiece monitoring device and method, and specifically to an eyepiece monitoring device and method that performs image-based eyepiece monitoring.

Recently, many accidents have occurred in the operation of ships and berthing and unberthing in ports. Many accidents occur mainly due to the inability to accurately check the situation around the ship or in the port through video when docking.

Accordingly, in the past, berthing is supported using various types of sensors such as ECDIS and radar. However, in the case of ECDIS, there are limitations due to the inaccuracy of GPS, the update cycle of AIS, and unregistered AIS moving objects, and in the case of radar, there are limitations in the non-navigable area. There are limitations due to presence and noise.

Therefore, there is a need to develop technology for monitoring the situation around the ship or in the port through video when docking.

The problem to be solved by this specification is to provide a berthing monitoring device and method for monitoring the vicinity and port of a ship.

Another problem to be solved by this specification is to provide a berthing monitoring device and method that determines a vessel that is the target of monitoring and calculates and outputs information about the monitoring target.

Another problem to be solved by this specification is to provide an eyepiece monitoring device and method using image segmentation.

The problem to be solved by the present invention is not limited to the above-mentioned problems, and problems not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings. .

According to one aspect of the present specification, in a berthing monitoring method performed by computing means, the steps of acquiring a port image including the sea and a plurality of ships using a camera installed in the port to capture images, an input image and Using an artificial neural network learned using a learning set that labels pixels corresponding to objects including the sea, ships, and features included in the input image with class values indicating the sea, ships, and features, respectively, Generating a segmentation image for the objects from a harbor image, tracking the plurality of ships based on first pixels assigned a class value indicating a ship in the segmentation image, and creating a representative representing each of the plurality of ships. Obtaining tracking information including location information of points, determining a target vessel to be guided to berth among the tracked vessels based on the tracking information, and determining the distance between the bow and quay of the target vessel based on the segmentation image. A berthing monitoring method may be provided including the step of acquiring berthing guide information including the bow distance, which is the distance, and the stern distance, which is the distance between the stern of the target ship and the quay wall.

In addition, according to another aspect of the present specification, a camera is installed in a port to capture images, a port image including the sea and a ship captured by the camera is acquired, and an input image and the sea and a ship included in the input image are used. And segmentation of the objects from the port image using an artificial neural network learned using a learning set that labels pixels corresponding to objects containing features with class values indicating the sea, ship, and feature, respectively. Generate an image, track the plurality of ships based on first pixels assigned a class value indicating the ship of the segmentation image, and generate tracking information including location information of representative points representing each of the plurality of ships. Obtaining, based on the tracking information, determines a target ship guided to berth among the tracked ships, and based on the segmentation image, the bow distance, which is the distance between the bow and quay wall of the target ship, and the stern and quay wall of the target ship A berthing monitoring device may be provided, including a control module that acquires berthing guide information including the stern distance, which is the distance between the sterns, and a communication module that transmits berthing guide information of the target ship to a remotely located terminal.

The solution to the problem of this specification is not limited to the above-mentioned solution, and the solution not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings. You will be able to.

According to an embodiment of the present specification, monitoring of the ship and the port can be performed by calculating berthing guide information for monitoring the vicinity of the ship and the port.

According to an embodiment of the present specification, berthing monitoring can be efficiently performed by determining the vessel that is the target of monitoring and calculating and outputting information about the determined monitoring target.

According to an embodiment of the present specification, a ship can be recognized using image segmentation, and a monitoring target can be determined based on this.

The effects of the present invention are not limited to the effects described above, and effects not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings.

1 is a diagram of image-based monitoring according to an embodiment.
Figure 2 is a diagram of an ocular monitoring device according to an embodiment.
3 and 4 are diagrams of an embodiment of an eyepiece monitoring device according to an embodiment.
Figure 5 is a diagram of viewing angle and depth of field according to one embodiment.
6 and 7 are diagrams of the installation location of the sensor module according to one embodiment.
Figure 8 is a diagram related to image analysis according to one embodiment.
9 to 11 are diagrams of object recognition steps according to an embodiment.
12 and 13 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.
14 and 15 are diagrams related to estimating location/movement information of an object according to an embodiment.
Figure 16 is a flowchart for determining a target vessel according to one embodiment.
Figure 17 is a diagram of an example of ship tracking according to an embodiment.
Figure 18 is a diagram of an example of target vessel determination based on location information according to an embodiment.
Figure 19 is a diagram of an example of target ship determination based on movement information according to an embodiment.
Figure 20 is a diagram of an example of target vessel determination based on navigation information according to an embodiment.
21 is a diagram of an example of target ship determination based on user input information according to an embodiment.
Figure 22 is a diagram related to acquiring berthing guide information of a target ship according to an embodiment.
Figure 23 is a flowchart of monitoring information output according to one embodiment.
Figure 24 is a diagram of an example of monitoring information output according to an embodiment.
25 to 27 are diagrams of another example of monitoring information output according to an embodiment.
Figure 28 is a diagram related to image-based monitoring based on a plurality of images according to an embodiment.
29 and 30 are diagrams of another example of monitoring information output according to an embodiment.
31 and 32 are diagrams of another example of monitoring information output according to an embodiment.
Figures 33 and 34 are diagrams related to viewpoint conversion according to one embodiment.
Figure 35 is a flowchart of berthing monitoring according to one embodiment.

The embodiments described in this specification are intended to clearly explain the spirit of the present invention to those skilled in the art to which the present invention pertains. Therefore, the present invention is not limited to the embodiments described in this specification, and the present invention The scope of should be construed to include modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification are general terms that are currently widely used as much as possible in consideration of their function in the present invention, but this may vary depending on the intention, custom, or the emergence of new technology of a person skilled in the art in the technical field to which the present invention pertains. You can. However, if a specific term is defined and used in an arbitrary sense, the meaning of the term will be described separately. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the term and the overall content of this specification, not just the name of the term.

The drawings attached to this specification are intended to easily explain the present invention, and the shapes shown in the drawings may be exaggerated as necessary to aid understanding of the present invention, so the present invention is not limited by the drawings.

In this specification, if it is determined that a detailed description of a known configuration or function related to the present invention may obscure the gist of the present invention, the detailed description thereof will be omitted as necessary.

The term 'port image' used in this specification can be understood as an image related to a port. For example, a port image includes an image captured through a camera installed in the port, an image containing at least part of the port, etc. can do.

According to one aspect of the present specification, a berthing monitoring method performed by computing means includes the steps of: acquiring a port image including the sea and a plurality of ships using a camera installed in the port to capture images; An artificial neural network learned using a learning set that labels pixels corresponding to input images and objects including the sea, ships, and features included in the input image with class values indicating the sea, ships, and features, respectively. generating a segmentation image for the objects from the harbor image using the port image; Obtaining tracking information including location information of representative points representing each of the plurality of ships by tracking the plurality of ships based on first pixels assigned class values indicating the ships of the segmentation image; determining a target vessel to be guided to berth among the tracked vessels based on the tracking information; and acquiring berthing guide information including a bow distance, which is the distance between the bow and the quay wall of the target ship, and a stern distance, which is the distance between the stern and the quay wall of the target ship, based on the segmentation image; A berthing monitoring method including a may be provided.

Here, the representative point may be one point on the segmentation area of each of the plurality of ships in the segmentation image.

Here, the representative point may be a point located at the center of the segmentation area of each of the plurality of ships.

Here, the step of acquiring the tracking information includes acquiring the distance between the representative point and the quay wall, and the target ship may be determined based on the distance between the representative point and the quay wall.

Here, the target ship may be determined to be a ship whose distance between the representative point and the quay wall among the plurality of ships is less than a preset value.

Here, the target ship may be determined to be a ship whose representative point is located in a preset area among the plurality of ships.

Here, the step of acquiring the tracking information includes acquiring movement information of the representative point based on location information of the representative point, and the target ship may be determined based on the movement information of the representative point.

Here, the target ship may be determined based on the size of the approach speed of the representative point among the plurality of ships to the quay wall.

Here, the step of acquiring the tracking information includes acquiring at least one of VTS information and AIS information of the plurality of ships, and the target ship may be determined based on at least one of the VTS information and AIS information. .

Here, the target ship may be determined based on a destination according to at least one of the VTS information and AIS information among the plurality of ships.

Here, the step of obtaining the tracking information includes obtaining user input information about the plurality of ships from a user terminal, and the target ship may be determined based on the user input information about the plurality of ships.

Here, the target ship may be determined to be a ship selected among the plurality of ships according to the user input information.

Here, when there are multiple ships that meet the conditions used to determine the target ship, the target ship may be determined to be one ship that meets the conditions later.

Here, the artificial neural network applies the sea, tugboats, ships excluding tugboats, and features to pixels corresponding to the input image and objects including the sea, tugboats, ships excluding tugboats, and features included in the input image, respectively. It can be learned using a learning set that labels the class values it indicates.

Here, the step of acquiring the berthing guide information includes extracting a pair of points corresponding to both ends of the bottom of the target ship in contact with the sea surface, and determining the bow distance and the stern distance based on the pair of points. It may include an acquisition step.

Here, the step of acquiring the berthing guide information is the bow speed, which is the speed at which the bow of the target ship approaches the quay wall, and the speed at which the stern of the target ship approaches the quay wall, based on the bow distance and the stern distance. It may include obtaining stern speed.

Here, the step of acquiring the berthing guide information may include obtaining the distance between the target ship and the other ship based on the segmentation image.

Here, the step of acquiring the berthing guide information may include acquiring the relative speed between the target ship and the other ship based on the distance between the target ship and the other ship.

Here, the step of acquiring the berthing guide information may include obtaining information related to the risk of collision of the ship with another ship or the quay wall.

Here, the berthing monitoring method includes outputting the berthing guide information together with the harbor image; It may further include.

Here, the harbor image may include a panoramic image in which a plurality of harbor images are registered.

Additionally, a recording medium recording a program for executing the above-described method may be provided.

Also, according to an aspect of the present specification, a camera installed in a port to capture images; Acquire a port image including the sea and a ship captured by the camera, and assign the sea, a ship, and a feature to pixels corresponding to an input image and objects including the sea, a ship, and a feature included in the input image, respectively. A segmentation image for the objects is generated from the harbor image using an artificial neural network learned using a learning set labeled with indicating class values, and a first pixel is assigned a class value indicating a ship in the segmentation image. Tracking the plurality of ships based on the tracking information to obtain tracking information including location information of representative points representing each of the plurality of ships, and determining a target ship to be guided to berth among the tracked ships based on the tracking information. and a control module that obtains berthing guide information including a bow distance, which is the distance between the bow and the quay wall (pier) of the target ship, and a stern distance, which is the distance between the stern and the quay wall of the target ship, based on the segmentation image; and a communication module that transmits berthing guide information of the target vessel to a remotely located terminal. A berthing monitoring device including a may be provided.

Hereinafter, a piercing monitoring device and method according to an embodiment will be described.

In this specification, monitoring refers to understanding or recognizing the surrounding situation. It not only detects a detection target such as a certain area or a specific object using various sensors and provides the detection result to the user, but also performs calculations based on the detection result. It should be interpreted broadly to include providing additional information.

Image-based monitoring may mean identifying or recognizing surrounding situations based on images. For example, monitoring may mean acquiring images around the ship when operating a ship, recognizing other ships or obstacles from these images, or obtaining information to calculate berthing guide information when a ship docks or unberths.

Berthing guide information refers to the surrounding environment required for berthing, such as recognition of other vessels or obstacles, understanding the port situation, whether the berth is accessible, distance and speed from the quay wall, distance and speed from other vessels, and identification of the presence of obstacles in the navigation route. It can mean information about. This specification mainly describes monitoring in cases where berthing is performed at ships and ports, but is not limited to this and may also be applied to driving of vehicles, operation of aircraft, etc.

1 is a diagram of image-based monitoring according to an embodiment. Referring to FIG. 1, image-based monitoring may include an image acquisition step (S10) and an image analysis step (S20).

The image acquisition step (S10) may refer to a step in which the device 10 acquires an image. Here, the types of images may vary, such as RGB images, IR images, depth images, lidar images, radar images, etc., and are not limited. Additionally, not only two-dimensional images but also three-dimensional images are possible.

The image analysis step (S20) may refer to the step of obtaining analysis results based on the image. As an example, the image analysis step (S20) may include calculating eyepiece guide information through the image. Alternatively, the image analysis step (S20) may refer to a step of analyzing the characteristics of objects included in the image. Alternatively, the image analysis step (S20) may include determining the situation represented by the image. or,

Details of the image acquisition step (S10) and the image analysis step (S20) will be described later. Hereinafter, information acquired through the image acquisition step (S10) or the image analysis step (S20) is referred to as monitoring information.

Figure 2 is a diagram of an ocular monitoring device according to an embodiment. Referring to FIG. 2 , the monitoring device 10 may include a sensor module 100, a control module 200, and a communication module 300.

The sensor module 100 can sense information about the ship, its surroundings, and the port. The sensor module 100 may include an automatic identification system (AIS), an image generation unit, a position measurement unit, an attitude measurement unit, a casing, etc.

The image generation unit may generate an image. The image generation unit may include a camera, lidar, radar, ultrasonic detector, etc. Examples of cameras include, but are not limited to, monocular cameras, binocular cameras, visible light cameras, IR cameras, and depth cameras.

The position measurement unit can measure the position of components included in the sensor module, such as a sensor module or an image generation unit. As an example, the location measurement unit may be a Global Positioning System (GPS). In particular, Real-Time Kinematic GPS (RTK-GPS) may be used to improve the accuracy of location measurement.

The location measurement unit may acquire location information at predetermined time intervals. Here, the time interval may vary depending on the installation location of the sensor module. For example, when a sensor module is installed on a moving object such as a ship, the position measurement unit can acquire position information at short time intervals. On the other hand, when the sensor module is installed on a fixture such as a port, the position measurement unit can acquire position information at long time intervals. The time interval for acquiring location information of the location measurement unit may be changed.

The posture measurement unit can measure the posture of components included in the sensor module, such as a sensor module or an image generation unit. As an example, the attitude measurement unit may be an inertial measurement unit (IMU).

The posture measurement unit may acquire posture information at predetermined time intervals. Here, the time interval may vary depending on the installation location of the sensor module. For example, when a sensor module is installed on a moving object such as a ship, the attitude measurement unit can acquire attitude information at short time intervals. On the other hand, when the sensor module is installed on a fixture such as a port, the attitude measurement unit can acquire attitude information at long time intervals. The time interval for acquiring posture information of the posture measurement unit may be changed.

The casing can protect sensor modules such as the image generation unit, position measurement unit, and attitude measurement unit.

At least one of an image generation unit, a position measurement unit, and an attitude measurement unit may exist inside the casing. The casing can prevent equipment such as the image generation unit present inside from being corroded by salt water. Alternatively, the casing can protect the equipment present inside by preventing or alleviating impact on it.

A cavity may be formed inside the casing to include an image generating unit, etc. therein. For example, the casing may have a rectangular parallelepiped shape with an empty interior, but the casing is not limited to this and may be provided in various shapes in which an image generating unit, etc. can be placed therein.

When the image generating unit is disposed inside the casing, an opening and closing opening may be formed in one area of the casing to ensure visibility of the image generating unit, or one area of the casing may be formed of a transparent material such as glass. The image generating unit can capture images of the ship's surroundings and the port through the opening or transparent area.

The casing may be made of a strong material to protect the image generation unit, etc. from external shock. Alternatively, the casing may be made of a material such as a seawater alloy to prevent corrosion due to salt.

The casing may include equipment for removing foreign matter from the image generating unit. For example, foreign substances adhering to the surface of the image generating unit can be physically removed using a wiper included in the casing. Here, the wiper may be provided in a linear or plate shape with the same or similar curvature as the surface so that it can come into close contact with the surface from which foreign substances are to be removed. As another example, foreign substances can be removed by applying water or washer fluid through a liquid spray included in the casing, or the foreign substances can be physically removed using a wiper after application.

Debris removal equipment can be operated manually, but can also be operated automatically. For example, foreign matter removal equipment may operate at predetermined time intervals. Alternatively, foreign matter removal equipment can be operated using a sensor that detects whether foreign matter is attached to the image generating unit. Alternatively, the image generation unit may use an image captured to determine whether a foreign substance is captured in the image, and then, if it is determined that a foreign substance exists, the foreign substance removal equipment may be operated. Here, whether a foreign substance is captured in the image may be determined through an artificial neural network.

One sensor module 100 may include a plurality of identical equipment, such as two or more identical cameras.

The control module 200 may perform image analysis. In addition, operations of receiving various data through the sensor module 100, operations of outputting various outputs through an output device, operations of storing various data in memory or obtaining various data from memory, etc. are performed by the control module 200. This can be done by control. Hereinafter, various operations or steps disclosed in the embodiments of the present specification may be interpreted as being performed by the control module 200 or under the control of the control module 200, unless otherwise specified.

Examples of the control module 200 include a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), state machine, and on-demand There may be semiconductors (Application Specific Integrated Circuit, ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.

The communication module 300 may transmit information from the device 10 to the outside or receive information from the outside. The communication module 300 can perform wired or wireless communication. The communication module 300 can perform bi-directional or unidirectional communication. For example, the device 10 may transmit information to an external output device through the communication module 300 and output the control results performed by the control module 200 through the external output device. Additionally, the communication module 300 may receive VTS information or Costal Intelligent Transport System (CITS) information related to the ship from a Vessel Traffic Service (VTS) that controls the ship.

The sensor module 100, control module 200, and communication module 300 may include a control unit. The control unit can process and calculate various information within the module and control other components that make up the module. The control unit may be physically provided in the form of an electronic circuit that processes electrical signals. A module may physically include only a single control unit, but alternatively, it may include a plurality of control units. As an example, the control unit may be one or more processors mounted on one computing device. As another example, the control unit may be provided by processors mounted on physically separated servers and terminals and collaborating through communication. Examples of control units include Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), state machine, and application specific semiconductor (Application Specific). There may be Integrated Circuit (ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.

The sensor module 100, control module 200, and communication module 300 may include a communication unit. The modules can transmit and receive information through the communication unit. For example, the sensor module 100 may transmit information obtained from the outside through the communication unit, and the control module 200 may receive information transmitted by the sensor module 100 through the communication unit. The communication unit may perform wired or wireless communication. The communication unit can perform bi-directional or unidirectional communication.

The sensor module 100, control module 200, and communication module 300 may include memory. The memory may store various processing programs, parameters for processing the programs, or data resulting from such processing. For example, the memory may store data required for learning and/or inference, artificial neural networks that are in progress or have been trained, etc. Memory refers to non-volatile semiconductor memory, hard disk, flash memory, RAM (Random Access Memory), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or other tangible non-volatile recording media. It can be implemented as follows.

The monitoring device 10 may include a plurality of identical modules, such as two or more sensor modules 100 . For example, one device 10 may include two sensor modules 100, and each sensor module 100 may include two cameras.

3 and 4 are diagrams of an embodiment of an eyepiece monitoring device according to an embodiment.

Referring to FIG. 3, the monitoring device 10 may include a sensor module 100 and a control module 200. The sensor module 100 may generate an image through the camera 130 and transmit the image to the control module 200 through the communication unit 110. Additionally, the control unit 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint conversion, which will be described later. The control module 200 may receive an image from the sensor module 100 through the communication unit 210 and perform image analysis, such as location/movement information estimation and image matching, which will be described later, through the control unit 220. Additionally, the control module 200 may transmit analysis results such as location/movement information and matched images to the cloud server through the communication unit 210. The cloud server can transmit the analysis results received from the control module 200 to a user terminal such as a smartphone, tablet, or PC, or receive instructions from the user terminal.

Referring to FIG. 4 , the monitoring device 10 may include a sensor module 100. The sensor module 100 may generate an image through the camera 130 and transmit the image to a cloud server through the communication unit 110. Additionally, the control unit 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint conversion, which will be described later. The cloud server may receive an image from the sensor module 100 and perform image analysis, such as location/movement information estimation and image matching, which will be described later. Additionally, the cloud server can transmit the results of image analysis to a user terminal such as a smartphone, tablet, or PC, or receive instructions from the user terminal.

The device 10 shown in FIGS. 2 to 4 is merely an example, and the configuration of the device 10 is not limited thereto.

As an example, device 10 may include an output module 400. The output module 400 may output the results of operations performed by the control module 200, etc. For example, the output module 400 may output analysis results. The output module 400 may be, for example, a display, a speaker, a signal output circuit, etc., but is not limited thereto. In this case, in addition to transmitting the information to an external output device such as a user terminal and the external output device outputting the information, the information may also be output through the output module 400.

As another example, device 10 may not include a sensor module. In this case, the control module 200 may receive information from an external sensor device and perform image-based monitoring operations, such as performing image analysis. For example, the control module 200 may perform image analysis by receiving information from AIS, cameras, lidar, radar, etc. already installed on a ship or port.

Additionally, the steps performed by each component in FIGS. 2 to 4 are not necessarily performed by the corresponding component and may be performed by another component. For example, in FIG. 3 above, the control unit 120 of the sensor module 100 is described as performing viewpoint conversion, but the control unit 220 of the control module 200 or the cloud server may also perform viewpoint conversion. .

Below, we will look at the monitoring device 10 and method in more detail.

Image acquisition for image-based monitoring may be performed through the sensor module 100. For example, an image may be acquired through an image generating unit included in the sensor module 100. Alternatively, the image may be acquired from an external sensor device as described above. Images for ship and port monitoring will typically include the sea, ships, buoys, obstacles, terrain, ports, sky, buildings, etc. Hereinafter, analysis and monitoring of images obtained through a visible light camera will be explained, but the method is not limited to this.

The field of view and depth of field may vary depending on the image generation unit. Figure 5 is a diagram of viewing angle and depth of field according to one embodiment. Referring to FIG. 5, the field of view (FOV) can mean the extent to which the image includes left and right or up and down, and is generally expressed as an angle (degree). A larger viewing angle may mean creating an image that includes a larger area left and right, or creating an image that includes a larger area up and down. Depth of field may mean the distance range at which an image is recognized as being in focus, and a deep depth of field may mean that the distance range at which the image is perceived as being in focus is wide. Referring to FIG. 5, depending on the depth of field (DOF), the image may include an area recognized as in focus (A1) and another area (A2). Hereinafter, the area included in the image is referred to as the imaging area (A1 + A2), and the area recognized as being in focus is referred to as the effective area (A1). Image analysis and monitoring can be performed based on the effective area, but the entire imaging area is referred to as the effective area (A1). Since it may be performed based on or based on a portion of the imaging area, the area used to perform image analysis and monitoring is called a monitoring area.

An example of a camera with a large viewing angle and shallow depth of field is a wide-angle camera. Examples of cameras with a small viewing angle and deep depth of field include high-magnification cameras and zoom cameras.

The sensor module 100 can be installed without limitation in its location or posture, such as on a lighting tower, crane, or ship within a port, and there is no limitation in its number. However, the installation location or number may vary depending on the characteristics such as the type and performance of the sensor module 100. For example, when the sensor module 100 is a camera, the sensor module 100 may be installed at an altitude of 15 m or more above the water for efficient monitoring, or a plurality of sensor modules 100 may be installed to have different imaging areas. Additionally, the position and posture of the sensor module 100 may be adjusted manually or automatically during or after installation.

6 and 7 are diagrams of the installation location of the sensor module 100 according to one embodiment. Referring to FIGS. 6 and 7 , the sensor module 100 may be installed in a fixed location such as a port or land, or may be installed on a moving object such as a ship. Here, when the sensor module 100 is installed on a ship, it can be installed on a ship subject to monitoring (hereinafter referred to as “target ship”) as shown in FIG. 7, and can be installed at or near the berth of the target ship as shown in FIG. 6. It may be installed on a third vessel that is not subject to monitoring, such as an auxiliary tugboat. In addition, the sensor module 100 can be installed on a drone, etc. to monitor the target ship.

Other components of the monitoring device 10 may be installed together with the sensor module 100 or in a separate location.

As described above, image analysis for image-based monitoring may include obtaining object characteristics. Examples of objects may include ships, ports, buoys, the sea, terrain, sky, buildings, people, animals, fire, smoke, etc. Examples of object characteristics may include the type of object, the location of the object, the distance to the object, and the absolute and relative speed and velocity of the object. For another example,

Image analysis for image-based monitoring may include recognizing/determining the surrounding situation. For example, image analysis may determine whether a fire situation has occurred from an image of a fire in a port, or determine whether an intruder has entered the port from an image captured of a person entering the port at an unscheduled time. As another example, image analysis may include detecting a fire from an image in which smoke is present.

Image analysis for image-based monitoring may be performed through the control module 200 or the control units 120 and 220 included in each module 100 and 200.

Figure 8 is a diagram related to image analysis according to one embodiment. Referring to FIG. 8, image analysis may include an object recognition step (S210) and a location/movement information estimation step (S220).

Image analysis may include an object recognition step (S210). The object recognition step (S210) may include recognizing an object included in the image. For example, object recognition may be determining whether an image includes an object such as a ship, tugboat, sea, or port. Furthermore, object recognition may be determining where an object exists in an image.

9 to 11 are diagrams of object recognition steps according to an embodiment.

Figure 9 is an image captured by a camera, and through the object recognition step, the object can be recognized as shown in Figure 10 or Figure 11.

Specifically, Figure 10 shows which object each pixel in the image corresponds to, which is also called segmentation. In this case, the object recognition step may mean a segmentation step. Through segmentation, characteristics corresponding to pixels in the image can be assigned or calculated from the image. This could also be said to have properties assigned or labeled to pixels. Referring to FIGS. 9 and 10 , a segmentation image like that of FIG. 10 can be obtained by performing segmentation based on the image captured by the camera of FIG. 9 . In Figure 10, the first pixel area (P1) is an area on the image of a pixel corresponding to a ship, the second pixel area (P2) is the sea, the third pixel area (P3) is the quay wall of the port, and the fourth pixel area ( P4) is the terrain, and the fifth pixel area (P5) is the area on the image of the pixel corresponding to the sky.

Although FIG. 10 illustrates calculating information about the type of object corresponding to each pixel in the image by performing segmentation, the information that can be obtained through segmentation is not limited to this. For example, characteristics such as location, coordinates, distance, and direction of an object may also be obtained through segmentation. In this case, different characteristics may be expressed independently or reflected simultaneously.

Table 1 is a table regarding labeling that simultaneously reflects information on the type and distance of an object according to an embodiment. Referring to Table 1, classes can be set by considering information about the type and distance of the object, and an identification value can be assigned to each class. For example, identification value 2 can be assigned by considering both terrain, which is information about the type of object, and short distance, which is information about distance. Table 1 is an example of a case where information on type and information on distance are considered together, and other information such as direction information, obstacle movement direction, speed, and route signs can also be considered. Additionally, not all identification values must include a plurality of information, nor must they include the same type of information. For example, certain identifiers only contain information about type (e.g., identifier 1 contains no information about distance), while other identifiers contain information about type and distance, etc. It can be expressed in various ways.

identification value class 0 sky and other One ocean 2 terrain + close range 3 terrain + medium range 4 terrain + range 5 Fixed obstacle + close range 6 Fixed obstacle + medium distance 7 Fixed obstacle + long distance 8 Dynamic Obstacle + Close Range 9 Dynamic Obstacle + Medium Distance 10 Dynamic Obstacle + Ranged

Figure 11 shows where an object exists in the image as a bounding box, which is also called detection. In this case, the object recognition step may mean the detection step. Compared to segmentation, detection can be seen as detecting where an object is contained in the form of a box rather than calculating characteristics for each pixel of the image. Referring to FIGS. 9 and 11 , a detection image like that of FIG. 11 can be obtained by performing detection based on the image captured by the camera of FIG. 9 . In Figure 11, you can see that a ship is detected on an image and the position of the ship is expressed as a square bounding box (BB). Although FIG. 11 shows that only one object is detected, two or more objects can be detected from one image.

Segmentation and detection can be performed using artificial neural networks. Segmentation/detection may be performed using a single artificial neural network, or multiple artificial neural networks may be used, each artificial neural network may perform segmentation/detection, and the results may be combined to produce the final result.

An artificial neural network is a type of algorithm modeled after the structure of a human neural network. It may include one or more layers containing one or more nodes or neurons, and each node may be connected through a synapse. Data input to the artificial neural network (input data) can be output (output data) through a node through a synapse, and information can be obtained through this.

Types of artificial neural networks include a convolution neural network (CNN), which extracts features using filters, and a recurrent neural network (RNN), which has a structure in which the output of a node is fed back to the input. Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), and relational networks (RN) can be applied and have limitations. That is not the case.

Before using an artificial neural network, a learning step is necessary. Alternatively, it can be trained using an artificial neural network. Hereinafter, the step of learning the artificial neural network will be expressed as the learning step, and the step of using it will be expressed as the inference step.

Artificial neural networks can be learned through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

12 and 13 are diagrams of the learning and inference steps of an artificial neural network according to an embodiment.

12 is an example of the learning step of an artificial neural network, in which an untrained artificial neural network receives learning data or training data, outputs output data, and compares the output data with labeling data. An artificial neural network can be trained through backpropagation of the error. Training data, output data, and labeling data may be images. Labeling data may include ground truth. Alternatively, labeling data may be data generated by a user or a program.

Figure 13 is an example of an inference step of an artificial neural network, in which a learned artificial neural network receives input data and outputs output data. Information that can be inferred in the inference step may vary depending on the information in the learning data in the learning step. Additionally, the accuracy of output data may vary depending on the degree of learning of the artificial neural network.

Image analysis may include a location/movement information estimation step (S220). The position/movement information estimation step S220 may include estimating information about the location and/or movement of at least some of the objects recognized in the object recognition step S210. Here, location information may include absolute position such as the coordinates of the object, relative position from a specific reference, distance (distance from an arbitrary point, distance range, etc.), direction, etc., and movement information may include absolute speed, relative speed, It may contain information about the movement of the object, such as speed.

The location/movement information of the object can be used when docking or unberthing a ship. For example, when berthing or unberthing a ship, the distance to the berth or quay wall, the approach speed based on these, the gap and relative speed with other ships, etc. can be used to assist or guide the safe berthing or unberthing of the ship.

The location/movement information of the object can be used when operating a ship. For example, it can assist or guide the safe navigation of ships by detecting other ships or obstacles around them, warning of collisions by using the distance to them and their moving speed, or creating/recommending a route. . Alternatively, autonomous navigation may be performed based on this information.

Position/movement information of an object may be calculated based on the image. For example, location/movement information of a ship can be calculated based on an image that includes the ship, sea, and land as objects. Hereinafter, an object for which location/movement information is estimated is referred to as a target object. For example, in the example above, a ship may be the target object. Additionally, there may be multiple target objects. For example, when estimating the location or speed of each of a plurality of ships included in an image, the plurality of ships may be target objects.

The location/movement information of an object may be expressed as a plurality of categories with a certain range. For example, distance information may be expressed as short distance, middle distance, and far distance, and direction information may be expressed as left direction, front direction, and right direction, etc. It would also be possible to combine these to express left near distance, right far distance, etc. Movement information can be expressed as high speed, low speed, etc.

The position/movement information of an object can be expressed as actual distance value, direction value, and speed value. For example, distance information may be expressed in meters (m), direction information may be expressed in degrees, and movement information may be expressed in cm/s.

The position/movement information of an object can be estimated based on an area or point. For example, the distance between a ship and a quay wall can be estimated by calculating the distance between a point on the ship and a point on the quay wall, or by calculating the shortest distance between a point on the ship and the quay wall. As another example, the spacing between ships can be estimated by calculating the distance between a point on a first ship and a point on a second ship. One point of the ship may correspond to a point of the ship in contact with the sea, or may correspond to the bow or stern of the ship, but is not limited thereto.

Position/movement information of an object can be estimated based on image pixels. As described above, when location/movement information is estimated based on a point, the point may correspond to a pixel in the image. Accordingly, the position/movement information of the object can be calculated based on the spacing between image pixels.

Distance information between points can be calculated based on the spacing between pixels. For example, a certain distance may be assigned to each pixel interval, and the distance between points may be calculated in proportion to the interval between pixels. As another example, the distance between pixels can be calculated based on the coordinate value of the pixel in the image, and the distance between points can be calculated based on this.

Movement information between points may be calculated based on changes in distance information between points. In this case, movement information can be calculated based on a plurality of images or video frames. For example, movement information between points can be calculated based on the distance between points in the previous frame, the distance between points in the current frame, and the time interval between frames.

14 and 15 are diagrams related to estimating location/movement information of an object according to an embodiment.

Referring to FIG. 14, the position/movement information estimation step is the position/movement information (f1, f2) with the quay wall (OBJ2) or the position/movement with other ships (OBJ3, OBJ4) when berthing or berthing of the ship (OBJ1). It may include estimating information (f3, f4). As shown in FIG. 14, position/movement information (f1, f2) between the ship OBJ1 and the quay OBJ2 can be calculated for two points of the ship OBJ1. In this case, the two points may correspond to the point where the ship OBJ1 touches the sea. Additionally, the distance between the ship OBJ1 and the quay OBJ2 may be the shortest distance between the two points and the quay OBJ2. The position/movement information (f3, f4) between the ship (OBJ1) and other ships (OBJ3, OBJ4) may be position/movement information between points corresponding to the bow or stern of the ships (OBJ1, OBJ3, OBJ4). . In this way, when location/movement information is used to assist or guide the berthing or berthing of a ship, it may be referred to as berthing guide information or berthing guide information.

Referring to FIG. 15, the location/movement information estimation step may include estimating the location/movement information (f5, f6) with respect to other vessels (OBJ6) or obstacles such as buoys (OBJ7) when operating the vessel (OBJ5). there is.

Port operation or management can be performed based on the data calculated in the location/movement information estimation stage. For example, if a ship collides with an insect repellent (fender), it will be possible to predict when to replace the fender by calculating the amount of impact from movement information such as the ship's speed.

In the above, we looked at an example of image analysis that estimates location/movement information after performing the object recognition step. Alternatively, object recognition and location/movement information estimation may be performed in one step. For example, segmentation or detection can be performed to recognize an object and at the same time estimate the object's location/movement information.

When all ships on the image are monitored, the amount of data calculation to obtain the ship's berthing guide information increases, which may slow down the processing speed of the device 10. In addition, by determining the target vessel that is the target of monitoring, berthing guide information only for the target vessel (e.g., bow distance, stern distance, etc.), and outputting the obtained berthing guide information, the user can exclude other unnecessary information. You can receive the information necessary for berthing. To this end, the control module 200 may perform image analysis to determine the target ship when at least one image acquired using the image generation unit includes a plurality of ships.

Figure 16 is a flowchart for determining a target vessel according to one embodiment.

Referring to FIG. 16, image-based monitoring may include a ship tracking step (S213), a tracking information acquisition step (S216), and a target ship determination step (S219).

According to one embodiment, the control module 200 may track a ship on an image (S213).

Tracking refers to the act of tracking feature points (objects, tracking targets) that change in space and time on an image. For example, tracking may include tracking the location, shape, movement, trajectory, information, etc. of feature points that change spatially and temporally in an image. Tracking may also include continuing to preserve images of feature points on the image.

According to one embodiment, the control module 200 may perform tracking of the ship based on the segmented image. Specifically, the control module 200 may track the ship based on the area in the segmented image where the class value of the pixel corresponds to the ship. For example, the control module 200 may track the class value of segmented images (continuous) of continuously captured images based on pixels corresponding to ships. Here, the control module 200 may track the representative point (pixel) of the ship within the ship area, whose class value is formed by the pixel corresponding to the ship, between successive frames.

Figure 17 is a diagram of an example of ship tracking according to an embodiment.

Referring to FIG. 17, the control module 200 may track ships in the image by tracking representative points 141, 142, and 143 representing the ships in continuously captured images. For example, the control module 200 may track one point on an area corresponding to each vessel in the segmentation image.

The representative points 141, 142, and 143 of the ship may be any feature points, but since the segmented image has a single class value, the control module 200 illustratively uses a point located in the center of the area corresponding to the ship. (141) can be determined as the representative point.

Of course, the control module 200 may determine the ship's representative points (141, 142, 143) as points in other locations, such as the top point, bottom point (142), left-most point, and right-most point (143) of the ship area. It is possible, and the control module 200 may determine one pixel as a representative point of the ship and track the outline of the area corresponding to the ship, or the area itself, instead of tracking it.

Various techniques can be used for tracking, examples of which include Kalman filter, Extended Kalman filter, Unscented Kalman filter, Particle filter, and Information filter. ), histogram filter, etc., and at least one of these may be applied to tracking.

Of course, this is not necessarily the case, and the control module 200 can track the ship based on images such as captured images, such as tracking the ship using a background image and a difference image. You can track ships.

Additionally, the tracking step (S213) may include the control module 200 tracking a ship other than a tugboat among the ships included in the image. For example, the control module 200 may track a ship that is not a tugboat among ships based on the segmentation image. Specifically, the control module 200 displays pixels corresponding to the input image and objects including the sea, a tugboat, ships other than a tugboat, and terrain features included in the input image, respectively. Image segmentation can be performed using an artificial neural network learned using a learning set labeled with class values indicating water, and ships excluding tugboats can be tracked. Here, the generated segmentation image may include pixels labeled to correspond to ships other than tugboats, and the control module 200 may track pixels corresponding to ships other than tugboats.

According to one embodiment, the control module 200 may obtain tracking information of the ship being tracked on the image (S216).

Tracking information may refer to information about the ship being tracked, and the tracking information may be used to determine the target ship. For example, but not limited to, tracking information may include location information of the ship, movement information of the ship, VTS information of the ship, AIS information of the ship, user input information, etc.

According to one embodiment, the step of acquiring tracking information (S216) may include the control module 200 acquiring location information of a ship from an image. For example, the control module 200 may acquire location information of a representative point of a ship being tracked in continuously captured images. Here, the location information of the representative point of the ship may be information about the distance between the representative point and the quay wall, the distance between the representative point and another object, the coordinates of the representative point on the image, etc.

According to one embodiment, the step of acquiring tracking information (S216) may include the control module 200 acquiring movement information of the ship from an image. For example, the control module 200 may obtain movement information of a representative point of a ship being tracked in continuously captured images. Here, the control module 200 may obtain movement information of the representative point based on the location information of the representative point. Here, the movement information of the representative point of the ship may be information on the approach speed of the representative point to the quay, the approach speed of the representative point to another object, the speed change of the representative point, etc.

According to one embodiment, the step of acquiring tracking information (S216) may include the control module 200 acquiring navigation information such as VTS information and AIS information of the tracked ship. For example, the control module 200 may obtain at least one of VTS information and AIS information. Here, the control module 200 can obtain information about the destination of the ship according to navigation information, the scheduled berthing time of the ship, etc.

According to one embodiment, the step of acquiring tracking information (S216) may include the control module 200 acquiring user input information about the tracked ship. For example, the control module 200 may obtain user input information used to determine a target ship from a user terminal.

However, acquisition of tracking information need not be limited to being performed by the above-described method and may be performed in various ways.

According to one embodiment, the control module 200 may determine the target ship based on the acquired tracking information (S219).

According to one embodiment, the control module 200 may determine the target ship based on the acquired location information of the ship. For example, the control module 200 may determine the target ship based on the distance between the ship and the quay wall.

Figure 18 is a diagram of an example of target vessel determination based on location information according to an embodiment.

Referring to FIG. 18, when there are a plurality of ships in the image, the control module 200 may determine a target ship to be monitored based on location information.

According to one embodiment, the control module 200 may determine the target ship based on the distance from the quay wall of the tracked ship. Here, the distance from the quay wall of the tracked ship may be the distance between the representative point of the tracked ship and the quay wall.

For example, the control module 200 may determine a ship whose distance from the quay wall is less than a critical distance among tracked ships as the target ship. Referring to FIG. 18, the control module 200 may determine a ship whose distance from the quay wall is smaller than a preset threshold distance (d ₀ ) among the tracked ships as the target ship. If the ship 151 determined to be the target ship moves and is located at a location where the distance from the quay wall is greater than d ₀ , the control module 200 may determine that the ship 151 is not the target ship. Additionally, the control module 200 may determine the ship 152 as the target ship when the ship 152 moves and is located at a location where the distance from the quay wall is closer than d ₀ .

The control module 200 can set various threshold distances used to determine the target vessel. For example, the control module 200 may set the critical distance based on the size of the recognized vessel. Specifically, the control module 200 can set the critical distance to be larger as the size of the ship increases. In addition, the control module 200 may determine as the target ship a ship whose distance to a specific object (structure, etc.) other than a quay wall among ships in the image is smaller than the critical distance.

According to one embodiment, the control module 200 may determine the target ship based on the location on the image of the tracked ship. Here, the position on the image of the tracked ship may be the position on the image of the representative point of the tracked ship. For example, the control module 200 may determine a ship located in a preset specific area of the image among ships in the image as the target ship. Referring to FIG. 18, the control module 200 may determine a ship 151 located within a preset square-shaped area 153 on the image among the ships being tracked as the target ship. The control module 200 may determine the ship 152 as the target ship when the ship 152 moves and is located within the area 153. Additionally, the control module 200 may exclude the ship 151 determined as the target ship from the target ships when the ship 151 moves and is located outside the area 153.

The control module 200 can set the area 153 in various ways. For example, the control module 200 can determine the size, shape, number, etc. of the area 153 in various ways. Specifically, the control module 200 may determine the area 153 as an area around the path through which ships usually enter from the berth where the image is captured, an area where ships mainly dock, etc. Also, conversely, the control module 200 may determine a ship located in an area other than the preset specific area 153 among ships in the image as the target ship.

According to one embodiment, the control module 200 may determine the target ship based on the acquired movement information of the ship. For example, the control module 200 may determine the target ship based on the speed at which the ship approaches the quay.

19 is a diagram illustrating an example of target vessel determination based on movement information according to an embodiment.

Referring to FIG. 19, when there are multiple ships in the image, the control module 200 may determine the target ship based on movement information.

According to one embodiment, the control module 200 may determine the target ship based on the speed of the tracked ship. For example, the control module 200 may determine a ship whose speed (absolute value of speed) is greater than the critical speed among ships in the image as the target ship. Referring to FIG. 16, the control module 200 may determine a ship 161 approaching at a speed v _o greater than the critical speed value among the tracked ships as the target ship. The control module 200 may determine that the ship 161, which is determined to be the target ship, is not the target ship when berthing is completed and the ship 161 is stationary at the berth. Additionally, the control module 200 may determine the ship 162 as the target ship when the anchored ship 162 moves at a speed (regardless of direction) greater than the critical speed.

The control module 200 can set the critical speed size used to determine the target vessel in various ways. For example, the control module 200 may determine the critical speed used to determine the target ship based on the size of the ship. Specifically, the control module 200 may determine that the critical speed increases as the size of the ship increases.

In addition, the control module 200 is also capable of determining a target ship among tracked ships based on conditions related to movement information, such as direction of speed, size, and change in speed (acceleration). For example, the control module 200 may determine a ship whose acceleration magnitude (absolute value of acceleration) is greater than a preset value among ships in the image as the target ship.

According to one embodiment, the control module 200 may determine the target ship based on the acquired navigation information of the ship. Here, navigation information may include information related to the navigation of a ship. For example, navigation information may be AIS information, VTS information, CITS information, etc. Navigation information may include information such as the ship's departure point, destination, mooring point, and navigation route in relation to the ship's voyage.

For example, the control module 200 may determine the target ship based on the destination according to navigation information of the tracked ship.

Figure 20 is a diagram of an example of target vessel determination based on navigation information according to an embodiment.

Referring to FIG. 20, when there are multiple ships in the image, the control module 200 may determine the target ship based on navigation information.

According to one embodiment, the control module 200 may determine the target ship based on the destination according to navigation information of the tracked ship. For example, the control module 200 may determine, among ships in the image, a ship whose destination according to navigation information is a berth on which a camera that captured the image is installed as the target ship. Referring to FIG. 17, among the ships being tracked, the ship 171, whose destination according to navigation information is a berth equipped with a camera that captured the image, can be determined as the target ship. When the ship 171 determined as the target ship departs or departs port according to navigation information, the control module 200 may determine that the ship 171 is not the target ship. In addition, the control module 200 may determine the ship 172 as the target ship when the ship 172 docks elsewhere and docks at a berth where a camera that captured the image is installed according to new navigation information. .

In addition, the control module 200 may determine the target vessel based on other conditions related to navigation information. For example, the control module 200 may determine a ship recognized on the image as the target ship at a time scheduled to dock at the berth according to navigation information.

According to one embodiment, the control module 200 may determine the target ship based on acquired user input information. User input information may include information input and received from the user's device. For example, user input information may include information input from the user related to selection of a vessel to perform monitoring, selection of berths, etc.

Figure 21 is a diagram of an example of target ship determination based on user input information according to an embodiment.

Referring to FIG. 21, when there are multiple ships in the image, the control module 200 may determine the target ship based on user input information. For example, the control module 200 may determine a ship selected as a target for monitoring according to user input information among tracked ships as the target ship. Referring to FIG. 18, the control module 200 may determine a ship 181 selected from the indicator 193 displayed on the screen according to user input information among the ships being tracked as the target ship. After the ship 181 determined as the target ship has completed berthing, the control module 200 may determine that the ship 181 that is not selected according to user input information is not the target ship. Additionally, the control module 200 may determine the ship 182 as the target ship when the unselected ship 182 is selected by the user according to user input information.

Here, user input information used to determine the target ship may be provided in various ways. For example, the user input information may be information for determining a target ship by selecting a ship from a list of ships entering or leaving a port. Additionally, the user input information may be information that determines the target ship, which is a ship located at a selected location on an output port image (for example, an image output from a user terminal).

In addition, the control module 200 may determine the image of the berth from which monitoring information is output among images captured from a plurality of berths. For example, the control module 200 may output an image of a berth selected according to user input information among images captured from a plurality of berths and determine a ship included in the image as the target ship.

Additionally, the step of determining a target ship based on the acquired tracking information (S216) may include the control module 200 determining one target ship on the image.

According to one embodiment, the control module 200 may determine as the target ship one of a plurality of ships that meet the conditions determined as the target ship on the image. For example, when there are multiple ships on the image that meet the conditions for being determined as a target ship, the control module 200 may determine one ship that is determined later as the target ship as the target ship. For example, referring to FIG. 18, when the ship 151 and the ship 152 are sequentially located within the area 153, the control module 200 initially has the ship 151 as the target ship and then ships 152. ) is located within the area 153, the target ship can be determined to be the ship 152. Conversely, if there are multiple ships on the image that meet the conditions for being determined as the target ship, the control module 200 may determine as the target ship one ship that is determined to be the target ship earlier.

However, determination of the target vessel need not be limited to being performed by the above-described method and may be performed in various ways.

Figure 22 is a diagram related to obtaining berthing guide information of a target ship according to an embodiment.

Referring to FIG. 22, the control module 200 may acquire berthing guide information of the target ship 191 among the tracked ships 191 and 192. Here, the berthing guide information may include bow distance, stern distance, bow speed, stern speed, distance between the target ship and other ships, and relative speed between the target ship and other ships. The bow distance may refer to the distance the bow of the ship is from the quay wall, and the stern distance may refer to the distance the stern of the ship is away from the quay wall. Bow speed may refer to the speed at which the bow of the ship approaches the quay wall, and stern speed may refer to the speed at which the stern of the ship approaches the quay wall.

According to one embodiment, the control module 200 may obtain the distance between the target ship 191 and the quay wall.

For example, the control module 200 may obtain the bow distance 193, which is the distance between the bow of the target ship 191 and the quay wall. Specifically, the control module 200 may determine one point corresponding to the bow of the target ship and obtain the distance between one point corresponding to the bow and the quay wall. For another example, the control module 200 may obtain the stern distance 194, which is the distance between the stern and the quay wall of the target ship 191. Specifically, the control module 200 may determine one point corresponding to the stern of the target ship and obtain the distance between the one point corresponding to the stern and the quay wall.

According to one embodiment, the control module 200 may extract a pair of points corresponding to both ends of the bottom of the ship in contact with the sea level. Here, the control module 200 may determine the pair of points as one point corresponding to the bow and one point corresponding to the stern. For example, the control module 200 may obtain the distance between one of a pair of points corresponding to both ends of the bottom of the ship in contact with the sea surface and the quay wall to obtain the bow distance. Additionally, the control module 200 may obtain the distance between the quay wall and the other one of a pair of points corresponding to both ends of the bottom of the ship in contact with the sea level in order to obtain the stern distance.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target ship 191 and instead of obtaining the bow and stern distances of the ship, the control module 200 extracts any one point or two or more points of the target ship 191. It is also possible to obtain the distance between the ship and the quay wall by extracting points, and it is also possible to obtain the distance between the ship and the quay wall based on the outline of the area or the area itself.

According to one embodiment, the control module 200 may obtain the speed at which the target ship 191 approaches the quay wall.

For example, the control module 200 may obtain the bow speed, which is the speed at which the bow of the target ship 191 approaches the quay wall. Specifically, the control module 200 may determine a point corresponding to the bow of the target ship 191 and obtain the speed at which the point corresponding to the bow approaches the quay wall. Here, the control module 200 may obtain the speed at which the bow of the target ship 191 approaches the quay wall based on the bow distance 193 of the target ship 191. The control module 200 compares the bow distance 193 of the target ship 191 in the current frame with the bow distance 193 of the target ship 191 in the subsequent frame to determine the speed at which the bow of the target ship 191 approaches the quay. can be obtained.

For another example, the control module 200 may acquire the stern speed, which is the speed at which the stern of the target ship 191 approaches the quay wall. Specifically, the control module 200 may determine a point corresponding to the stern of the target ship 191 and obtain the speed at which the point corresponding to the stern approaches the quay wall. Here, the control module 200 may obtain the speed at which the stern of the target ship 191 approaches the quay wall based on the stern distance 194 of the target ship 191. The control module 200 compares the stern distance 194 of the target ship 191 in the current frame with the stern distance 194 of the target ship 191 in subsequent frames to determine the speed at which the stern of the target ship 191 approaches the quay wall. can be obtained.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target ship 191 and instead of obtaining the bow speed and stern speed of the ship, the control module 200 extracts any one point or two or more points of the target ship 191. It is also possible to obtain the speed at which a ship approaches the quay wall by extracting points, and it is also possible to obtain the speed at which the ship approaches the quay wall based on the outline of the area or the area itself.

According to one embodiment, the control module 200 may obtain the distance between the target ship 191 and another ship 192.

For example, the control module 200 may obtain the distance between the bow of the target ship 191 and another ship 192. Specifically, the control module 200 may determine a point corresponding to the bow of the target ship 191 and obtain the distance between the point corresponding to the bow and another ship 192.

For another example, the control module 200 may obtain the distance between the stern of the target ship 191 and another ship 192. Specifically, the control module 200 may determine one point corresponding to the stern of the target ship 191 and obtain the distance between the one point corresponding to the stern and the quay wall.

According to one embodiment, the control module 200 may extract a pair of points corresponding to the bow end and the stern end from the area corresponding to the ship. Here, the control module 200 may determine the pair of points as one point corresponding to the bow and one point corresponding to the stern. For example, the control module 200 may acquire the distance between the other vessel and one of a pair of points corresponding to the fore end and the stern end in the area corresponding to the vessel to obtain the distance between the vessel and the other vessel. You can. Here, the control module 200 may obtain the distance between other ships based on the point that is closer to the other ship among a pair of points corresponding to the fore end and the stern end.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target ship 191, and instead of obtaining the distance between the bow and stern of the ship and another ship, the control module 200 extracts points corresponding to the bow and stern of the target ship 191. Alternatively, it is possible to obtain the distance between a ship and another ship by extracting two or more points, and it is also possible to obtain the distance between a ship and another ship based on the outline of the area or the area itself.

According to one embodiment, the control module 200 may obtain the relative speed between the target ship 191 and another ship 192.

For example, the control module 200 may obtain the speed at which the bow of the target ship 191 approaches another ship 192. Specifically, the control module 200 may determine a point corresponding to the bow of the target ship 191 and obtain the speed at which the point corresponding to the bow approaches another ship 192. Here, the control module 200 may obtain the speed at which the bow of the target ship 191 approaches the other ship 192 based on the distance between the bow of the target ship 191 and the other ship 192. The control module 200 compares the distance between the bow of the target ship 191 and the other ship 192 in the current frame with the distance between the bow of the target ship 191 and the other ship 192 in the subsequent frame. The speed at which the bow of the target ship 191 approaches the other ship 192 can be obtained.

For another example, the control module 200 may obtain the speed at which the stern of the target ship 191 approaches another ship 192. Specifically, the control module 200 may determine a point corresponding to the stern of the target ship 191 and obtain the speed at which the point corresponding to the stern approaches another ship 192. Here, the control module 200 may obtain the speed at which the stern of the target ship 191 approaches the other ship 192 based on the distance between the stern of the target ship 191 and the other ship 192. The control module 200 compares the distance between the stern of the target ship 191 and the other ship 192 in the current frame and the distance between the stern of the target ship 191 and the other ship 192 in the subsequent frame. The speed at which the stern of the target ship 191 approaches another ship 192 can be obtained.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target ship 191, and instead of obtaining the speed at which the bow and stern of the ship approaches the other ship 192, the control module 200 extracts the points corresponding to the bow and stern of the target ship 191. It is also possible to obtain the relative speed between the target ship 191 and another ship 192 by extracting one point or two or more points, and the target ship 191 and the target ship 191 based on the outline of the area or the area itself. It is okay to obtain the relative speed between different vessels 192.

Figure 23 is a flowchart of monitoring information output according to one embodiment.

Referring to FIG. 23, berthing monitoring may further include a step (S30) of outputting monitoring information. There is no limit to the information output in the monitoring information output stage as long as it is information related to image-based monitoring, such as images of the surroundings of the ship, the ocean, or the port, and the types and distance/speed of objects included in the images.

The device 10 can visually output monitoring information. For example, the device 10 may output monitoring information through an output module such as a display. For another example, the device 10 may transmit monitoring information to the user's terminal through the communication module 300 and transmit a control signal to output the monitoring information through the display of the user terminal.

The monitoring information output step (S30) may include displaying an image acquired using an image generation unit in the image acquisition step. In addition, it may include displaying various images related to image-based monitoring, such as images that have gone through a preprocessing step, images after segmentation or detection, and images after viewpoint conversion.

Additionally, the monitoring information output step (S30) may include the control module 200 displaying the position/movement information estimated in the image analysis step through an output module or a user terminal.

Figure 24 is a diagram of an example of monitoring information output according to an embodiment.

Referring to FIG. 24, the image and eyepiece guide information may be displayed together. As shown in FIG. 24, the displayed berthing guide information may include the bow distance, bow speed, stern distance, and stern speed of the target ship.

The monitoring information output step (S30) may include the device 10 providing information to the user in other ways, such as outputting sound or vibration in addition to visual display. For example, the control module 200 sounds a warning sound when there is a risk that the target ship collides with a quay wall, another ship, or an obstacle, when the approach speed to the quay wall is higher than the standard speed when berthing, or when the ship deviates from the course and operates. It can be output through an output module or user terminal.

Image-based monitoring may include surveillance. Here, surveillance may mean providing users with security-related information, such as monitoring intruders or unregistered vessels approaching the port, and information on emergency situations, such as fires.

Device 10 can monitor intruders based on whether a person is included in the image and when the image was captured. For example, the control module 200 may determine that an intruder exists if a person is included in a port image captured at a time when work is not in progress at the port.

Device 10 may perform vessel surveillance based on whether a vessel is included in the image. For example, the control module 200 can monitor ships by providing information to the user when a ship that is not registered in AIS is detected.

Additionally, the device 10 may perform surveillance by detecting people or ships based on images through segmentation or detection.

25 to 27 are diagrams of another example of monitoring information output according to an embodiment.

Referring to FIGS. 25 and 26, the image and berthing guide information may be displayed together, and if there are multiple target vessels, a plurality of berthing guide information may be output.

As shown in FIG. 25, the location where the berthing guide information is output may correspond to the location of the target ship on the image. For example, berthing guide information for the left target ship may be displayed on the left, and berthing guide information for the right target ship may be displayed on the right.

Additionally, as shown in FIG. 26, berthing guide information may be output along with the identifier of the target ship. For example, a random identifier called 'Ship 1' may be assigned to the target ship on the left side of the image and 'Ship 2' may be given to the target ship on the right side of the image, and the assigned identifier and berthing guide information for each target ship may be output together. . Additionally, the assigned identifier and berthing guide information may be displayed around the target vessel on the image.

Also, referring to FIG. 27, the image and the berthing guide information may be displayed together, and if there are multiple ships and only one target ship is determined, only the berthing guide information of the target ship may be output. For example, if the target ship is determined to be the left ship in progress of berthing, only the berthing guide information of the left ship determined to be the target ship may be displayed on the screen.

Above, we looked at image-based monitoring based on a single image. In addition, image-based monitoring can be performed based on multiple images. When image analysis is performed based on multiple images, the total monitoring area of the image-based monitoring device 10 may increase or monitoring accuracy may be improved.

Figure 28 is a diagram related to image-based monitoring based on a plurality of images according to an embodiment.

Referring to FIG. 28, the image acquisition step may include a first image acquisition step (S11) and a second image acquisition step (S12), and the image analysis step (S20) is obtained in the first image acquisition step (S11). Image analysis may be performed based on the first image and the second image acquired in the second image acquisition step (S12).

Image analysis can be performed after generating one image based on multiple images. For example, the control module 200 may generate a panoramic image in which a plurality of images are registered and perform image analysis on the panoramic image. Specifically, the control module 200 may generate a registered image or a fused image by registering or fusing the first image and the second image, and perform image analysis based on the generated registered image or fusion image.

Alternatively, the final analysis result may be calculated based on the results of image analysis based on each of the plurality of images. For example, after performing image analysis on a first image to obtain first monitoring information and performing image analysis on a second image to obtain second monitoring information, a final information is obtained based on the first monitoring information and the second monitoring information. Monitoring information can be obtained.

As an example of a method of obtaining final monitoring information from a plurality of monitoring information, there may be a method of calculating the final monitoring information by considering a plurality of monitoring information by weight.

Alternatively, the final monitoring information can be calculated based on whether the plurality of monitoring information do not match each other or whether the difference is greater than a threshold such as a specific value (hereinafter referred to as “error occurrence”). For example, the final monitoring information is calculated by considering a plurality of monitoring information by weight based on whether an error occurs, the final monitoring information is calculated by prioritizing specific monitoring information among the plurality of monitoring information, or the specific monitoring information is used for other monitoring. There may be a method of correcting with information or ignoring the corresponding monitoring information, but it is not limited to this.

The plurality of images may be images of the same type. For example, if a sensor module includes two identical image generation units or two identical sensor modules including one image generation unit are deployed to perform image-based monitoring, the first image and the second image are of the same type. It can be.

The monitoring areas of the plurality of images may be different. For example, the first image may monitor a near distance from the image generating unit, and the second image may monitor a far distance. Alternatively, the first image may monitor the left side from the image generating unit, and the second image may monitor the right side.

Additionally, the device 10 can output monitoring information obtained by analyzing a plurality of images together with appropriate images.

The device 10 may output monitoring information obtained by analyzing a plurality of images together with an image including the target ship among the plurality of images. For example, the control module 200 analyzes a plurality of images acquired by a plurality of cameras, at least some of which capture images of different areas, and then selects a target ship among the plurality of images through an output module or a user terminal. The captured first image may be output together with the first eyepiece guide information calculated from the first image.

29 and 30 are diagrams of another example of monitoring information output according to an embodiment.

Referring to FIGS. 29 and 30 , when a target ship is determined after analyzing a plurality of images, berthing guide information of the target ship may be displayed together with an image including the target ship among the plurality of images. For example, if the target ship is determined to be 'Ship 1', the berthing guide information for 'Ship 1' may be displayed along with the image of the berth containing 'Ship 1'. Additionally, if the target vessel is determined to be 'Ship 3', berthing guide information for 'Ship 3' may be displayed along with an image of the berth containing 'Ship 3'.

The device 10 may output monitoring information obtained by analyzing a plurality of images together with the destination image of the ship among the plurality of images. For example, the control module 200 analyzes a plurality of images acquired by a plurality of cameras that capture images of areas at least partially overlapping, and then, when the target ship is captured together in the plurality of images, the plurality of images A control signal can be generated so that the image acquired using a camera installed at the destination according to the navigation information of the target vessel and the berthing guide information calculated from the image are output together.

31 and 32 are diagrams of another example of monitoring information output according to an embodiment.

31 and 32 show eyepiece guide information displayed together with first and second images, which are images of each adjacent berth, acquired by a plurality of cameras that capture images of an area at least partially overlapping.

31 and 32, vessel 252 is captured in two images together at the same time. If the destination of the ship 252 is not the berth of the first image but the berth of the second image, it is desirable to output the berthing guide information of the ship 252 passing toward another place together with the first image rather than the second image. You may not. As shown in FIG. 31, the first image is preferably output along with berthing guide information of the ship 251 whose berth in the first image is the destination according to navigation information. In addition, as shown in FIG. 32, it is preferable that the berthing guide information of the ship 252 whose destination is the berth of the second image according to the navigation information is output together with the second image.

Of course, the monitoring information obtained by analyzing a plurality of images may be output together with any of the plurality of images, and is not limited to the above description.

Image-based monitoring may further include a viewpoint conversion step.

In general, images generated by an image generation unit such as a camera may appear in a perspective view. Converting this to a top view (planar view), side view, or another perspective view can be called viewpoint conversion. Of course, the top view or side viewpoint image may be converted to another viewpoint, and the image generation unit may generate a top view image or a side viewpoint image, and in this case, viewpoint conversion may not need to be performed.

Figures 33 and 34 are diagrams related to viewpoint conversion according to one embodiment.

Referring to FIG. 33, a different perspective view image can be obtained through viewpoint conversion of the perspective view image. Here, viewpoint conversion can be performed so that the quay wall OBJ8 is positioned along the horizontal direction (left and right directions in the image) on the image. Referring to FIG. 34, a top view image can be obtained through viewpoint conversion of the perspective view image. Here, the top view image may be a view looking down at the sea level from a direction perpendicular to the sea level. Additionally, as in FIG. 33, viewpoint conversion may be performed so that the quay wall OBJ9 is positioned along the horizontal direction in the image.

After acquiring the image and performing viewpoint conversion, image analysis can be performed to obtain information necessary for the vessel's berthing guide. According to one example, acquisition of eyepiece guide information may be performed based on a segmented image whose viewpoint has been converted. Specifically, the perspective can be converted from a perspective view segmentation image to a top view segmentation image, and berthing guide information for the ship can be obtained based on the area in which the class value of the pixel in the top view segmented image corresponds to the ship. Of course, the segmentation image whose viewpoint has been converted may be a segmentation image whose viewpoint has been converted to various viewpoints such as a side view rather than a top view. In addition, instead of extracting two points to obtain location/movement information of the ship, the outline of the area, Alternatively, the location/movement information of the vessel may be obtained based on the area itself.

Of course, this is not necessarily the case, and it is also possible to perform it based on an image such as a captured video whose viewpoint has been converted.

In addition, after acquiring the image and performing viewpoint conversion, monitoring information can be output, such as displaying the image to the user. In this case, information about the surrounding situation can be more easily provided to the user through viewpoint conversion.

Viewpoint conversion of an image can be performed in various ways.

As an example of viewpoint transformation, inverse projective mapping (IPM) can be performed. A two-dimensional image is created when light reflected from a subject in three-dimensional space is incident on the image sensor through the lens of the camera, and the relationship between two dimensions and three dimensions depends on the image sensor and lens. For example, Equation 1 and can be expressed together.

Here, the matrix on the left side means two-dimensional image coordinates, the first matrix on the right side means intrinsic parameters, the second matrix means extrinsic parameters, and the third matrix means three-dimensional coordinates. Specifically, fx and fy are focal lengths, cx and cy are principal points, and r and t are rotation and translation transformation parameters, respectively.

The viewpoint can be changed by projecting a two-dimensional image onto an arbitrary three-dimensional plane through back projection transformation. For example, a perspective view image can be converted to a top view image through back projection transformation, or converted to another perspective view image.

Internal parameters may be required for viewpoint conversion. As an example of a method for obtaining internal parameters, the Zhang method can be used. The Zhang method is a type of polynomial model and is a method of obtaining internal parameters by photographing a grid of known grid size at various angles and distances.

For viewpoint conversion, information about the location and/or posture of the image generation unit/sensor module that captured the image may be required. This information may be obtained from a position measurement unit and an attitude measurement unit.

Alternatively, information about the location and/or posture may be obtained based on the location of the fixture included in the image. For example, at a first viewpoint, the image generating unit may be placed in a first position and/or a first posture and generate a first image including a target fixture, which is a fixed object such as terrain or a building. Afterwards, at a second time point, the image generating unit may generate a second image including the target fixture. Compare the position of the target fixture on the first image and the position of the target fixture on the second image to calculate a second position and/or posture, which is the position and/or posture of the image generating unit at the second viewpoint. can do.

Additionally, the accuracy of image analysis may vary depending on the selection of the reference plane when converting the viewpoint. For example, when converting a perspective image into a top view image, the accuracy of image analysis based on the top view image may vary depending on the height of the reference plane. In order to accurately calculate the distance between objects on sea level, it may be desirable for the reference plane to be sea level when converting the viewpoint. Since the height of the sea level may change over time, it may be desirable to perform viewpoint conversion considering the height of the sea surface to improve the accuracy of image analysis.

The above-described viewpoint conversion method is only an example and viewpoint conversion may be performed in a different way. The viewpoint conversion information includes the matrix of Equation 1 described above, parameters, coordinates, location and/or posture information, etc. Includes information needed for

Image-based monitoring may include a pre-processing step. Preprocessing refers to all types of processing performed on an image, including image normalization, image brightness equalization, histogram equalization, image resize, and upscaling to change the resolution/size of the image. and downscaling, cropping, noise removal, etc. Here, noise may include fog, rain, water droplets, sea clutter, fine dust, direct sunlight, salt, and combinations thereof, and removing noise means removing or reducing noise components included in the image. It can be included.

Looking at normalization as an example of preprocessing, normalization may mean calculating the average of the RGB values of all pixels of an RGB image and subtracting this from the RGB image.

Looking at defogging as another example of preprocessing, defogging may mean converting an image taken in a foggy area to look like an image taken in a clear area through preprocessing. Figure 13 is a diagram related to fog removal according to one embodiment. Referring to FIG. 13, through fog removal, an image taken of a foggy area as shown in (a) of FIG. 13 can be converted into an image from which the fog has been removed, as shown in (b) of FIG. 13.

Looking at water droplet removal as another example of preprocessing, water droplet removal may mean converting an image of water droplets on the front of the camera so that it appears as if the water droplets have been removed through preprocessing.

According to one embodiment, after the image acquisition step (S10), an image analysis step (S20) may be performed through a pre-processing step. For example, image analysis may be performed after preprocessing an image acquired using an image generation unit. Image analysis can be facilitated or accuracy improved through image preprocessing.

Image preprocessing can be performed through artificial neural networks. For example, an image taken of a foggy area can be input into an artificial neural network and converted to look like an image taken of a clear area, or an image containing noise can be input into an artificial neural network to obtain an image with the noise removed. You can. Examples of artificial neural networks include, but are not limited to, GAN.

Alternatively, image preprocessing can be performed using an image mask. For example, an image taken in a foggy area can be converted to look like an image taken in a clear area by applying an image mask. Here, examples of image masks include deconvolution filters and sharpen filters, and image masks may be generated through artificial neural networks such as GAN, but are not limited thereto.

In the above, we looked at the case where image analysis is performed after preprocessing the image. Alternatively, it may be possible to perform image analysis including preprocessing. For example, when the image analysis step includes segmentation or detection, the result of segmentation or detection of an image containing noise may be implemented to be equivalent to the result of performing segmentation or detection of an image without noise.

Figure 35 is a flowchart of image-based monitoring according to one embodiment.

Referring to Figure 35, berthing monitoring includes an image acquisition step (S1010), a segmentation step (S1210), a ship tracking step (S1213), a tracking information acquisition step (S1216), a target ship determination step (S1219), and a berthing guide information acquisition step. (S1220) and a monitoring information output step (S1030). Each step can be implemented as described above.

The device 10 may acquire an image including an object through the image acquisition step (S1010). The device 10 may obtain a segmentation image by performing a segmentation step (S1210) based on the image. The device 10 may perform a ship tracking step (S1213) to track at least one ship recognized based on the segmentation image. The device 10 may obtain tracking information by performing the tracking information acquisition step (S1216). Here, the tracking information may include at least one of ship location information, ship movement information, ship navigation information, and user input information. The device 10 may perform a target ship determination step (S1219) to determine a target ship among the ships being tracked based on the acquired tracking information. The device 10 may perform the berthing guide information acquisition step (S1220) to obtain berthing guide information of the determined target ship based on the segmentation image. Here, the berthing guide information may include at least one of the bow distance and the stern distance of the target ship. The device 10 may perform a monitoring information output step (S1030) and output eyepiece guide information together with the image.

The embodiment of FIG. 35 is merely an example, and image-based monitoring may be performed in a different way.

As an example, some steps may not be performed in the embodiment of Figure 35. Referring to FIG. 35, the step of outputting monitoring information (S1030) may not be performed.

As another example, other steps may be added in the embodiment of FIG. 35, such as adding a step of matching images or adding a step of converting the viewpoint of the image.

As another example, some steps may be replaced with other steps in the embodiment of FIG. 35, such as the segmentation step being replaced with a detection step.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

In the above, the configuration and features of the present invention have been described based on the examples, but the present invention is not limited thereto, and various changes or modifications may be made within the spirit and scope of the present invention. It is obvious, and therefore, it is stated that such changes or modifications fall within the scope of the attached patent claims.

10: Monitoring device
100: sensor module
110: Department of Communications
120: control unit
130: camera
200: control module
210: Department of Communications
220: control unit
300: Communication module

Claims (11)

In a ship monitoring method performed by computing means,
Obtaining a port image using a camera installed in the port to capture images;
An artificial neural network learned using a learning set that labels pixels corresponding to input images and objects including the sea, ships, and features included in the input image with class values indicating the sea, ships, and features, respectively. generating a segmentation image for the objects from the harbor image using the port image;
Obtaining at least one of VTS information and AIS information of a plurality of ships;
Tracking the plurality of vessels using at least one of the VTS information and AIS information;
determining a target ship among the plurality of ships being tracked based on a distance between a representative point representing the ship on the segmentation image and a quay (pier); and
Based on the segmentation image, obtaining guide information including a bow distance, which is the distance between the bow of the target ship and the quay wall, and a stern distance, which is the distance between the stern of the target ship and the quay wall; comprising,
Vessel monitoring methods. According to paragraph 1,
The step of determining the target ship is,
A step of determining, among the plurality of ships being tracked, a ship whose distance between the representative point representing the ship on the segmentation image and the quay is less than a preset value as the target ship.
Vessel monitoring methods. According to paragraph 1,
The bow distance and the stern distance are,
Obtained based on a pair of points corresponding to both ends of the bottom of the target ship in contact with the sea level,
Vessel monitoring methods. According to paragraph 1,
The above guide information is:
Including a bow speed, which is the speed at which the bow of the target ship approaches the quay wall, and a stern speed, which is the speed at which the stern of the target ship approaches the quay wall, obtained based on the bow distance and the stern distance.
Vessel monitoring methods. According to paragraph 1,
The above guide information is:
Including the distance between the target ship and other ships obtained based on the segmentation image,
Vessel monitoring methods. According to clause 5,
The above guide information is:
Containing a relative speed between the target ship and the other ship obtained based on the distance between the target ship and the other ship,
Vessel monitoring methods. According to paragraph 1,
The artificial neural network is,
Labeling pixels corresponding to the input image and objects including the sea, tugboats, ships excluding tugboats, and features included in the input image with class values indicating the sea, tugboats, ships excluding tugboats, and features, respectively. Learned using one learning set,
Vessel monitoring methods. According to paragraph 1,
Further comprising: outputting the guide information together with the harbor image,
Vessel monitoring methods. According to paragraph 1,
The port image above is,
Containing a panoramic image in which a plurality of port images are registered,
Vessel monitoring methods. Recording a program for executing the method according to any one of claims 1 to 9,
Recording media. Cameras installed in ports to capture images;
Acquire a harbor image captured by the camera, and label pixels corresponding to the input image and objects including the sea, ship, and feature included in the input image with class values indicating the sea, ship, and feature, respectively. Generate segmentation images for the objects from the harbor image using an artificial neural network learned using one learning set, obtain at least one of VTS information and AIS information of a plurality of ships, and obtain the VTS information and AIS information Tracking the plurality of ships using at least one of the plurality of ships to be tracked, determining a target ship based on the distance between a representative point representing the ship on the segmentation image and a pier among the plurality of ships being tracked, and the segmentation A control module that obtains guide information including a bow distance, which is the distance between the bow of the target ship and the quay wall, and a stern distance, which is the distance between the stern of the target ship and the quay wall, based on the image; and
Including a communication module that transmits the guide information to a remotely located terminal.
Ship monitoring device.