JP2021005333A - Self-removal monitoring system for medical indwelling device and self-removal monitoring method for medical indwelling device - Google Patents

Abstract

To provide self-removal monitoring means for a medical indwelling device that allows for automatic recognition of actions related to self-removal, and quick and accurate notification to a monitoring person.SOLUTION: A self-removal monitoring system 100 for a medical indwelling device includes: an imaging unit 110; a coordinate conversion unit 122 that performs coordinate conversion to rotate coordinate axes set in a monitored image; a monitoring target identification unit 123 that identifies a monitoring target person 10 in the monitored image; an insertion position identification unit 124 that identifies an insertion position P of a medical indwelling device 11 inserted into a body of the monitoring target person 10; a motion recognition unit 125 that recognizes a motion of the monitoring target person 10; an action requirement determination unit 127 that determines whether some action is required depending on whether or not a combination of the insertion position P and the motion is a specific combination; and an action request notification output unit 130 that outputs an action request notification when the action requirement determination unit 127 determines that the action is required.SELECTED DRAWING: Figure 1

Description

The present invention is a medical treatment caused by self-removal of various medical devices (in the present specification, these are collectively referred to as "medical clothing devices") used by being attached to a patient's body in a medical field such as a catheter. The present invention relates to a "self-removal monitoring system for medical equipment" or a "monitoring method for self-removal prevention of medical equipment" for the purpose of preventing accidents.

In the past, medical accidents often occurred in the medical field due to self-removal of medical equipment related to life support. In order to avoid this, it was unavoidable that the patient's hands were tied to the bed using a restraining band to restrain the patient. Such suppression was not a desirable coping method from the viewpoint that it not only causes mental distress to the patient but also tends to cause functional deterioration due to excessive exercise restriction.

As a countermeasure to the above-mentioned problems, a restraint cover for preventing self-removal (see Patent Document 1) devised so as to reduce the pain of the patient, allow a certain amount of exercise, and suppress the self-removal act. Alternatively, a joint suppressor (see Patent Document 2) that can prevent the patient from self-removal by fixing the wrist joint in a dorsiflexed state at a predetermined angle without restraining the movement of the entire arm. Is used.

However, it is not a practically acceptable response to uniformly restrain all patients in various states by physical restraint means, and patient monitoring work for the purpose of preventing self-removal is performed in many medical settings. It was a heavy burden as an indispensable and unmissable work.

As a means of reducing the burden of patient monitoring work for the purpose of preventing self-removal, various alarms are emitted by various mechanisms when a "medical clothing device" such as a catheter is removed without permission. Devices and the like (Patent Documents 3 and 4) have been proposed.

In addition, there are already many general-purpose monitoring systems (Patent Document 5) that notify the observer of the analysis result of the monitoring image of the patient to be monitored, including the patient wearing the medical equipment. It is used in the medical field.

Here, much of the self-extraction that endangers the patient's life is caused by the patient's own actions in a lying position, such as on a nursing bed. However, it has been difficult to recognize a patient in a lying state as a person to be monitored with sufficient accuracy by conventional image recognition means including the medical monitoring system disclosed in Patent Document 5.

Due to the above-mentioned difficulty in image recognition and the like, the automation of the "self-extraction" monitoring work has not yet progressed sufficiently. Monitoring that can appropriately identify the person to be monitored from the monitoring image, automatically recognize the operation related to self-extraction, and automatically notify the monitor quickly and accurately when a response is required. Means were urgently needed in many medical institutions.

Japanese Unexamined Patent Publication No. 2000-42016 JP-A-2017-00651 Registered Utility Model Gazette No. 3131849 Japanese Unexamined Patent Publication No. 2018-138094 Japanese Unexamined Patent Publication No. 2014-7685

The present invention has been made in view of the above circumstances, and appropriately identifies the monitored person from the monitored image, automatically recognizes the operation related to self-extraction, and monitors when it is necessary to take measures. The purpose of the present invention is to provide a "self-removal monitoring means for medical equipment" that can automatically give a quick and accurate notification to a person.

The present invention solves the above-mentioned problems by the following solutions.

(1) A self-removal monitoring system for medical equipment, which is a coordinate conversion that rotates an imaging unit that photographs a person to be monitored and a coordinate axis set in a surveillance image captured by the imaging unit at an arbitrary angle. Of the coordinate conversion unit that performs the above, the monitoring target identification unit that identifies the monitoring target person in the monitoring image, and the medical equipment worn on the body of the monitoring target person specified by the monitoring target identification unit. The combination of the mounting position specifying unit that specifies the mounting position, the motion recognition unit that recognizes the motion of the monitored person specified by the monitoring target identifying section, the mounting position, and the motion is a specific combination. A response request determination unit that determines the necessity of response depending on whether or not the response is required, a response request notification output unit that outputs a response request notification when the response necessity determination unit determines that a response is required, and a response request notification output unit. Self-extraction monitoring system equipped with.

The invention of (1) is a coordinate that performs coordinate conversion for rotating a coordinate axis set in a monitoring image in a self-extraction monitoring system including a monitoring target identification unit, a mounting position identification unit, and a motion recognition unit. By providing a conversion unit as well, the recognition accuracy of the monitored person who is not in an upright state, such as a patient in a lying state, is improved. As a result, in hospitals and the like, the occurrence or risk of occurrence of "self-extraction" can be quickly recognized with sufficient accuracy and notified to the observer, and the monitoring burden on the observer can be reduced.

(2) The response request notification output unit has a monitoring screen for displaying the monitoring image, and the mounting position specifying unit allows the observer to tap the monitoring screen or click on the monitoring screen. The self-removal monitoring system according to (1), wherein the position specified by the above is specified as a mounting position of the medical equipment.

The invention of (2) further provides, in the invention of (1), a means for the observer to specify the wearing position of the medical equipment worn on the body of the monitored person by a simple operation. did. As a result, for example, even when the mounting position of the medical dressing device is hidden by a futon or the like and mechanical recognition in the image is difficult, the mounting position of the medical dressing device can be accurately specified. , The occurrence or risk of "self-extraction" can be quickly recognized with sufficient accuracy and notified to the observer.

(3) The motion recognition unit includes a skeleton extraction unit that extracts the skeleton of a monitored person composed of skeleton lines connecting a plurality of feature points, and the three-dimensional space of the feature points extracted by the skeleton extraction unit. (1) or (2), which includes a feature point position specifying portion that specifies a position within the inside, and recognizes the operation of the monitored person based on information related to the fluctuation of the position of the feature point. Self-extraction monitoring system described in.

In the invention of (3), in the invention of (1) or (2), a plurality of feature points of the monitored person are connected by using the motion recognition unit, for example, an image analysis means such as "OpenPose" described later. By extracting the skeleton of the image and analyzing the position and speed of each of these feature points, the motion of the monitored person can be recognized. According to this, various movements that the observer should recognize can be recognized with higher accuracy regardless of the body shape of the monitored person, and the display of the monitored image can be displayed only by the skeleton information. By doing so, it is possible to set the monitoring mode in consideration of the privacy of the monitored person.

(4) The motion recognition unit further includes a line-of-sight direction detection unit that detects the line-of-sight direction of the monitored person in the surveillance image, and the response necessity determination unit determines the mounting position in the line-of-sight direction. The self-extraction monitoring system according to any one of (1) to (3), which determines that it is necessary to take measures when the relative fluctuation amount and fluctuation speed with respect to the visual direction are a predetermined fluctuation amount and fluctuation speed. ..

The invention of (4) further includes, in any of the inventions (1) to (3), means capable of detecting the line-of-sight direction of the person to be monitored. As a result, it detects the pre-movement that is likely to lead to a removal accident, which is difficult to recognize in advance only by analyzing the movement of the trunk and limbs of the monitored person, and notifies the monitor of information on the risk of self-removal. can do.

(5) A bleeding detection unit that detects the outflow of blood from a change in color around the mounting position in the monitoring image is further provided, and the bleeding detection unit has a predetermined amount or more of blood in the response necessity determination unit. The self-extraction monitoring system according to any one of (1) to (4), which determines that it is necessary to take measures even when the outflow of blood is detected.

The invention of (5) further includes, in any of the inventions (1) to (4), a means for detecting bleeding around the wearing position of the medical equipment. As a result, it is possible to notify the observer of any removal accident without delay, including the removal accident that is difficult to recognize in advance only by the motion analysis of the monitored person.

(6) This is a self-removal monitoring method for medical equipment, in which a monitoring shooting step in which the photographing unit photographs a monitored person and a coordinate axis set in the monitoring image captured by the photographing unit are rotated at an arbitrary angle. The coordinate conversion step, the monitoring target identification step in which the monitoring target identification unit identifies the monitoring target person in the monitoring image captured in the monitoring shooting step, and the monitoring in which the mounting position specifying unit is specified in the monitoring target identification step. A mounting position specifying step for specifying the mounting position of the medical equipment worn on the body of the target person, and an operation in which the motion recognition unit recognizes the motion of the monitored target person specified in the monitoring target specifying step. A response request determination step and a response request notification output in which the recognition step, the response necessity determination unit determines the necessity of correspondence depending on whether or not the combination of the mounting position and the operation is a specific combination, and the response request notification output. The unit includes a response request notification output step that outputs a response request notification when it is determined in the response necessity determination step that a response is required, and in the monitoring target identification step, the coordinate axes are A self-extraction monitoring method for identifying a monitoring target that is not in an upright state in the monitoring image during rotation.

The invention of (6) is a self-extraction monitoring method including a monitoring target identification step, a mounting position identification step, and an operation recognition step, and further, in the monitoring target identification step, the coordinates in the monitoring image are rotated. By allowing the patient to lie down, the recognition accuracy of the patient or the like is improved. As a result, in hospitals and the like, the occurrence or risk of occurrence of "self-extraction" can be quickly recognized with sufficient accuracy and notified to the observer, and the monitoring burden on the observer can be reduced.

(7) The response request notification output unit has a monitoring screen for displaying the monitoring image, and in the mounting position specifying step, the observer taps the monitoring screen or clicks on the monitoring screen. The self-removal monitoring method according to (6), wherein the position specified by the operation is specified as the mounting position.

In the invention of (7), in the invention of (6), the observer performs a process of designating the wearing position of the medical equipment worn on the body of the monitored person by a simple operation. As a result, for example, even when the mounting position of the medical dressing device is hidden by a futon or the like and mechanical recognition in the image is difficult, the mounting position of the medical dressing device can be accurately specified. , The occurrence or risk of "self-extraction" can be quickly recognized with sufficient accuracy and notified to the observer.

(8) In the motion recognition step, a skeleton extraction process for extracting the skeleton of the monitored person composed of skeleton lines connecting a plurality of feature points and the feature points extracted by the skeleton extraction process are described. The feature point position identification process for specifying the position in the three-dimensional space is performed, and the movement of the monitored person is recognized based on the information related to the fluctuation of the position of the feature point, (6) or (7). ) For self-extraction monitoring method.

In the invention of (8), in the invention of (6) or (7), a plurality of feature points of the monitored person are connected by using the motion recognition step, for example, an image analysis means such as "OpenPose" described later. It is a process that can recognize the movement of the monitored person by extracting the skeleton that is formed and analyzing the position and speed of each of these feature points. According to this, various movements that the observer should recognize can be recognized with higher accuracy regardless of the body shape of the monitored person, and the display of the monitored image can be displayed only by the skeleton information. By doing so, it is possible to set the monitoring mode in consideration of the privacy of the monitored person.

(9) In the motion recognition step, a line-of-sight direction detection process for detecting the line-of-sight direction of the monitored person in the surveillance image is further performed, and in the response necessity determination step, the line-of-sight direction is said. The method according to any one of (6) to (8), wherein it is determined that a response is required when the relative fluctuation amount and fluctuation speed with respect to the direction in which the mounting position is visually observed are the predetermined fluctuation amount and fluctuation speed. Self-extraction monitoring method.

The invention of (9) is the process of further detecting the line-of-sight direction of the monitored person in any of the inventions (6) to (8). As a result, it detects the pre-movement that is likely to lead to a removal accident, which is difficult to recognize in advance only by analyzing the movement of the trunk and limbs of the monitored person, and notifies the monitor of information on the risk of self-removal. can do.

(10) In the motion recognition step, a bleeding detection process for detecting the outflow of blood from the change in color around the mounting position in the monitoring image is further performed, and in the response necessity determination step, the above-mentioned The self-extraction monitoring method according to any one of (6) to (9), wherein it is determined that a countermeasure is required when the motion recognition unit detects an outflow of blood exceeding a predetermined amount.

The invention of (10) is the process of further performing the process of detecting bleeding around the wearing position of the medical clothing device in any of the inventions (6) to (9). As a result, it is possible to notify the observer of any removal accident without delay, including the removal accident that is difficult to recognize in advance only by the motion analysis of the monitored person.

According to the present invention, the monitored person is appropriately identified from the monitored image, the operation related to self-extraction is automatically recognized, and when a response is required, the monitor is automatically notified promptly and accurately. It is possible to provide "self-removal monitoring means for medical equipment" that can be performed in the above.

It is a figure which shows the outline of the structure and operation of the self-removal monitoring system of the medical equipment. It is a block diagram which provides the explanation of the basic structure of the self-removal monitoring system of the medical equipment. It is a flowchart which shows the operation flow of the self-removal monitoring method of the medical equipment. It is a schematic diagram provided for the operation explanation of the coordinate conversion processing in the self-extraction monitoring system and the monitoring method of the medical equipment. It is a schematic diagram provided for the operation explanation of the wearing position identification processing in the self-removal monitoring system and the monitoring method of the medical equipment. It is a schematic diagram which provides the operation explanation at the time of performing the attachment position identification process in the self-removal monitoring system and the monitoring method of the medical equipment by the manual operation by the observer. It is a drawing which shows typically an example of the mode of detecting the motion and the line-of-sight direction of the monitored person recognized by the motion recognition unit which constitutes the self-removal monitoring system of the medical wear device of this invention. An example of a storage table in a storage device for storing evaluation criteria referred to for determining the necessity of response in the response necessity determination unit constituting the self-removal monitoring system for medical equipment. It is a drawing which shows.

<Self-removal monitoring system>
[overall structure]
As shown in FIG. 1, in the self-extraction monitoring system (self-extraction monitoring system 100) of the present invention, first, the monitored person staying area (hospital room) 210 in which the monitored person (patient) 10 is staying in a hospital or the like. The inside of the monitored area such as the above is photographed by the photographing unit (surveillance camera) 110, and by analyzing the photographed surveillance image, the monitored person (patient) 110 performs a specific operation with respect to the wearing position P of the medical equipment (patient). For example, the operation of the monitored person 10 in FIG. 1) is recognized. Then, it is necessary to take measures depending on whether or not the combination of the position information related to the wearing position P of the medical clothing device and the specific operation (for example, the operation of the monitored person 10A in FIG. 1) is a specific combination. No is judged. As a result, when it is determined that the response is necessary, the response request notification is output to the response request notification output unit 130 of the observer stay area 220 or the like, the mobile information processing terminal carried by the observer 20, or the like. .. Then, the observer 20A who has received the response request notification can prevent many medical accidents due to self-removal by promptly starting the necessary response.

FIG. 2 is a block diagram showing the configuration of the self-extraction monitoring system 100. As shown in FIG. 2, the self-extraction monitoring system 100 includes at least a photographing unit 110, an arithmetic processing unit 120, and a response request notification output unit 130.

Further, the arithmetic processing unit 120 includes at least a coordinate conversion unit 122, a monitoring target identification unit 123, a mounting position identification unit 124, an operation recognition unit 125, and a correspondence necessity determination unit 127. However, as will be described later, the coordinate conversion unit 122 can also configure the self-extraction monitoring system 100 as a part of the photographing unit 110. When the photographing unit 110 is provided with the coordinate conversion unit 122, the coordinate conversion unit 122 is not necessarily an indispensable constituent requirement in the arithmetic processing unit 120. It is preferable that the arithmetic processing unit 120 is further configured to include the coordinate setting unit 121, if necessary.

Here, the coordinate conversion unit 122 that performs the coordinate conversion step (S13) for rotating the coordinate axes such as the XY coordinate axes set in the surveillance image by an arbitrary angle is specifically subjected to an image processing program that processes the surveillance image. Can be configured. Alternatively, the coordinate conversion unit 122 is configured by a camera body rotation mechanism or an image sensor rotation mechanism that physically rotates the camera body (or only the image sensor inside the camera body) constituting the photographing unit 110 in a predetermined direction. You can also.

For example, as shown in FIG. 4, the self-extraction monitoring system 100 performs a coordinate conversion step (S13) in which the coordinate axes in the monitoring image are rotated by 90 ° by the coordinate conversion unit 122, so that the head H in the monitoring image Since the trunk B and leg L are located along the horizontal direction (x-axis direction) (in a lying state), it is difficult to extract them as "people" from the surveillance image in an upright state. The monitored person 10 is also identified with high accuracy, as in the case of an upright "person" in which the head H, the trunk B, and the leg L are located in this order along the vertical direction (y-axis direction). be able to.

The self-extraction monitoring system 100 can be configured by arranging the photographing unit 110, the arithmetic processing unit 120, and the response request notification output unit 130 at appropriate distances from each other. However, the configuration of the self-extraction monitoring system of the present invention is not limited to the configuration consisting of such an arrangement. For example, it may be configured as a device in which the photographing unit 110 and the arithmetic processing unit 120 are integrated, or a configuration in which a part of the arithmetic processing unit 120 is mounted in the photographing unit 110.

[Shooting section]
The photographing unit 110 is a so-called surveillance camera. Then, the photographing unit 110 performs a monitoring photographing step (S12) for continuously photographing the monitored area.

The surveillance camera constituting the photographing unit 110 captures the three-dimensional space constituting the monitored area as a two-dimensional image by, for example, providing the coordinate setting unit 121 described later in the arithmetic processing unit 120. It can consist of a monocular camera. In this case, it is not necessary to introduce an expensive 3D camera, and the introduction cost of the self-extraction monitoring system 100 can be significantly reduced.

Further, the photographing unit 110 can be composed of a camera including an image sensor 111, a photographing lens 112, and an image processing unit 113. The image sensor 111 captures an image formed by the photographing lens 112 and sends it to the image processing unit 113. The image processing unit 113 processes the data output from the image sensor 111, converts it into image data, and transmits it to the arithmetic processing unit 120.

As described above, the photographing unit 110 can be configured to include the coordinate conversion unit 122. In this case, the photographing unit 110 has a rotation mechanism for rotating the camera body, a rotation mechanism for rotating only the image sensor 111, or coordinates as arithmetic processing in the image processing unit 113 as a mechanism for performing the coordinate conversion step (S13). Any of the programs for performing the conversion (S13) may be provided.

[Calculation processing unit]
The arithmetic processing unit 120 performs arithmetic processing necessary for monitoring the image data transmitted from the photographing unit 110. The arithmetic processing unit 120 can be configured by using, for example, a personal computer, a tablet terminal, a smartphone, or the like. Alternatively, the arithmetic processing unit 120 can be configured as a dedicated device specialized for monitoring operation. In any of these configurations, the arithmetic processing unit 120 includes hardware such as a CPU, a memory, and a communication unit. Then, the arithmetic processing unit 120 having such a configuration can specifically execute various operations and monitoring methods described below by executing a computer program (monitoring program).

The arithmetic processing unit 120 is connected to the photographing unit 110 so that the image data can be received from the photographing unit 110. This connection can be a wired connection using a dedicated communication cable or a wired LAN connection. Further, the connection is not limited to a wired connection, and may be a connection using various wireless communications such as a wireless LAN, short-range wireless communication, and a mobile phone line. The arithmetic processing unit 120 may not be arranged in the vicinity of the photographing unit 110, but may be arranged in a remote place away from the photographing unit 110.

(Coordinate setting part)
The coordinate setting unit 121 sets the coordinates that can be specified by associating the position corresponding to the floor surface or the ground in the surveillance image captured by the photographing unit 110 with the actual size in the three-dimensional space of the monitored area (S11). )I do. It is also possible to set the above coordinates by equipping the photographing unit 110 with a 3D camera, various distance sensors, etc., and thereby obtaining depth information in the monitored area. In this case, the photographing unit 110 The coordinate setting unit 121 independent of the above is not necessarily an indispensable configuration requirement.

(Coordinate conversion part)
The coordinate conversion unit 122 performs the coordinate conversion step (S13) described above. This coordinate transformation can be performed as an arithmetic process on the image data related to the surveillance image. When the photographing unit 110 is provided with the coordinate conversion function, it is not essential that the arithmetic processing unit 120 is provided with the coordinate conversion unit 122.

(Monitoring target identification part)
The monitoring target identification unit 123 automatically detects an arbitrary patient or the like in the monitoring image captured by the imaging unit 110 during the monitoring operation, and performs a monitoring target identification step (S14) to identify this as a monitoring target person. Do. Further, it is more preferable that the monitoring target identification unit 123 has a function of individually identifying a plurality of monitoring target persons in parallel at the same time. The monitoring target can be identified by various conventionally known image analysis methods, but a machine learning type image recognition device having a neural network (so-called "image recognition device using deep learning technology") is used. It is preferable to configure the monitoring target identification unit 123. As a typical example of the "image recognition device using the deep learning technology", an image analysis device using the "You only view (YOLO)" as an image recognition algorithm can be mentioned.

However, usually, in the image recognition process using a known image recognition algorithm such as "YOLO", the head H (see FIG. 4) is relatively vertically above each part of the body (upright). In many cases, a "person" that is not in the state) cannot be detected as a "person", and the relative position of each part of the body is changing in the "lying state" such as a patient sleeping in a bed. In some cases, the person could not be identified with sufficient accuracy. In the self-extraction monitoring system 100, by performing the coordinate conversion step (S13) of rotating the coordinate axes such as the XY coordinates set in the monitoring image at an arbitrary angle, the monitored person who is not in the upright state in the monitoring image may be monitored. , It is possible to identify with high accuracy. Specifically, in the self-extraction monitoring system 100, the coordinate conversion unit 122 rotates the coordinate axes of the monitoring image to an arbitrary angle to align the Y-axis direction of the XY coordinate axes with the upright direction of the monitoring target. It is possible to prevent a decrease in the cognitive accuracy of the monitored object in the non-upright state in the image.

(Mounting position identification part)
As shown in FIG. 5, for example, the wearing position specifying unit 124 is a medical equipment 11 worn on the body of the monitored person 10 (10a, 10b, 10c) specified by the monitoring target specifying unit 123. The mounting position (S15) for specifying the mounting position P (Pa, Pb, Pc) of the above is performed. The mounting position specifying unit 124 (S15) can be configured by the above-mentioned image analysis device. Thereby, the mounting position P (Pa, Pb, Pc) can be mechanically specified.

Alternatively, the mounting position specifying unit 124 may be a mechanism for specifying a position designated by an arbitrary manual operation by the observer as a mounting position. The arbitrary manual operation is, for example, a tap operation or a click operation on the monitoring screen by the observer. The manual operation for designating the mounting position is, for example, as shown in FIG. 6, a touch panel type monitoring screen displayed on the response request notification output unit (mobile terminal) 130 by the observer with a finger 21. It is a tap operation that taps an arbitrary position in the surveillance image displayed on the surveillance screen on the display screen. In this way, when a function that allows the mounting position P to be specified by a manual operation by the observer is provided, the mounting position specifying unit 124 is provided with a mechanism that specifies the position designated by the function as the mounting position. can do.

(Motion recognition unit)
The motion recognition unit 125 performs motion recognition (S16) for recognizing the motion of the monitored person specified by the monitoring target identification unit 123. Here, the "movement" of the monitored person means the position and movement of each part of the body. In addition, the "position and movement of each part of the body" includes all "positions and movements of the monitored person" that can be grasped by the position changes of the feature points of the skeleton, such as the position change of the monitored person and the change of posture without the position change. "Position and movement of each part of the body" is included. The identification of these "operations" can be performed by various conventionally known image analysis methods.

Further, the motion recognition unit 125 includes a skeleton extraction unit (not shown) that extracts the skeleton of the monitored person composed of skeleton lines connecting a plurality of feature points, and the positions of the feature points extracted by the skeleton extraction unit. It is preferable to further include a feature point position specifying portion (not shown) for specifying the above. The skeleton extraction unit extracts the skeleton of the monitored person, which is composed of skeleton lines connecting a plurality of feature points (see FIG. 7).

The extraction of the skeleton of the monitored person as shown in FIG. 7 can be specifically performed by any of various conventionally known methods or a combination thereof. As an example, the skeleton of a "human" can be extracted from a two-dimensional surveillance image by using a technique called "OpenPose" disclosed in the following document. In this case, the motion recognition unit 125 can recognize the "motion" of the monitored person based on the information related to the fluctuation of the "position in the monitored area three-dimensional space" of the above-mentioned feature points.
"Zhe Cao et al. Realtime Multi-Person 2D Human Pose Estimation using Part Affinity Fields, CVPR 2017"

Further, it is more preferable that the motion recognition unit 125 further includes a line-of-sight direction detection unit (not shown) that detects the line-of-sight direction of the monitored person in the surveillance image. The specific means for detecting the line-of-sight direction is not particularly limited, but for example, as shown in FIG. 7, the skeleton extraction unit and the feature point position identification unit also perform the positions of both ears and nose of the “monitored subject”. In a triangle formed by connecting three corresponding "feature points" for detecting the direction of the line of sight, from the midpoint of the base connecting the points corresponding to the positions of both ears to the point corresponding to the position of the nose. The direction toward a certain apex can be detected as the line-of-sight direction e of the monitored person.

(Bleeding detector)
Further, it is preferable that the arithmetic processing unit 120 further includes a bleeding detection unit 126 that detects the outflow of blood from the change in color around the mounting position P in the monitoring image. As long as it has a function of detecting bleeding, the specific configuration of the bleeding detection unit 126 is not particularly limited, but for example, it is determined that a specific color (red) appears around a specific position (mounting position P) and the determination result is obtained. The bleeding detection unit 126 can be configured by an image analysis program or the like that can output the information related to the above to the response necessity determination unit 127.

(Correspondence necessity judgment unit)
The response necessity determination unit 127 performs the response necessity determination (S17) for determining the necessity of correspondence depending on whether or not the combination of the mounting position P and the operation of the monitored person is a specific combination. Further, when the arithmetic processing unit 120 includes the bleeding detection unit 126, the response necessity determination unit 127 can use the information related to the bleeding as the determination material for the necessity determination.

The response necessity determination unit 127 collates the combination of the mounting position P acquired by the mounting position specifying unit 124 and the motion recognition unit 125 with the motion of the monitored person with the evaluation criteria registered in advance in various storage means. By doing so, it is determined whether or not it is necessary to deal with the monitored person. The response necessity determination unit 127 can be configured by various information processing devices, programs, and the like capable of such simple matching analysis of information.

FIG. 8 shows, in addition to the acquired information related to the combination of the wearing position P and the movement of the monitored person, for storage in the storage device for storing the evaluation criteria for evaluating the acquired information related to bleeding. This is an example of a table. As illustrated in FIG. 8, the occurrence of self-extraction can be recognized in advance or promptly from the acquired information related to the line-of-sight direction, hand movement, and bleeding, and the necessity of countermeasures can be determined.

[Response request notification output section]
The response request notification output unit 130 has a monitoring screen for displaying a monitoring image, and when the response necessity determination unit 127 determines that a response is necessary, the determination result of the response necessity determination unit 127 Is provided as an information output means capable of outputting the image as a response request notification in a format recognizable by the observer 20.

The response request notification output unit 130 is information related to the determination result output by the response necessity determination unit 127, such as a personal computer with a monitor or a small portable information processing terminal that can be easily carried by the observer. Is configured by various information processing terminals capable of appropriately displaying recognizable video, text, audio, etc., or various information display devices including a monitor capable of responding to such a request. be able to.

<Self-extraction monitoring method>
[overall structure]
FIG. 3 is a flowchart showing the flow of the self-extraction monitoring method of the present invention that can be executed by using the self-extraction monitoring system 100.

(Coordinate setting step)
In the coordinate setting step (S11), the coordinate setting unit 121 performs the coordinate setting process. The coordinate setting process performed in the coordinate setting step (S11) preferably means that the coordinate setting unit 121 positions a position corresponding to the floor surface or the ground in the surveillance image captured by the photographing unit 110 in the monitored area three-dimensional space. This is a process of setting coordinates that can be specified in association with the actual dimensions, that is, three-dimensional coordinates that also have depth information about the monitored area.

The coordinate setting step (S11) is a preliminary preparation for monitoring the monitoring area, and the actual monitoring as the actual operation is started by the subsequent steps. In other words, the coordinate setting step (S11) may be performed at least once after the imaging unit 110 is installed, prior to the start of the actual operation of the monitoring. For example, if the above coordinates are set appropriately when the self-extraction monitoring system 100 is installed, the system will operate unless there is a particular change in the shooting conditions of the monitoring image, such as a change in the arrangement of the shooting unit 110. It is possible to eliminate the need to set the coordinates again inside.

In addition, when the photographing unit 110 has a distance measuring unit that measures the distance by irradiating infrared rays or the like, or a configuration corresponding to the distance measuring unit, the actual distance measuring unit corresponds to the background of the monitored image from the photographing unit. The distance to the point is actually measured, and based on the measured value, the coordinates associated with the actual position (actual size, actual distance) in the monitored area three-dimensional space can be set as described above.

(Surveillance shooting step)
In the surveillance shooting step (S12), the shooting unit 110 performs surveillance shooting. Here, in the surveillance shooting, still images are continuously shot at predetermined intervals, and a monitoring operation described later is performed as a series of shot images. However, by making the shooting intervals very short, substantially It can also be regarded as performing a monitoring operation as a movie shooting.

(Coordinate conversion step)
In the coordinate conversion step (S13), the coordinate conversion unit 122 performs coordinate conversion to rotate the coordinate axes set in the monitoring image captured by the photographing unit 110. This coordinate transformation can be performed by rotating the coordinate axes as an arithmetic process as a process in software. Further, the coordinate conversion step (S13) can be performed by rotating the monitoring image itself while keeping the coordinate axes fixed. The rotation of the surveillance image can be performed by converting the image data as an arithmetic process as a process in the software, and as a simpler implementation method, a surveillance camera constituting the image pickup unit 110 or a surveillance camera thereof is configured. It can also be performed by actually rotating the image sensor.

By continuously rotating the monitoring image 360 ° in the coordinate conversion step (S13), the "person" in the monitoring image in all posture states such as standing, sitting, and lying down in the monitoring target identification step (S14). Can be identified with equal high accuracy. For example, the coordinate conversion unit 122 continuously performs coordinate conversion in which the coordinate axes in the monitoring image are sequentially rotated by 360 °, so that the monitored person 10 in the non-upright state shown in FIG. 4 (a) is also monitored. When the coordinate axes are rotated by 90 °, it can be identified with high accuracy in the same way as a "person" in an upright state.

(Monitoring target identification step)
In the monitoring target identification step (S14), the monitoring target identification unit 123 determines whether or not the monitoring target person in the monitoring image captured by the photographing unit is detected and specified. In particular, for the monitored person who is in a state other than the upright state in the monitored image, the monitored target that is not in the upright state in the monitored image is specified during the rotation of the coordinate axes by the coordinate conversion process described above. When the monitored person is detected and specified (S14, Yes), the process proceeds to the mounting position specifying step (S15), and when the monitored person is not detected (S14, No), the monitoring shooting step (S12) is performed. ) And continue the monitoring operation.

Specifically, the detection of the monitored person can be performed by any of various conventionally known methods, or a combination thereof. For example, "people" in the monitoring area can be extracted by background subtraction. This background subtraction is a known technique, and by taking the difference between the image data acquired by the surveillance camera 24 and the background image of the surveillance area A acquired in advance, a moving monitoring target is extracted. It is a technology.

In the present invention, it is often necessary to monitor the person to be monitored while lying on a bed or the like. In many conventional image recognition programs and image recognition devices using the programs, the positional relationship of each part of the body in an upright state, that is, an upright state is used as a feature point for recognition. Therefore, when the monitored person is in a non-upright state, that is, when the positional relationship between the body of the head and the legs (vertical relationship) is different from that in the upright state, the recognition accuracy may be significantly reduced. .. However, in the coordinate conversion step (S13), by appropriately rotating the coordinate axes, even the monitored person lying on the bed can be recognized with high accuracy like a human being in an upright state. ..

Further, for the identification of the detected monitored person, the combination of the present invention and the image recognition technology using deep learning, which has been attracting attention in the field of image recognition in recent years for the dramatic improvement of the recognition rate, is effective. By combining with such an image recognition technique, it is possible to automatically detect and identify an individual monitored person in a monitored image with an extremely high recognition accuracy rate. It is also possible to classify and recognize a large number of monitoring targets by gender or age, and simultaneously detect and identify them.

The image recognition technology using deep running is disclosed below, for example.
"Deep Learning and Image Recognition, Operations Research"
(Http: //www.orsj.o.jp/archive2/or60-4/or60_4_198.pdf)

(Mounting position identification step)
In the wearing position specifying step (S15), the wearing position specifying unit 124 specifies the wearing position of the medical equipment worn on the body of the monitored person specified in the monitoring target specifying step (S14). Specifically, for example, as shown in FIG. 5, the wearing position P (Pa, Pb, Pc) of the medical equipment 11 worn on the body of the monitored person 10 (10a, 10b, 10c). The mounting position is specified. The identification of the mounting position can be performed mechanically by various image analysis means described above, or the position specified by an arbitrary manual operation by the observer is specified as the mounting position. It can also be done by.

The arbitrary manual operation by the observer for performing the mounting position specifying step (S15) is, for example, a tap operation or a click operation on the monitoring screen by the observer. By performing the mounting position identification step (S15) manually by the observer, medical treatment is performed even when the mounting position of the medical equipment is hidden by a futon or the like and mechanical recognition in the image is difficult. It is possible to accurately identify the mounting position of the equipment.

(Motion recognition step)
In the motion recognition step (S16), the motion recognition unit 125 recognizes the motion of the monitored person specified in the monitoring target specifying step (S14). This motion recognition can be performed by various known image analysis methods. The image analysis method for performing the motion recognition step S16 is not limited to a specific method, but the skeleton extraction unit for extracting the skeleton of the monitored person composed of skeleton lines connecting a plurality of feature points and the skeleton extraction unit It is preferable to use the motion recognition method performed by the motion recognition unit 125 including the feature point position specifying unit for specifying the position of the extracted feature points in the three-dimensional space. Hereinafter, an embodiment in which the motion recognition step S16 is performed by the motion recognition unit 125 including the skeleton extraction unit and the feature point position specifying unit will be specifically described.

FIG. 7 is a diagram showing a state in which the motion (position and movement of each part of the body) of the monitored person in the monitored area space is recognized by the motion recognizing unit 125 including the skeleton extraction unit and the feature point position specifying unit. Is. Here, the posture and the line-of-sight direction e of the monitored person 10 lying on the bed are recognized by the motion recognition unit 125. In this way, the motion recognition unit 125 including the skeleton extraction unit can extract the skeleton 10'of the monitored person 10. Here, in the present specification, the "skeleton" of the monitored person is a linear figure formed by connecting a plurality of feature points of the monitored person.

The extraction of the skeleton of the monitored person as shown in FIG. 7 can be specifically performed by any of various conventionally known methods or a combination thereof. As an example, the skeleton of a "human" can be extracted from a two-dimensional surveillance image by using a technique called "OpenPose" disclosed in the following document.
"Zhe Cao et al. Realtime Multi-Person 2D Human Pose Estimation using Part Affinity Fields, CVPR 2017"

For example, by using the above-mentioned "OpenPose", it is possible to extract a skeleton formed by connecting a plurality of feature points of a monitored person. Then, the "position in the monitored area space" of each of these feature points is specified by using, for example, the coordinates preset by the above coordinate setting step, and the positions and velocities of all the specified feature points are specified. By analyzing the above, it is possible to recognize all movements (positions and movements of each part of the body) of the monitored person including the direction of the line of sight.

(Correspondence necessity judgment step)
In the response necessity determination step (S17), the response necessity determination unit 127 determines whether or not the combination of the wearing position of the medical equipment in the monitored person and the operation of the monitored person is a specific combination. Judgment of necessity of correspondence is performed. Further, when the arithmetic processing unit 120 includes the bleeding detection unit 126, the response necessity determination unit 127 can use the information related to the bleeding as the determination material for the necessity determination.

The response necessity determination is made by collating the combination of the mounting position P acquired by the mounting position specifying unit 124 and the motion recognition unit 125 with the motion of the monitored person with the evaluation criteria registered in advance in various storage means. Will be done. As illustrated in FIG. 8, the occurrence of self-extraction can be recognized in advance or promptly from the acquired information related to the line-of-sight direction, hand movement, and bleeding, and the necessity of countermeasures can be determined.

In the response necessity determination step (S17), when the response necessity determination unit 127 determines that the response is necessary from the operation of the monitored person (S17, Yes), the response request notification output step (S18) If it is determined from the operation of the monitored person that no action is required (S17, No), the process returns to the monitoring shooting step (S12).

(Response request notification output step)
In the response request notification output step (S18), when the arithmetic processing unit 120 determines that it is necessary to respond to the monitored person, the response request notification is output to the response request notification output unit 130. The format of the response request notification is not limited to a specific format as long as the observer 20 can identify which monitoring target person needs to respond. A box with a specific color that distinguishes the monitored person from other targets, skeletal information of the monitored person who can recognize the state of operation, and a sign indicating that a response is required (blinking light, warning sound, etc.) ) Etc. are preferably output from the response request notification output unit 130 held by the observer 20.

In the response request notification output step (S18), the arithmetic processing unit 120 determines whether or not to end the monitoring. The end of monitoring is determined to be the end when, for example, an instruction to end monitoring is input. When it ends (S19, Yes), the monitoring operation ends, and in other cases (S19, No), the process returns to the monitoring shooting step (S12) and the monitoring is continued.

100 Self-extraction monitoring system 110 Imaging unit (surveillance camera)
111 Image sensor 112 Imaging lens 120 Arithmetic processing unit 121 Coordinate setting unit 122 Coordinate conversion unit 123 Monitoring target identification unit 124 Mounting position identification unit 125 Motion recognition unit 126 Correspondence necessity determination unit 130 Correspondence request notification output unit (mobile terminal)
210 Monitored person stay area (hospital room)
220 Observer stay area 10 (10A) Observer 11 Medical equipment P Wearing position 20 Observer S11 Coordinate setting step S12 Surveillance shooting step S13 Coordinate conversion step S14 Monitoring target identification step S15 Wearing position identification step S16 Motion recognition step S17 Correspondence necessity judgment step S18 Correspondence request notification output step

Claims (10)

A self-removal monitoring system for medical equipment
The shooting department that shoots the person to be monitored,
A coordinate conversion unit that performs coordinate conversion that rotates the coordinate axes set in the surveillance image captured by the imaging unit at an arbitrary angle, and a coordinate conversion unit.
A monitoring target identification unit that identifies a monitoring target person in the monitoring image,
A mounting position specifying unit that specifies the mounting position of the medical equipment worn on the body of the monitored person specified by the monitoring target specifying unit,
An operation recognition unit that recognizes the operation of the monitored person specified by the monitoring target identification unit,
A response necessity determination unit that determines the necessity of correspondence depending on whether or not the combination of the mounting position and the operation is a specific combination, and
A response request notification output unit that outputs a response request notification when it is determined by the response necessity determination unit that a response is required is provided.
Self-extraction monitoring system. The response request notification output unit has a monitoring screen for displaying the monitoring image.
The mounting position specifying unit identifies a position designated by the observer by a tap operation on the monitoring screen or a click operation on the monitoring screen as a mounting position of the medical equipment.
The self-extraction monitoring system according to claim 1. The motion recognition unit includes a skeleton extraction unit that extracts the skeleton of a monitored person composed of skeleton lines connecting a plurality of feature points, and a position of the feature points extracted by the skeleton extraction unit in the three-dimensional space. The movement of the monitored person is recognized based on the information related to the fluctuation of the position of the feature point, which includes the feature point position specifying unit for specifying the feature point.
The self-extraction monitoring system according to claim 1 or 2. The motion recognition unit further includes a line-of-sight direction detection unit that detects the line-of-sight direction of the monitored person in the surveillance image.
The response necessity determination unit determines that a response is necessary when the relative fluctuation amount and fluctuation speed in the line-of-sight direction with respect to the visual direction of the mounting position are a predetermined fluctuation amount and fluctuation speed.
The self-extraction monitoring system according to any one of claims 1 to 3. A bleeding detection unit that detects blood outflow from a change in color around the mounting position in the monitoring image is further provided.
The response necessity determination unit determines that a response is necessary even when the bleeding detection unit detects an outflow of blood in a predetermined amount or more.
The self-extraction monitoring system according to any one of claims 1 to 4. It is a self-removal monitoring method for medical equipment.
A surveillance shooting step in which the shooting department shoots the monitored person,
A coordinate conversion step that rotates the coordinate axes set for the surveillance image captured by the imaging unit at an arbitrary angle,
The monitoring target identification unit identifies the monitoring target person in the monitoring image captured in the monitoring shooting step, and the monitoring target identification step.
The wearing position specifying step specifies the wearing position of the medical equipment worn on the body of the monitored person specified in the monitoring target specifying step, and the wearing position specifying step.
A motion recognition step in which the motion recognition unit recognizes the motion of the monitored person specified in the monitoring target identification step, and a motion recognition step.
A response necessity determination step in which the response necessity determination unit determines the necessity of correspondence depending on whether or not the combination of the mounting position and the operation is a specific combination.
A response request notification output step that outputs a response request notification when the response request notification output unit determines that a response is required in the response necessity determination step,
Including,
In the monitoring target identification step, a self-extraction monitoring method for identifying a monitoring target that is not in an upright state in the monitoring image while the coordinate axes are rotating. The response request notification output unit has a monitoring screen for displaying the monitoring image.
In the mounting position specifying step, a position designated by the observer by a tap operation on the monitoring screen or a click operation on the monitoring screen is specified as the mounting position.
The self-extraction monitoring method according to claim 6. In the motion recognition step, a skeleton extraction process for extracting the skeleton of the monitored person composed of skeleton lines connecting a plurality of feature points and the three-dimensional space of the feature points extracted by the skeleton extraction process. The feature point position specifying process for specifying the position in the inside is performed, and the operation of the monitored person is recognized based on the information related to the fluctuation of the position of the feature point.
The self-extraction monitoring method according to claim 6 or 7. In the motion recognition step, a line-of-sight direction detection process for detecting the line-of-sight direction of the monitored person in the monitoring image is further performed.
In the response necessity determination step, it is determined that correspondence is necessary when the relative fluctuation amount and fluctuation speed in the line-of-sight direction with respect to the visual direction of the mounting position are a predetermined fluctuation amount and fluctuation speed. ,
The self-extraction monitoring method according to any one of claims 6 to 8. In the motion recognition step, a bleeding detection process for detecting the outflow of blood from a change in color around the mounting position in the monitoring image is further performed.
In the response necessity determination step, when the motion recognition unit detects the outflow of blood in a predetermined amount or more, it is determined that the response is necessary.
The self-extraction monitoring method according to any one of claims 6 to 9.

Images