TW202309777A - Energy-efficient detection of abnormal passenger behavior in elevator - Google Patents

Abstract

A video-based method of abnormal passenger behavior detection in an elevator car is provided, and energy saving could be achieved accordingly. In short, firstly, videos shot by a camera in the elevator car are analyzed in a less computing-resource consuming manner. Then only when there are more passengers in the elevator, especially increasing from no passenger to at least one passenger, videos shot at this moment are analyzed in a more computing-resource consuming manner, e.g., by applying a human pose estimation, for the purpose of abnormal passenger behavior detection.

Description

Abnormal behavior detection of passengers in elevators that can save energy consumption

Generally speaking, the present invention relates to a method and an elevator device for detecting abnormal behavior of passengers based on video images in an elevator car. More specifically, for example, the method and the elevator device proposed in the specific embodiments of the present invention can take different analyzes on the video images in the elevator car correspondingly according to the increase or decrease of the number of passengers in the elevator car. Detect abnormal passenger behavior in the elevator car and save energy.

Through the method of image recognition, abnormal behaviors of people in various fields can be automatically identified, and the cost of manual monitoring is saved, which saves time and is fast and effective. For this, reference can be made to, for example, CN102241349B, CN111507185B, CN108805093B, and the Republic of China Patent Application No. 110127653 belonging to the same applicant.

The advantages of this automatic identification are more significant in the elevator environment. Since the elevator car is a confined space, it is not easy to be found if passengers feel unwell or any emergency happens in it. Although there are generally emergency call buttons in elevators, if the situation is serious, or especially when the passengers are elderly or young children. It is also not necessarily possible to activate the emergency call button to communicate with service personnel and obtain assistance.

To judge whether the passenger's behavior is abnormal in the elevator car or other specific spaces, it can be judged by the passenger's body posture. For example, in the case of an elevator, cameras are generally installed to capture video images of passengers in the elevator car. Further use of the captured video images for Human Pose Estimation (Human Pose Estimation) or other AI (Artificial Intelligence, AI) image analysis can further determine the abnormal body posture and behavior of passengers.

This case first realized that artificial intelligence has powerful functions. As mentioned above, through human body posture estimation or other similar methods of image analysis using artificial intelligence, it is possible to judge whether the behavior of passengers in elevator cars or other specific spaces abnormal. But artificial intelligence often involves huge calculations and is therefore very energy-intensive.

And when artificial intelligence is used to judge whether the passenger's behavior is abnormal in the elevator car, the problem of energy consumption is even more serious. Generally speaking, elevators run non-stop all day, even year-round. For the sake of passenger safety, the cameras in the elevators are also continuously shooting video images in the elevator cabins in real time throughout the day. It is conceivable that the video image data accumulated throughout the day will be quite large, and if all of these video image data are to be analyzed using artificial intelligence in real time, the energy consumption will be considerable, and it will also affect the hardware requirements. The requirements must also be strict. For example, the biggest problem with energy consumption is that no matter what kind of energy consumption is used, it will be converted into waste heat, which must be discharged to allow the system to operate normally.

In view of this, one of the characteristics of this case is that according to the increase or decrease of the number of passengers in the elevator car, different methods of sampling and processing can be adopted for the obtained video images in the elevator car, so as to save computing resources, energy, or Even communication bandwidth. For example, when there is no one in the elevator car, only a simple and fast analysis mode can be used. The purpose is only to confirm that the elevator car is still unoccupied, and it is not necessary to judge whether there are abnormal behaviors of passengers. Only when there are passengers in the elevator car, the complex analysis mode is used, and artificial intelligence or methods that consume more computing resources and energy are used to perform sampling and image content analysis to detect abnormal behavior of passengers.

In one embodiment, this case proposes a method for detecting abnormal behavior of passengers based on video images in an elevator car, wherein a camera device is fixedly installed in the elevator car and connected to a control module, and the camera device takes pictures of the The video image in the elevator car is provided to the control module for analysis. The method includes: the control module analyzes the video image provided by the camera device in a first analysis mode; and according to the number of passengers in the elevator car Change, the control module analyzes the video image provided by the photographing device in a second analysis mode, and thereby judges that the behavior of the passengers in the elevator car is abnormal. .

In addition, in other embodiments, the elevator device proposed in this case has a camera device and a control module, which are connected to the camera device to receive video images provided by the camera device, and perform the above method.

The features, advantages, or similar expressions mentioned in this specification do not mean that all the features and advantages that can be realized by the present invention should be included in any single embodiment of the present invention. Rather, it should be understood that the expressions about features and advantages mean that specific features, advantages, or characteristics described in conjunction with specific embodiments are included in at least one specific embodiment of the present invention. Therefore, the discussion of the features and advantages, and similar expressions in this specification are related to the same specific embodiment, but not necessarily.

These features and advantages of the present invention will become more apparent by referring to the following description and the appended claims or using the embodiments of the present invention as mentioned below.

References to "a specific embodiment" or similar expressions in this specification mean that the specific features, structures, or characteristics described in conjunction with the specific embodiment are included in at least one specific embodiment of the present invention. Therefore, in this specification, the occurrences of "in a specific embodiment" and similar expressions do not necessarily refer to the same specific embodiment.

＜System Architecture＞

Fig. 1 is a system architecture diagram showing an elevator device 100 in an embodiment of the present invention, to illustrate the basic operation and components of the elevator device 100, but it should be understood that the purpose of the following description is an example, and unnecessary details are omitted for simplicity. However, it should be noted that although the elevator device 100 is used for illustration below, the present invention can also be applied to other places similar to the elevator device, such as train carriages or even some stores.

As shown in FIG. 1 , the elevator car 1 and the counterweight 3 of the elevator device 100 are suspended on the hoistway 2 by steel cables 4 . The steel cable 4 is wound on the traction sheave 6 of the traction machine. When the traction sheave 6 rotates, the friction force between the traction sheave 6 and the steel cable 4 drives the elevator car 1 to rise or fall. The traction machine includes an electric motor and a braking device in addition to the traction sheave 6 . The motor rotates and stops the traction sheave 6 . The braking device keeps the traction sheave 6 from rotating, and maintains the traction sheave 6 in a static state, and the elevator car 1 is also stationary thereupon.

In addition, a control module 10 and a communication module 11 are arranged in the machine room 7 .

The control module 10 controls the operation of the elevator device 100, especially the ascending, descending or standing still of the elevator car 1, so as to transport passengers to the destination floor. The control module 10 is connected to the elevator car 1 through a control cable 13 . The transmission and reception of information between the elevator car 1 and the control module 10 is performed through the control cable 13 . However, in other embodiments, the elevator car 1 and the control module 10 can also communicate information through wireless communication.

In this embodiment, the elevator car 1 is provided with, for example, a display 14 , an intercom 15 , a video camera 16 and a weighing device 17 . For example, the control module 10 receives information from the walkie-talkie 15 , the video camera 16 and the scale device 17 through the control cable 13 .

The display 14 is an example of a device for presenting information to passengers. The intercom 15 includes a microphone and a speaker. The sound information obtained by the microphone is output to the control module 10 . The camera 16 photographs, for example, the inside of the elevator car 1 . The image information captured by the camera 16 is output to the control module 10 .

The scale device 17 detects the loaded weight of the elevator car 1 . It is also possible to arrange the scale device 17 at the end of the steel cable 4 . The information of the carrying weight detected by the scale device 17 is output to the control module 10 .

The control module 10 includes circuits such as an input/output interface, a processor, and a memory as its hardware resources. The control module 10 implements the function of controlling the operation of the elevator device 100 through the processor executing the program stored in the memory. The control module 10 can also have a plurality of processors. The control module 10 can also have a plurality of memories. That is to say, the functions of the control module 10 can be realized by cooperative operation of multiple processors and multiple memories, and a part or all of each function can also be realized by hardware.

But it should be known that a part or all of the control module 10 can also be realized through a personal computer, a workstation, or a cloud through a network.

The communication module 11 is a device for the control module 10 to communicate with the outside. The communication module 11 can communicate with the outside through the communication line 21 (which can be a wired or wireless line), so as to transmit information from the elevator device to the outside and receive information from the outside. Generally speaking, the object that the communication module 11 can communicate with is pre-designated, for example, it can be the service center of the elevator device 100 or the management room of the building.

It is also mentioned that the camera 16 can be implemented with reference to the IP Cam used in elevators in the prior art. In addition, the camera 16 and the control module 10 can be connected in a wired or wireless manner, and the present invention is not intended to be limited thereto.

In one embodiment, one or more cameras 16 can include, for example, a Raspberry Pi chip (not shown) and can be used as a client, using the ImageZMQ network communication module to stream the captured frames (frame) in real time (live streaming) ) to the control module 10 as the host, but the present invention is not limited thereto. Regarding the ImageZMQ network communication module, you can refer to the technical document "Transporting OpenCV images via ZMQ" published by the Python Package Index software package repository on its website https://pypi.org/project/imagezmq/, which will not be described in detail in this article .

Because the ImageZMQ network communication module can use the publish-subscribe (Publish/Subscribe) data distribution mode, the publisher continuously obtains and publishes the latest images taken by the camera 16, and the subscriber can only request the current image from the publisher when processing the data. The latest released image, so it can ensure that the image processed every time is the latest without wasting computing resources. In addition, the ImageZMQ network communication module can support multiple clients, so the control module 10 can perform the steps shown in Figure 2 through the video images captured by one or more cameras 16, but only one will be used below The camera 16 is taken as an example for description.

In step 200 : the camera 16 as the client transmits the captured images to the control module 10 as the host in a real-time streaming manner. Generally speaking, in the case of real-time streaming, the frame rate of the camera 16 can usually be set at 24 or 30 fps, that is, 24 or 30 images per second are sent to the control module 10 for analysis. In the initial situation, the sampling rate (sampling rate) in the analysis mode adopted by the control module 10 is first set to be consistent with the frame rate of the camera 16, that is, each frame transmitted by the camera 16 is sampled for subsequent Analysis of image content.

Step 202: In this step, the control module 10 performs simple image analysis on the sampled images to determine whether the elevator car 1 is empty (that is, there are no passengers). The specific way of judging whether the elevator car 1 is empty can refer to existing practices, such as CN101357726A or CN105692376, but generally speaking, this part does not even need to use artificial intelligence to judge, so the control module 10 in step 202 needs Computing resources and energy are quite limited. If step 202 determines that the elevator car 1 is empty, then proceed to step 204 . On the contrary, if the judgment is negative, that is, it is judged that there are passengers in the elevator car 1, then go to step 214 . In another embodiment, in order to avoid misjudgment, the control module 10 can continuously perform step 202 for several times, for example, within a certain time (for example, 5 seconds), and each judgment result shows that the elevator car 1 is empty. Go to step 204 , otherwise go to step 214 .

Step 204: Since it is determined that the elevator car 1 is empty and there are no passengers therein, the control module 10 actually does not need to perform much subsequent processing, and only needs to confirm whether the elevator car 1 is still in an empty state. Therefore, the analysis mode adopted at this time may be simpler than the initial analysis mode in step 200 . For example, in step 204, the sampling frequency of the control module 10 at this time can be greatly reduced, for example, it only needs to sample once every 1 second or every 2 seconds. Compared with the sampling frequency of the control module 10 in step 200, the computing resources and energy consumed by the control module 10 can be greatly reduced. As shown in Figure 2, after reducing the sampling frequency, return to the judgment of step 202 to confirm whether the elevator car 1 is still in an empty state. If the judgment of step 202 is no this time, it means that the elevator car 1 was originally empty. Passengers enter now, and step 214 is performed at this time.

Step 214: Since it is determined that there are passengers in the elevator car 1, the analysis mode of the control module 10 at this time needs to carry out further image analysis to identify the number of people in the elevator car 1, and then analyze each person according to the identified number of people. A passenger performs body posture estimation or other artificial intelligence (AI) image analysis to determine the abnormal body posture and behavior of individual passengers (step 216 ). For example, according to the number of recognized people, the recognized people in the screen can be segmented (crop) into smaller images one by one, and then the skeleton can be detected using 2D Human Pose Estimation , and used as the input of the fall detection model (trained with a single skeleton). In general, processing smaller images also saves computational resources.

In addition, it is worth noting that if the determination of whether the elevator car 1 is empty in step 202 is also performed through image analysis, the determination of step 202 and the identification of the number of passengers in step 214 can be performed in the same step, and when the identified If the number of people is zero, segmentation and skeleton detection will not be performed.

If step 216 determines that the behavior of the passenger is abnormal, the control module 10 can notify the service center of the elevator device 100 through the communication module 11 (step 220), if not, return to step 200 to continue monitoring. In step 214 , human body pose estimation and other image analysis for detecting abnormal passenger behavior can be performed using any suitable method in the prior art, or artificial intelligence, which is not intended to be limited by the present invention. However, in order to ensure that abnormal behaviors of passengers can be detected correctly, the image analysis method adopted in step 214 generally consumes more computing resources and energy, but this is only limited to the results that have to be analyzed when there are passengers in the elevator, in other words , when there is no passenger in the elevator, it will not go to step 214, so computing resources and energy can be effectively saved.

In addition, in an embodiment, as the number of people who recognize the elevator car 1 is different each time in step 214 , the subsequent image analysis method may also be different. For example, when there are multiple passengers in the elevator car 1, the image of one passenger is likely to be blocked by other passengers, so at this time the control module 10 may need to use a more complex artificial intelligence model to perform "multiple Multi-Person 3D Pose Estimation (Multi-Person 3D Pose Estimation)" to detect whether there is abnormal behavior, and therefore consume more computing resources and energy. In contrast, if there is only one passenger in the elevator car 1, there will be no blocking by other passengers, so the control module 10 only needs to perform "Single-Person 3D Pose Estimation (Single-Person 3D Pose Estimation)". Estimation)” to detect whether there is abnormal behavior, and has the opportunity to consume less computing resources and energy than multi-person human pose estimation.

＜Modification 1＞

In a variant implementation, in step 202 , it is determined whether the elevator car 1 is empty, except that the control module 10 performs image analysis on the sampled images as described above, other methods can also be adopted. For example, the scale device 17 shown in FIG. 1 can detect the carrying weight of the elevator car 1, and output the information of the detected carrying weight to the control module 10, so the control module 10 can judge according to the carrying weight Whether or not cab 1 is empty, and even the number of passengers can be estimated. In addition, the walkie-talkie 15 shown in FIG. 1 can also receive the sound of the elevator car 1, and output the detected sound information to the control module 10, so the control module 10 can judge whether the elevator car 1 is empty according to the sound information. .

＜Modification 2＞

In a variant implementation, in addition to the control module 10 lowering the sampling frequency in step 204, the control module 10 can also send a control command to the camera 16 to switch the shooting mode of the camera 16, for example, from the original resolution of 4MP (2688 x 1520) and a frame rate of 30fps is switched to a resolution of 1MP = 1280 x 720 and a frame rate of 10fps to save energy and network bandwidth for subsequent control module 10 on image analysis. Optionally, the control module 10 can also switch the streaming mode between the cameras 16, that is, any parameters used when the cameras 16 want to transmit images to the control module 10, or the control module 10 receives the video from the cameras 16 Adjust any parameters that were used when the image was transferred. For example, although the video camera 16 shoots images with high resolution (such as 4MP), the video is additionally resized (Resized) to convert to a lower resolution (such as 1MP or even higher) when streaming. Low) images are transmitted, which can also save energy and network bandwidth consumed by the follow-up control module 10 for image analysis.

The foregoing disclosure is not intended to limit the invention to the precise forms disclosed or to the particular field of use. Thus, various alternative embodiments and/or modifications of the invention are contemplated in light of the invention, whether explicitly described or implied herein.

The various embodiments provided herein can be implemented using hardware, software, or a combination of hardware and software, and various hardware and software components can be combined to include one or more components of a combination of software and/or hardware, without Deviate from the spirit of the present invention. Where applicable, the order of the various steps described herein may be changed, combined into combined steps, and/or split into sub-steps to provide features described herein.

Although the present invention is disclosed above with various embodiments, it is not intended to limit the scope of the present invention. Anyone with ordinary knowledge in the technical field may make some changes and modifications without departing from the spirit and scope of the present invention. . The scope of protection of the present invention should be defined by the scope of the appended patent application.

1: Elevator car 2: Elevator road 3: Balance weight 4: steel cable 5: Traction machine 6: Traction wheel 7: Machinery Room 8: Motor 9: Brake device 10: Control Modules 11:Communication module 13: Control cable 14: Display 15: walkie-talkie 16: Camera 17: scale device 21: Communication Lines 100: Elevator device 200~220: Steps

For an immediate appreciation of the advantages of the invention, the invention briefly described above shall be referred to in detail by reference to specific embodiments as shown in the accompanying drawings. With the understanding that these drawings depict only typical embodiments of the invention and are therefore not to be considered as limiting the scope of the invention, the invention is illustrated with additional clarity and detail with reference to the accompanying drawings, in which:

Fig. 1 shows the system architecture diagram of the elevator device in the embodiment of the present invention;

Fig. 2 shows a flow chart of a method according to an embodiment of the present invention;

200~220: steps

Claims (10)

A method for detecting abnormal behavior of passengers based on video images in an elevator car, wherein a camera device is fixedly installed in the elevator car and connected with a control module, the camera device captures video images in the elevator car and provides To analyze the control module, the method includes: a. The control module analyzes the video image provided by the camera device in a first analysis mode; and b. Due to the change of the number of passengers in the elevator car, the control module analyzes the video image provided by the camera device in a second analysis mode, and judges that the behavior of the passengers in the elevator car is abnormal. The method according to claim 1, wherein the second analysis mode further includes performing a human body pose estimation, while the first analysis mode does not include human body pose estimation. The method according to claim 2, wherein step b further includes analyzing the video image provided by the photographing device by the control module in response to the passengers in the elevator car from scratch. The method as claimed in claim 1, wherein step b further includes responding to the increase of passengers in the elevator car, the control module analyzes the video image provided by the photographing device in the second analysis mode, wherein the second analysis mode consumes The computing resource of the control module is more than that of the first analysis mode. The method of claim 1, wherein the first analysis mode uses a first sampling frequency, and the second analysis mode uses a second sampling frequency, and the second sampling frequency is different from the first sampling frequency. The method according to claim 5, wherein step b further includes, in response to the increase of the number of passengers in the elevator car, the control module analyzes the video image provided by the photographing device in the second analysis mode, and the second sampling frequency is greater than the first sampling frequency. The method according to claim 1, wherein step a further comprises the control module analyzing the video image provided by the photographing device in the first analysis mode and thereby estimating the number of passengers in the elevator car. The method of claim 1, wherein step a further includes the control module controlling the photographing device to take pictures in a first shooting mode, wherein step b further includes responding to changes in the number of passengers in the elevator car, the control module Shooting with the photographing device in a second shooting mode. The method according to claim 1, wherein step a further includes performing streaming between the control module and the camera device in a first streaming mode, and wherein step b further includes responding to changes in the number of passengers in the elevator car, the Streaming is performed between the control module and the camera device in a second streaming mode. An elevator device comprising: an elevator car; A photographic device fixedly installed in the elevator car to capture video images inside the elevator car; A control module, connected to the camera device to receive video images provided by the camera device, and perform the method according to any one of claims 1-9.

Images