CN112507880A - Model training method, hand washing behavior detection method, device, equipment and medium - Google Patents
Model training method, hand washing behavior detection method, device, equipment and medium Download PDFInfo
- Publication number
- CN112507880A CN112507880A CN202011448943.2A CN202011448943A CN112507880A CN 112507880 A CN112507880 A CN 112507880A CN 202011448943 A CN202011448943 A CN 202011448943A CN 112507880 A CN112507880 A CN 112507880A
- Authority
- CN
- China
- Prior art keywords
- hand washing
- hand
- image
- target
- behavior detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005406 washing Methods 0.000 title claims abstract description 227
- 238000001514 detection method Methods 0.000 title claims abstract description 110
- 238000012549 training Methods 0.000 title claims abstract description 67
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000002372 labelling Methods 0.000 claims abstract description 10
- 230000006399 behavior Effects 0.000 claims description 111
- 238000012545 processing Methods 0.000 claims description 42
- 230000006870 function Effects 0.000 claims description 9
- 238000013527 convolutional neural network Methods 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- 238000005457 optimization Methods 0.000 abstract description 3
- 210000004247 hand Anatomy 0.000 description 19
- 238000010586 diagram Methods 0.000 description 10
- 210000003811 finger Anatomy 0.000 description 8
- 230000009471 action Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 210000001145 finger joint Anatomy 0.000 description 5
- 230000008859 change Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000004898 kneading Methods 0.000 description 3
- 210000003813 thumb Anatomy 0.000 description 3
- 238000005452 bending Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 239000000344 soap Substances 0.000 description 2
- 210000000707 wrist Anatomy 0.000 description 2
- 210000003857 wrist joint Anatomy 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009340 pathogen transmission Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
- 238000009736 wetting Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a model training method, a hand washing behavior detection device, hand washing behavior detection equipment and a hand washing behavior detection medium, wherein the hand washing behavior detection device comprises the following steps: obtaining the coordinates of key points of the human body posture skeleton in each original sample image through a human body posture recognition algorithm; determining the area of the hand in the original sample image according to the coordinates of the key points; according to the area of the hand in the original sample image, intercepting the hand image from the original sample image to form a sample image; and training the initial hand washing behavior detection model according to a sample image library formed by the sample images and the labeling result of each sample image to obtain a target hand washing behavior detection model. In the model training method, on one hand, the required sample data amount is greatly reduced, and the workload required by labeling samples is reduced; on the other hand, the target hand washing behavior detection model trained based on the sample image library does not need secondary optimization, and the model has higher detection accuracy on step classification of hand washing behavior.
Description
Technical Field
The embodiment of the invention relates to the field of image recognition, in particular to a model training method, a hand washing behavior detection device, hand washing behavior detection equipment and a medium.
Background
In hospitals and related places, in order to manage hand hygiene of medical staff, whether hand washing behaviors of the medical staff in a hand washing process meet standard hand washing specifications needs to be detected.
At present, standard hand washing standard propaganda and explanation can be carried out only by using picture propaganda, training explanation and other modes. During the hand washing process of medical personnel, effective detection cannot be realized, so that the problem of pathogen transmission through hands can be caused due to the fact that hand washing behaviors of the medical personnel are not standardized.
Disclosure of Invention
The invention provides a model training method, a hand washing behavior detection device, hand washing behavior detection equipment and a hand washing behavior detection medium, and aims to solve the technical problem that hand washing behavior detection cannot be carried out at present.
In a first aspect, an embodiment of the present invention provides a model training method, including:
acquiring hand washing videos of three angles of each sample acquisition person; wherein the three angles are: the system comprises a left side, a front side and a right side of a sample collector, wherein the height of a camera on the left side of the sample collector is different from that of a camera on the right side of the sample collector;
for each sample collector, carrying out mirror image processing on the hand washing video on the corresponding left side, and carrying out mirror image processing on the hand washing video on the corresponding right side to obtain the hand washing video after mirror image processing;
converting the hand washing video and the hand washing video subjected to mirror image processing into images to form an original sample image library;
obtaining the coordinates of key points of a human body posture skeleton in each original sample image in an original sample image library through a human body posture identification algorithm;
determining the area of the hand in the original sample image according to the coordinates of the key points of the human body posture skeleton in the original sample image;
according to the area where the hand is located in the original sample image, intercepting a hand image from the original sample image to form a sample image;
training an initial hand washing behavior detection model according to a sample image library formed by the sample images and the labeling result of each sample image to obtain a target hand washing behavior detection model; the target hand washing behavior detection model is used for detecting step classification of hand washing behaviors in the target image.
In a second aspect, an embodiment of the present invention provides a hand washing behavior detection method, including:
acquiring coordinates of key points of a human body posture skeleton in a target image through a human body posture identification algorithm;
determining the region where the hand is located in the target image according to the coordinates of the key points of the human body posture skeleton in the target image;
according to the area of the hand in the target image, intercepting the hand image from the target image to form an image to be identified;
inputting the image to be recognized into a target hand washing behavior detection model to obtain step classification of hand washing behaviors in the target image; wherein the target hand washing behavior detection model is obtained by using the model training method according to the first aspect.
In a third aspect, an embodiment of the present invention provides a model training apparatus, including:
the second acquisition module is used for acquiring hand washing videos of three angles of each sample acquisition person; wherein the three angles are: the system comprises a left side, a front side and a right side of a sample collector, wherein the height of a camera on the left side of the sample collector is different from that of a camera on the right side of the sample collector;
the mirror image processing module is used for carrying out mirror image processing on the hand washing video on the corresponding left side and carrying out mirror image processing on the hand washing video on the corresponding right side for each sample collector to obtain the hand washing video after mirror image processing;
the fourth determining module is used for converting the hand washing video and the hand washing video subjected to mirror image processing into images to form an original sample image library;
the first acquisition module is used for acquiring the coordinates of key points of the human body posture skeleton in each original sample image in the original sample image library through a human body posture recognition algorithm;
the first determining module is used for determining the area of the hand in the original sample image according to the coordinates of the key points of the human body posture skeleton in the original sample image;
the second determining module is used for intercepting a hand image from the original sample image according to the area of the hand in the original sample image to form a sample image;
and the third determining module is used for training the initial hand washing behavior detection model according to a sample image library formed by the sample images and the labeling result of each sample image to obtain a target hand washing behavior detection model.
In a fourth aspect, an embodiment of the present invention provides a hand washing behavior detection apparatus, including:
the third acquisition module is used for acquiring the coordinates of key points of the human body posture skeleton in the target image through a human body posture recognition algorithm;
a fifth determining module, configured to determine, according to coordinates of key points of the human body posture skeleton in the target image, an area where the hand is located in the target image;
the sixth determining module is used for intercepting a hand image from the target image according to the area of the hand in the target image to form an image to be identified;
a seventh determining module, configured to input the image to be recognized to a target hand washing behavior detection model, and obtain step classification of hand washing behaviors in the target image; wherein the target hand washing behavior detection model is obtained by using the model training method according to the first aspect.
In a fifth aspect, an embodiment of the present invention further provides a computer device, where the computer device includes:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the model training method as provided in the first aspect or the hand washing behaviour detection method as provided in the second aspect.
In a sixth aspect, embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the program, when executed by a processor, implements the model training method as provided in the first aspect or the hand washing behavior detection method as provided in the second aspect.
The embodiment of the invention provides a model training method, a hand washing behavior detection device, equipment and a medium, wherein the method comprises the following steps: the method comprises the steps of obtaining hand washing videos of three angles of each shot sample collection person, wherein the three angles are as follows: the left side, the front side and the right side of the sample collector, wherein the height of the camera on the left side of the sample collector is different from the height of the camera on the right side of the sample collector; for each sample collector, carrying out mirror image processing on the hand washing video on the corresponding left side, and carrying out mirror image processing on the hand washing video on the corresponding right side to obtain the hand washing video after mirror image processing; converting the hand washing video and the hand washing video subjected to mirror image processing into images to form an original sample image library; obtaining the coordinates of key points of a human body posture skeleton in each original sample image in an original sample image library through a human body posture identification algorithm; determining the area of the hand in the original sample image according to the coordinates of the key points of the human body gesture skeleton in the original sample image; according to the area of the hand in the original sample image, intercepting the hand image from the original sample image to form a sample image; and training the initial hand washing behavior detection model according to a sample image library formed by the sample images and the labeling result of each sample image to obtain a target hand washing behavior detection model, wherein the target hand washing behavior detection model is used for step classification of hand washing behavior detection in the target image. In the model training method, on one hand, hand washing videos at multiple angles can be acquired in a mirror image processing mode, a large number of original sample images can be acquired in a short time, high efficiency is achieved, and the model training efficiency is improved; on the other hand, the hand image is intercepted from the original sample image through a human body posture recognition algorithm to form a sample image, so that the influence on the scale of the sample image library caused by the sex, the height, the age, the clothes and the external scene change of a hand washing user is eliminated, the required sample data volume is greatly reduced, and the model training efficiency is improved; on the other hand, the region where the hand is located in the original sample image determined by using the mature human body posture recognition algorithm can be directly used as the marking data of the sample database, so that a large amount of workload of manually selecting the target by frame is saved, and the model training efficiency is improved; in another aspect, the target hand washing behavior detection model trained based on the sample image library does not need to be secondarily adjusted, and the model has high detection accuracy for step classification of hand washing behavior.
Drawings
FIG. 1 is a schematic flow chart of a model training method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a region where a hand is located;
fig. 3 is a schematic structural diagram of a target handwashing behavior detection model according to an embodiment of the present invention;
fig. 4 is a schematic flow chart of a hand washing behavior detection method according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a model training apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a hand washing behavior detection device according to another embodiment of the present invention;
fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Fig. 1 is a schematic flow chart of a model training method according to an embodiment of the present invention. The embodiment is suitable for a scene of training a model capable of identifying hand washing behavior step classification. The embodiment may be performed by a model training device, which may be implemented by software and/or hardware, and the power internet of things terminal modeling device may be integrated in a computer device. As shown in fig. 1, the model training method provided in this embodiment includes the following steps:
step 101: and acquiring coordinates of key points of the human body posture skeleton in each original sample image in the original sample image library through a human body posture identification algorithm.
Specifically, the original sample image library includes a plurality of original sample images. Handwashing activity may be included in the original sample image.
In an implementation manner, the obtaining process of the original sample image library in this embodiment may be: multiple images including hand washing activity may be obtained from the network and used as a library of raw sample images.
In another implementation manner, the acquisition process of the original sample image library in this embodiment may be as shown in steps 1 to 3 below.
Step 1: and acquiring hand washing videos of three angles of each sample collection person.
Wherein, three angles are: the left side, the front and the right side of the sample collection personnel, the height of the camera on the left side of the sample collection personnel is different from the height of the camera on the right side of the sample collection personnel.
In this embodiment, the number of the sample collectors is not limited. Illustratively, the number of sample collection personnel may be 20.
Further, the height of the camera on the front of the sample collector is the same as the height of the hands of the sample collector during hand washing. Optionally, the height of the left camera is higher than the height of the hands of the sample collection person during hand washing, and the height of the right camera is lower than the height of the hands of the sample collection person during hand washing. Or the height of the left camera is lower than that of the hands of the sample collection personnel during hand washing, and the height of the right camera is higher than that of the hands of the sample collection personnel during hand washing.
The hand washing video shot is the video for completing seven standard hand washing actions (also called seven-step washing methods). Among the seven standard hand washing actions are: the first step is as follows: washing palms, wetting hands with running water, smearing liquid soap (or soap), making palms opposite, and kneading fingers together; the second step is that: washing the back finger joints, rubbing the palms against the backs of the hands along the finger joints, and exchanging the hands; the third step: washing the finger slits on the palm sides, wherein the palm centers are opposite, and the two hands are crossed and mutually rubbed along the finger slits; the fourth step: washing the back of the finger, bending each finger joint, putting the back of the finger on the palm of the other hand by half fist making, rotating and kneading, and exchanging the two hands; the fifth step: washing the thumb, holding the thumb of the other hand with one hand, rotating and rubbing the thumb, and exchanging the two hands; and a sixth step: washing the finger tips, bending the joints of the fingers, folding the finger tips on the center of the other hand, rotating and kneading, and exchanging the two hands; the seventh step: wash the wrist and arm, rub the wrist and arm, and exchange the two hands.
Step 2: and for each sample collector, carrying out mirror image processing on the hand washing video on the corresponding left side, and carrying out mirror image processing on the hand washing video on the corresponding right side to obtain the hand washing video after mirror image processing.
And step 3: and converting the hand washing video and the hand washing video subjected to mirror image processing into images to form an original sample image library.
Assuming that in the hand washing videos of the left side, the front side and the right side of the sample collection person collected in step 1, the height of the camera on the left side is higher than the height of the hand of the sample collection person during hand washing, and the height of the camera on the right side is lower than the height of the hand of the sample collection person during hand washing, the three collected angles are defined as: upper left, front and lower right. Then in step 2, after the left hand washing video is mirrored, the upper right hand washing video may be obtained. After the right hand washing video is subjected to mirror image processing, the hand washing video with the lower left angle can be obtained. Thus, after the mirroring process, five-angle hand washing videos can be obtained: upper left, lower left, front, upper right, and lower right.
In step 3, the hand washing video obtained by shooting and the hand washing video after mirror image processing can be converted into images through the existing tools or algorithms. The conversion is implemented, for example, by OpenCV. The converted image is determined as the original sample image in step 101.
In this embodiment, through the process of obtaining the original sample image library in steps 1 to 3, a large number of original sample images can be obtained in a short time, so that high efficiency is achieved, and thus, the efficiency of model training is improved.
Typically, hand washing occurs in the vicinity of a hand washing station or a hands-free bottle, and standard hand washing steps are divided into seven steps, each of which lasts about 30 seconds. The sample collection personnel can carry out actions such as rubbing the hands of the sample collection personnel by placing the hands in front of the chest, and the sex, the height, the age, the pattern and the color of clothes, the action habits and the like of different hand cleaners are different. In this embodiment, in order to reduce the influence of the sex, the height, the age, the pattern color of clothes, the action habit, and the like of the sample acquiring person on the trained model, the hand image may be captured from the original sample image to form a sample image, and then training is performed based on the sample image, so as to improve the detection accuracy and the generalization ability of the trained target hand washing behavior detection model.
In order to capture the hand image from the original sample image, firstly, in step 101, a human body posture recognition algorithm is adopted to obtain coordinates of key points of a human body posture skeleton in each original sample image in an original sample image library. The key points in this embodiment may refer to joints of the human body.
For example, the human body gesture recognition algorithm in the present embodiment may be: openpost algorithm. The OPENPLE algorithm is a human body posture recognition framework which is improved on the basis of a Convolutional posture recognition machine (CPM) and is provided for multi-person posture recognition optimization. The traditional CPM is from top to bottom, namely, approximate areas of each person are detected firstly, and each approximate area carries out key point identification of a single person; the openpost algorithm is "bottom-up", i.e., all the key points in the image are detected first, and then the key points are connected to the respective skeletons of each person in the image according to other information. The traditional top-down thought has the following disadvantages: 1. the multi-person posture recognition is easily influenced by trunk shielding; 2. every person needs to run one-time key point detection, which consumes time; 3. depending on the human detection algorithm, if a certain human body is not detected, the algorithm will not detect the skeleton.
The openpost algorithm is free of the above 3 point problem. The OPENPLE algorithm continues the design of a CPM multilayer structure and the characteristics of a full convolution network, and expands the reception field, so that the network can simultaneously consider information of a plurality of key points and find out mutual contact. Compared with CPM, a Partial Affinity Field (PAF) concept is introduced, so that the network can contain the trunk information in the learning range, the importance degree of the PAF is parallel to the key points of the joints, and two different full convolution networks are respectively used for training the two elements. In conclusion, compared with the CPM, the OPENPLE algorithm has better effect and higher speed on the detection of the multi-person trunk. Therefore, the openpost algorithm adopted in the embodiment can improve the speed of intercepting the hand image, thereby improving the training speed of the model.
Alternatively, the output result of the openpost algorithm may be in JAVA description language Object Notation (JSON) format. The coordinates of the keypoints and the names of the keypoints are identified in the output result.
It is understood that the coordinates of the key points of the human body gesture skeleton in the original sample image may be multiple.
Step 102: and determining the area of the hand in the original sample image according to the coordinates of the key points of the human body posture skeleton in the original sample image.
Specifically, after the coordinates of the key points of the human body gesture skeleton in the original sample image are obtained, the area where the hand is located in the original sample image may be determined based on the coordinates.
Alternatively, step 102 may include the following two steps.
Step 1021: and determining target key points related to the hand range in the coordinates of the key points of the human body posture skeleton in the original image.
Optionally, the key points useful for intercepting the hand range include key points at wrist joints and key points at all finger joints. Thus, in step 1021, the determined target keypoints may comprise: the coordinates of the key points at the wrist joints and the coordinates of the key points at all finger joints.
Step 1022: and determining the area of the hand in the original image according to the target key point.
One possible implementation of step 1022 is: determining a first minimum coordinate value and a first maximum coordinate value in the x-axis direction in the target key point; determining a second minimum coordinate value and a second maximum coordinate value in the y-axis direction in the target key point; forming a first coordinate point by the first minimum coordinate value and the second minimum coordinate value, and forming a second coordinate point by the first maximum coordinate value and the second maximum coordinate value; generating a rectangular frame based on the first coordinate point and the second coordinate point; and determining the area where the rectangular frame is located as the area where the hand is located in the original image.
Fig. 2 is a schematic diagram of a region where a hand is located. The implementation of step 1022 is described in detail below in conjunction with fig. 2. As shown in fig. 2, all the target key points 21 are traversed to find out the first minimum coordinate value and the first maximum coordinate value mapped by the X-axis in the coordinate point set, which are marked as XMINAnd XMAXAnd finding out the second minimum coordinate value and the second maximum coordinate value mapped by the Y axis and recording as YMINAnd YMAX. The first coordinate point 22 and the second coordinate point 23 are formed by combining the X-axis extreme value and the Y-axis extreme value. Wherein the first coordinate point 22 has the coordinate of (X)MIN,YMIN) The second coordinate point 23 has the coordinate of (X)MAX,YMAX). A rectangular frame 24 is generated by the two coordinate points, and the area where the rectangular frame 24 is located is determined as the area where the hand is located in the original image.
The coordinate system in which the coordinates in step 1021 and step 1022 are located is the image coordinate system.
Step 103: and according to the area of the hand in the original sample image, intercepting the hand image from the original sample image to form a sample image.
Specifically, the area where the hand is located in the original sample image is determined, and the area is captured from the original sample image to form a sample image for a subsequent training model.
With continued reference to fig. 2, after the rectangular frame 24 is determined, the original sample image is captured according to the rectangular frame 24 to obtain the two-hand range image containing the same character. In this embodiment, the interception may be performed by an existing image interception algorithm.
Optionally, in order to improve the universality of the trained model, the captured hand image may be subjected to data enhancement processing to form a sample image. Wherein the data enhancement processing comprises: random rotation, random mirror image, random brightness adjustment and the like, and aims to simulate a large number of sample images with different scenes and angles through a small number of video samples.
Step 104: and training the initial hand washing behavior detection model according to a sample image library formed by the sample images and the labeling result of each sample image to obtain a target hand washing behavior detection model.
The target hand washing behavior detection model is used for detecting step classification of hand washing behaviors in the target image.
Specifically, the standard result of the sample image in the present embodiment refers to the correct step classification of hand washing behavior in the sample image.
The target image in this embodiment refers to an image to be detected. The step classification of the hand washing behavior in the target image refers to which step in the seven-step washing technique the hand washing behavior in the target image belongs.
In step 104, the target hand washing behavior detection model is obtained by establishing a loss function and continuously iterating and updating the gradient through a back propagation algorithm, so that the predicted classification result and the actual classification result of the initial hand washing behavior detection model are sufficiently fitted.
The training process of step 104 may be: inputting the sample image into an initial hand washing behavior detection model for training to obtain an output result; determining loss parameters according to the output result and the corresponding labeling result in the sample image; and performing back propagation updating on the initial hand washing behavior detection model according to the loss parameters until the training is finished, and determining the initial hand washing behavior detection model at the end of the training as a target hand washing behavior detection model. The loss parameter in this embodiment may be a numerical value, a vector, or a matrix. More specifically, the loss parameter may be determined from a mean square error function, a relative entropy error function, or a cross entropy error function.
According to the model training method in the embodiment, the hand range of the human body is automatically found out through the human body posture recognition algorithm, and the hand image is used as the sample image for model training, so that the influence of the sex, the height, the age, the clothes and the external scene change of a hand washing user on the scale of the sample image library is eliminated, and the required sample data size is greatly reduced. Meanwhile, the workload of manually selecting hand range images is saved. The trained target hand washing behavior detection model does not need to be subjected to secondary network optimization after the scene is migrated, and is more suitable for being applied to an actual hand washing recognition software system, left and right hand parameters obtained through gesture recognition can effectively distinguish the difference of left and right hand positions in the same hand washing step, and the positions of the two hands can be detected according to the overall skeleton gesture when the two hands are overlapped in the picture, so that the detection accuracy is improved.
Fig. 3 is a schematic structural diagram of a target hand washing behavior detection model according to an embodiment of the present invention. As shown in fig. 3, the target hand washing behavior detection model trained in this embodiment includes three layers of convolutional neural networks connected in sequence. Wherein, every layer of convolution neural network includes that the connection is in proper order: convolutional layers, activation function layers, and pooling layers.
Because the images of the two hands have smaller sizes and do not contain redundant noise information of the background, the three-layer convolutional neural network with shallower depth is adopted. Through testing, other depth recognition networks with higher complexity and deeper are not as effective as the three-layer convolutional neural network in processing hand range images.
The embodiment provides a model training method, which comprises the following steps: the method comprises the steps of obtaining hand washing videos of three angles of each shot sample collection person, wherein the three angles are as follows: the left side, the front side and the right side of the sample collector, wherein the height of the camera on the left side of the sample collector is different from the height of the camera on the right side of the sample collector; for each sample collector, carrying out mirror image processing on the hand washing video on the corresponding left side, and carrying out mirror image processing on the hand washing video on the corresponding right side to obtain the hand washing video after mirror image processing; converting the hand washing video and the hand washing video subjected to mirror image processing into images to form an original sample image library; obtaining the coordinates of key points of a human body posture skeleton in each original sample image in an original sample image library through a human body posture identification algorithm; determining the area of the hand in the original sample image according to the coordinates of the key points of the human body gesture skeleton in the original sample image; according to the area of the hand in the original sample image, intercepting the hand image from the original sample image to form a sample image; and training the initial hand washing behavior detection model according to a sample image library formed by the sample images and the labeling result of each sample image to obtain a target hand washing behavior detection model, wherein the target hand washing behavior detection model is used for step classification of hand washing behavior detection in the target image. In the model training method, on one hand, hand washing videos at multiple angles can be acquired in a mirror image processing mode, a large number of original sample images can be acquired in a short time, high efficiency is achieved, and the model training efficiency is improved; on the other hand, the hand image is intercepted from the original sample image through a human body posture recognition algorithm to form a sample image, so that the influence on the scale of the sample image library caused by the sex, the height, the age, the clothes and the external scene change of a hand washing user is eliminated, the required sample data size is greatly reduced, and the model training efficiency is improved; on the other hand, the region where the hand is located in the original sample image determined by using the mature human body posture recognition algorithm can be directly used as the marking data of the sample database, so that a large amount of workload of manually selecting the target by frame is saved, and the model training efficiency is improved; in another aspect, the target hand washing behavior detection model trained based on the sample image library does not need to be secondarily adjusted, and the model has high detection accuracy for step classification of hand washing behavior.
Fig. 4 is a schematic flow chart of a hand washing behavior detection method according to an embodiment of the present invention. The embodiment is suitable for scenes for identifying the hand washing behavior step classification. The embodiment may be implemented by a hand washing behavior detection apparatus, which may be implemented by software and/or hardware, and may be integrated in a computer device. As shown in fig. 4, the hand washing behavior detection method provided by this embodiment includes the following steps:
step 401: and acquiring coordinates of key points of the human body posture skeleton in the target image through a human body posture identification algorithm.
The target image in the present embodiment may be an image cut out from a captured video. The video can be formed by shooting the hand washing process of the detection personnel beside a hand washing table in a hospital and related places. The hand washing behavior detection method provided by this embodiment may detect images in the video to detect step classifications of hand washing behavior.
Step 402: and determining the region of the hand in the target image according to the coordinates of the key points of the human body posture skeleton in the target image.
Step 403: and intercepting the hand image from the target image according to the area of the hand in the target image to form an image to be identified.
The implementation processes and technical principles in step 401 and step 101, step 402 and step 102, and step 403 and step 103 are similar, and are not described herein again.
Step 404: and inputting the image to be recognized into the target hand washing behavior detection model to obtain the step classification of the hand washing behavior in the target image.
The target hand washing behavior detection model is obtained by adopting a model training method in the embodiment and various optional implementation manners shown in fig. 1.
Optionally, within the target hand washing behavior detection model, the confidence levels of seven hand washing steps corresponding to the image to be recognized may be obtained, and the step with the highest confidence level is taken as the actual output result.
It should be noted that if the sum of the obtained confidences of the seven hand washing steps is smaller than a preset threshold, or the confidences of the seven hand washing steps are all smaller than the preset threshold, it is indicated that the hand washing behavior in the target image is not a standard hand washing step, and a prompt message may be output to remind the detection person, that is, the hand washing person to perform the hand washing regulation.
Optionally, after step 404, the following steps are further included: and determining whether the hand washing process of the detection personnel meets the preset hand washing standard or not according to the step classification of the hand washing behaviors in the plurality of target images corresponding to the same detection personnel. Wherein the plurality of target images are images arranged in time sequence.
Illustratively, the pre-set hand washing protocol may be a seven-step washing technique. Based on the mode, whether the specification of the seven-step washing method is finished in the hand washing process of the detector can be detected. Namely, on the basis of realizing the step classification of detecting the hand washing action, the hand washing process is supervised, and the management of the hand hygiene of medical staff is further enhanced.
The embodiment of the invention provides a hand washing behavior detection method, which comprises the following steps: acquiring coordinates of key points of a human body posture skeleton in a target image through a human body posture identification algorithm; determining the region where the hand is located in the target image according to the coordinates of the key points of the human body posture skeleton in the target image; according to the area of the hand in the target image, intercepting the hand image from the target image to form an image to be identified; and inputting the image to be recognized into the target hand washing behavior detection model to obtain the step classification of the hand washing behavior in the target image. According to the hand washing behavior detection method, the hand image is firstly captured from the target image to form the image to be recognized, the image to be recognized is detected based on the target hand washing behavior detection model at the pre-trained position, and the detection efficiency and accuracy are high, so that the hand hygiene of medical staff can be effectively managed based on the method.
Fig. 5 is a schematic structural diagram of a model training apparatus according to an embodiment of the present invention. As shown in fig. 5, the model training apparatus provided in this embodiment includes the following modules: a first obtaining module 51, a first determining module 52, a second determining module 53 and a third determining module 54.
The first obtaining module 51 is configured to obtain, through a human body posture recognition algorithm, coordinates of key points of a human body posture skeleton in each original sample image in the original sample image library.
Optionally, the human body posture recognition algorithm is as follows: openpost algorithm.
Optionally, the method further comprises: the device comprises a second acquisition module, a mirror image processing module and a fourth determination module.
And the second acquisition module is used for acquiring hand washing videos of three angles of each photographed sample acquisition person.
Wherein, three angles are: the left side, the front and the right side of the sample collection personnel, the height of the camera on the left side of the sample collection personnel is different from the height of the camera on the right side of the sample collection personnel.
And the mirror image processing module is used for carrying out mirror image processing on the hand washing video on the corresponding left side and carrying out mirror image processing on the hand washing video on the corresponding right side to obtain the hand washing video after mirror image processing for each sample collector.
And the fourth determination module is used for converting the hand washing video and the hand washing video subjected to mirror image processing into images to form an original sample image library.
The first determining module 52 is configured to determine, according to the coordinates of the key points of the human body gesture skeleton in the original sample image, an area where the hand is located in the original sample image.
Optionally, the first determining module 52 is specifically configured to: determining target key points related to a hand range in the coordinates of the key points of the human body posture skeleton in the original image; and determining the area of the hand in the original image according to the target key point.
In terms of determining the region where the hand is located in the original image according to the target key point, the first determining module 52 is specifically configured to: determining a first minimum coordinate value and a first maximum coordinate value in the x-axis direction in the target key point; determining a second minimum coordinate value and a second maximum coordinate value in the y-axis direction in the target key point; forming a first coordinate point by the first minimum coordinate value and the second minimum coordinate value, and forming a second coordinate point by the first maximum coordinate value and the second maximum coordinate value; generating a rectangular frame based on the first coordinate point and the second coordinate point; and determining the area where the rectangular frame is located as the area where the hand is located in the original image.
And a second determining module 53, configured to intercept the hand image from the original sample image according to the area where the hand is located in the original sample image, so as to form a sample image.
And a third determining module 54, configured to train the initial hand washing behavior detection model according to a sample image library formed by the sample images and an annotation result of each sample image, so as to obtain a target hand washing behavior detection model.
The target hand washing behavior detection model is used for detecting step classification of hand washing behaviors in the target image.
Optionally, the target handwashing behavior detection model includes three layers of convolutional neural networks connected in series. Wherein, every layer of convolution neural network includes that the connection is in proper order: convolutional layers, activation function layers, and pooling layers.
The model training device provided by the embodiment of the invention can execute the model training method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 6 is a schematic structural diagram of a hand washing behavior detection device according to another embodiment of the present invention. As shown in fig. 6, the hand washing behavior detection apparatus provided in this embodiment includes the following modules: a third obtaining module 61, a fifth determining module 62, a sixth determining module 63 and a seventh determining module 64.
And the third obtaining module 61 is configured to obtain coordinates of key points of the human body posture skeleton in the target image through a human body posture recognition algorithm.
And a fifth determining module 62, configured to determine, according to the coordinates of the key points of the human body posture skeleton in the target image, an area where the hand is located in the target image.
And a sixth determining module 63, configured to intercept the hand image from the target image according to the area where the hand is located in the target image, and form an image to be identified.
And a seventh determining module 64, configured to input the image to be recognized into the target hand washing behavior detection model, so as to obtain step classification of hand washing behaviors in the target image.
The target hand washing behavior detection model is obtained by adopting a model training method in the embodiment and various optional modes as shown in fig. 1.
Optionally, the apparatus further comprises: and the eighth determining module is used for determining whether the hand washing process of the detection personnel meets the preset hand washing standard or not according to the step classification of the hand washing behaviors in the plurality of target images corresponding to the same detection personnel. Wherein the plurality of target images are images arranged in time sequence.
The hand washing behavior detection device provided by the embodiment of the invention can execute the hand washing behavior detection method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present invention. As shown in fig. 7, the computer device includes a processor 70 and a memory 71. The number of the processors 70 in the computer device may be one or more, and one processor 70 is taken as an example in fig. 7; the processor 70 and the memory 71 of the computer device may be connected by a bus or other means, as exemplified by the bus connection in fig. 7.
The memory 71 is used as a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as program instructions and modules corresponding to the model training method in the embodiment of the present invention (for example, the first obtaining module 51, the first determining module 52, the second determining module 53, and the third determining module 54 in the model training apparatus, or the third obtaining module 61, the fifth determining module 62, the sixth determining module 63, and the seventh determining module 64 in the hand washing behavior detecting apparatus). The processor 70 executes various functional applications of the computer device and the model training method, i.e., implements the above-described model training method or the hand washing behavior detection method, by executing software programs, instructions, and modules stored in the memory 71.
The memory 71 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the computer device, and the like. Further, the memory 71 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the memory 71 may further include memory located remotely from the processor 70, which may be connected to a computer device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The present invention also provides a storage medium containing computer-executable instructions which, when executed by a computer processor, perform a method of model training, the method comprising:
acquiring hand washing videos of three angles of each sample acquisition person; wherein the three angles are: the system comprises a left side, a front side and a right side of a sample collector, wherein the height of a camera on the left side of the sample collector is different from that of a camera on the right side of the sample collector;
for each sample collector, carrying out mirror image processing on the hand washing video on the corresponding left side, and carrying out mirror image processing on the hand washing video on the corresponding right side to obtain the hand washing video after mirror image processing;
converting the hand washing video and the hand washing video subjected to mirror image processing into images to form an original sample image library;
obtaining the coordinates of key points of a human body posture skeleton in each original sample image in an original sample image library through a human body posture identification algorithm;
determining the area of the hand in the original sample image according to the coordinates of the key points of the human body posture skeleton in the original sample image;
according to the area where the hand is located in the original sample image, intercepting a hand image from the original sample image to form a sample image;
training an initial hand washing behavior detection model according to a sample image library formed by the sample images and the labeling result of each sample image to obtain a target hand washing behavior detection model; the target hand washing behavior detection model is used for detecting step classification of hand washing behaviors in the target image.
Of course, the storage medium provided by the embodiments of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also perform related operations in the model training method provided by any embodiments of the present invention.
The present invention also provides a storage medium containing computer-executable instructions for performing a hand washing performance detection method when executed by a computer processor, the method comprising:
acquiring coordinates of key points of a human body posture skeleton in a target image through a human body posture identification algorithm;
determining the region where the hand is located in the target image according to the coordinates of the key points of the human body posture skeleton in the target image;
according to the area of the hand in the target image, intercepting the hand image from the target image to form an image to be identified;
inputting the image to be recognized into a target hand washing behavior detection model to obtain step classification of hand washing behaviors in the target image; the target hand washing behavior detection model is obtained by the model training method provided by the embodiment and optional modes shown in fig. 1.
Of course, the storage medium provided by the embodiments of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also execute the related operations in the hand washing behavior detection method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a computer device, or a network device) to execute the model training method according to the embodiments of the present invention.
It should be noted that, in the embodiment of the model training device or the hand washing behavior detection device, the included units and modules are only divided according to the functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (9)
1. A method of model training, comprising:
acquiring hand washing videos of three angles of each sample acquisition person; wherein the three angles are: the system comprises a left side, a front side and a right side of a sample collector, wherein the height of a camera on the left side of the sample collector is different from that of a camera on the right side of the sample collector;
for each sample collector, carrying out mirror image processing on the hand washing video on the corresponding left side, and carrying out mirror image processing on the hand washing video on the corresponding right side to obtain the hand washing video after mirror image processing;
converting the hand washing video and the hand washing video subjected to mirror image processing into images to form an original sample image library;
obtaining the coordinates of key points of a human body posture skeleton in each original sample image in an original sample image library through a human body posture identification algorithm;
determining the area of the hand in the original sample image according to the coordinates of the key points of the human body posture skeleton in the original sample image;
according to the area where the hand is located in the original sample image, intercepting a hand image from the original sample image to form a sample image;
training an initial hand washing behavior detection model according to a sample image library formed by the sample images and the labeling result of each sample image to obtain a target hand washing behavior detection model; the target hand washing behavior detection model is used for detecting step classification of hand washing behaviors in the target image.
2. The method of claim 1, wherein the human gesture recognition algorithm is: openpost algorithm.
3. The method of claim 1, wherein the determining the area of the original sample image where the hand is located according to the coordinates of the key points of the human body posture skeleton comprises:
determining target key points related to a hand range in the coordinates of the key points of the human body posture skeleton in the original image;
and determining the area of the hand in the original image according to the target key point.
4. The method according to claim 3, wherein the determining the region of the original image where the hand is located according to the target key point comprises:
determining a first minimum coordinate value and a first maximum coordinate value in the x-axis direction in the target key point;
determining a second minimum coordinate value and a second maximum coordinate value in the y-axis direction in the target key point;
combining the first minimum coordinate value and the second minimum coordinate value into a first coordinate point, and combining the first maximum coordinate value and the second maximum coordinate value into a second coordinate point;
generating a rectangular frame based on the first coordinate point and the second coordinate point;
and determining the area where the rectangular frame is located as the area where the hand is located in the original image.
5. The method of any of claims 1 to 4, wherein the target handwashing behavior detection model comprises three layers of sequentially connected convolutional neural networks; wherein, every layer of convolution neural network includes that the connection is in proper order: convolutional layers, activation function layers, and pooling layers.
6. A hand washing behavior detection method, comprising:
acquiring coordinates of key points of a human body posture skeleton in a target image through a human body posture identification algorithm;
determining the region where the hand is located in the target image according to the coordinates of the key points of the human body posture skeleton in the target image;
according to the area of the hand in the target image, intercepting the hand image from the target image to form an image to be identified;
inputting the image to be recognized into a target hand washing behavior detection model to obtain step classification of hand washing behaviors in the target image; wherein the target hand washing behavior detection model is obtained by the model training method according to any one of claims 1 to 6.
7. The method of claim 6, wherein after the step of obtaining hand washing behavior in the target image is categorized, the method further comprises:
determining whether the hand washing process of the detection personnel meets a preset hand washing standard or not according to the step classification of the hand washing behaviors in a plurality of target images corresponding to the same detection personnel; wherein the plurality of target images are images arranged in time sequence.
8. A computer device, characterized in that the computer device comprises:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the model training method of any one of claims 1-7.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the model training method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011448943.2A CN112507880A (en) | 2020-12-09 | 2020-12-09 | Model training method, hand washing behavior detection method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011448943.2A CN112507880A (en) | 2020-12-09 | 2020-12-09 | Model training method, hand washing behavior detection method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112507880A true CN112507880A (en) | 2021-03-16 |
Family
ID=74971364
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011448943.2A Pending CN112507880A (en) | 2020-12-09 | 2020-12-09 | Model training method, hand washing behavior detection method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112507880A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113569671A (en) * | 2021-07-13 | 2021-10-29 | 北京大数医达科技有限公司 | Abnormal behavior alarm method and device |
| CN114120411A (en) * | 2021-11-29 | 2022-03-01 | 杭州百世伽信息科技有限公司 | Automatic hand hygiene management method |
| CN114694248A (en) * | 2022-03-10 | 2022-07-01 | 苏州爱可尔智能科技有限公司 | Hand hygiene monitoring method, system, equipment and medium based on graph neural network |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110705520A (en) * | 2019-10-22 | 2020-01-17 | 上海眼控科技股份有限公司 | Object detection method, device, computer equipment and computer readable storage medium |
| CN111046744A (en) * | 2019-11-21 | 2020-04-21 | 深圳云天励飞技术有限公司 | Method and device for detecting attention area, readable storage medium and terminal equipment |
| CN111598081A (en) * | 2020-04-09 | 2020-08-28 | 浙江工业大学 | Automatic seven-step hand washing method operation normative detection method |
| CN111709461A (en) * | 2020-05-27 | 2020-09-25 | 北京百度网讯科技有限公司 | Training method, device and equipment of animal recognition model and storage medium |
| CN111783662A (en) * | 2020-06-30 | 2020-10-16 | 北京字节跳动网络技术有限公司 | Attitude estimation method, estimation model training method, device, medium and equipment |
-
2020
- 2020-12-09 CN CN202011448943.2A patent/CN112507880A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110705520A (en) * | 2019-10-22 | 2020-01-17 | 上海眼控科技股份有限公司 | Object detection method, device, computer equipment and computer readable storage medium |
| CN111046744A (en) * | 2019-11-21 | 2020-04-21 | 深圳云天励飞技术有限公司 | Method and device for detecting attention area, readable storage medium and terminal equipment |
| CN111598081A (en) * | 2020-04-09 | 2020-08-28 | 浙江工业大学 | Automatic seven-step hand washing method operation normative detection method |
| CN111709461A (en) * | 2020-05-27 | 2020-09-25 | 北京百度网讯科技有限公司 | Training method, device and equipment of animal recognition model and storage medium |
| CN111783662A (en) * | 2020-06-30 | 2020-10-16 | 北京字节跳动网络技术有限公司 | Attitude estimation method, estimation model training method, device, medium and equipment |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113569671A (en) * | 2021-07-13 | 2021-10-29 | 北京大数医达科技有限公司 | Abnormal behavior alarm method and device |
| CN114120411A (en) * | 2021-11-29 | 2022-03-01 | 杭州百世伽信息科技有限公司 | Automatic hand hygiene management method |
| CN114120411B (en) * | 2021-11-29 | 2022-05-31 | 杭州百世伽信息科技有限公司 | Automatic hand hygiene management method |
| CN114694248A (en) * | 2022-03-10 | 2022-07-01 | 苏州爱可尔智能科技有限公司 | Hand hygiene monitoring method, system, equipment and medium based on graph neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112507880A (en) | Model training method, hand washing behavior detection method, device, equipment and medium | |
| CN108595008B (en) | Human-computer interaction method based on eye movement control | |
| CN104463152B (en) | A kind of gesture identification method, system, terminal device and Wearable | |
| CN108509026B (en) | Remote maintenance support system and method based on enhanced interaction mode | |
| CN106569613A (en) | Multi-modal man-machine interaction system and control method thereof | |
| CN109359514B (en) | DeskVR-oriented gesture tracking and recognition combined strategy method | |
| CN108960192A (en) | Action identification method and its neural network generation method, device and electronic equipment | |
| CN105068646B (en) | The control method and system of terminal | |
| CN110837792A (en) | Three-dimensional gesture recognition method and device | |
| JP2016014954A (en) | Method for detecting finger shape, program thereof, storage medium of program thereof, and system for detecting finger shape | |
| CN116766213B (en) | Bionic hand control method, system and equipment based on image processing | |
| CN112199994A (en) | Method and device for detecting interaction between 3D hand and unknown object in RGB video in real time | |
| KR20230080938A (en) | Method and apparatus of gesture recognition and classification using convolutional block attention module | |
| CN114049683A (en) | Post-healing rehabilitation auxiliary detection system, method and medium based on three-dimensional human skeleton model | |
| CN114332927A (en) | Classroom hand-raising behavior detection method, system, computer equipment and storage medium | |
| Qin et al. | Real-time markerless hand gesture recognition with depth camera | |
| Ueng et al. | Vision based multi-user human computer interaction | |
| CN109377518A (en) | Target tracking method, device, target tracking equipment and storage medium | |
| CN113282164A (en) | Processing method and device | |
| CN108108648A (en) | A kind of new gesture recognition system device and method | |
| Dutta et al. | A hand gesture-operated system for rehabilitation using an end-to-end detection framework | |
| Soroni et al. | Hand Gesture Based Virtual Blackboard Using Webcam | |
| CN110175629B (en) | Human body action similarity calculation method and device | |
| CN109814712B (en) | Gesture recognition system based on virtual interaction device | |
| CN114495173A (en) | Posture recognition method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |