CN117953591B - Intelligent limb rehabilitation assisting method and device - Google Patents

Abstract

The present invention relates to the field of computer vision technology, and in particular to an intelligent limb rehabilitation auxiliary method and device. The method comprises the following steps: collecting limb motion data through an image sensor to obtain limb motion image data; performing posture estimation on the limb motion image data to obtain posture estimation data, and performing key point detection on the posture estimation data to obtain limb motion key point detection data; performing action recognition on the limb motion key point detection data to obtain limb motion action recognition data; generating action parameters for the limb motion key point detection data according to the limb motion action recognition data to obtain limb action parameter data, so as to perform intelligent limb rehabilitation auxiliary operations. The present invention can provide accurate and reliable intelligent limb rehabilitation auxiliary parameters to reduce dependence on manpower, reduce manpower costs and time costs.

Description

Intelligent limb rehabilitation assisting method and device

Technical Field

The present invention relates to the field of computer vision technology, and in particular to an intelligent limb rehabilitation auxiliary method and device.

Background technique

Conventional limb rehabilitation assistance methods often use rehabilitation technology or assistive equipment to perform limb rehabilitation assistance operations. In this process, they often rely on manual assistance or simple equipment assistance, and often rely on the experience and skills of rehabilitation therapists. However, human resources are limited and cannot meet all needs. Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain "information" from images or multidimensional data. The information here refers to information defined by Shannon that can be used to help make a "decision". Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. How to combine limb rehabilitation assistance methods with computer vision technology has become a problem.

Summary of the invention

In order to solve the above technical problems, the present invention proposes an intelligent limb rehabilitation auxiliary method and device to solve at least one of the above technical problems.

The present application provides an intelligent limb rehabilitation assisting method, comprising the following steps:

Step S1: collecting limb movement data through an image sensor to obtain limb movement image data;

Step S2 comprises:

Step S21: performing posture estimation on the limb motion image data to obtain posture estimation data;

Step S22: performing local key point detection on the posture estimation data to obtain first limb movement key point detection data;

Step S23: performing global key point detection on the posture estimation data to obtain second limb movement key point detection data;

Step S3: performing action recognition on the limb movement key point detection data to obtain limb movement action recognition data, wherein the limb movement key point detection data includes first limb movement key point detection data and second limb movement key point detection data;

Step S4 comprises:

Perform action classification mapping according to the limb movement action recognition data to obtain action classification mapping data;

Perform action timing division according to the limb movement key point detection data and the action classification mapping data to obtain action timing division data;

Extract motion features according to the motion time sequence division data to obtain motion feature data, wherein the motion feature data includes motion duration data, motion speed data, motion acceleration data and motion angle change data;

Motion parameters are generated according to the motion classification mapping data, motion timing division data and motion feature data to obtain limb motion parameter data for performing intelligent limb rehabilitation auxiliary operations.

The present invention uses image sensors to collect limb movement data, avoids invasive detection or operation of users, and improves user comfort and acceptance. The data collected by the image sensor can be processed and analyzed in real time, so that rehabilitation assistance operations can be adjusted and fed back in time according to the actual situation of the user. By performing posture estimation and key point detection on limb movement image data, the user's movement state and key point information can be accurately obtained, thereby realizing parameter formulation, generation and monitoring that are more in line with the actual situation of the user. The method uses motion recognition technology to analyze the limb movement key point detection data, and can automatically identify the user's movement without manual intervention. And the corresponding motion parameters are automatically generated according to the recognition results, further simplifying the operation process of rehabilitation assistance operations. Compared with traditional rehabilitation assistance methods, this method does not require the use of expensive equipment or complex operations, has a lower cost, and is easy to implement and promote, which helps to improve the coverage and popularity of rehabilitation services and reduce dependence on human operations.

In the present invention, through the posture estimation of step S21, the overall posture information of the limb can be obtained, which helps to better understand the overall motion state of the limb and provide an important reference for subsequent key point detection. Step S22 performs local key point detection on the posture estimation data, which can finely identify the local key points of the limb. These local key points can represent the important parts and motion characteristics of the limb, which helps to more accurately analyze and understand the details of the limb movement. Step S23 performs global key point detection on the posture estimation data, which can capture the key point information of the limb as a whole. The global key points can reflect the overall structure and motion state of the limb, which helps to more comprehensively analyze and understand the overall motion characteristics of the limb. The present invention can improve the accuracy and stability of key point detection, help to more accurately identify and locate the key points of the limb, and provide reliable data support for subsequent action recognition and parameter generation.

In the present invention, by performing action classification mapping according to the limb movement action recognition data, the motion data can be mapped to a specific action category, thereby achieving classification and recognition of different actions, which helps the system to better understand the user's movement behavior and provide effective data support for subsequent rehabilitation assistance. According to the limb movement key point detection data and the action classification mapping data, the action timing division can be performed, and the entire movement process can be decomposed into different time periods or stages, which helps to analyze the movement process more carefully and accurately, and provides a more reliable basis for subsequent feature extraction and parameter generation. According to the action timing division data, action feature extraction can be performed, and information about the movement can be obtained from multiple aspects, such as duration, speed, acceleration, and angle change, etc. These feature data can reflect various aspects of the movement, and provide more reference basis for subsequent parameter generation and rehabilitation assistance. According to the action classification mapping data, the action timing division data, and the action feature data, the action parameter generation can be performed, and the type, timing, and feature information of the action can be comprehensively considered, so as to obtain more accurate limb movement parameter data.

Preferably, step S1 specifically includes:

Step S11: collecting first limb movement data at a preset first threshold angle data through an image sensor to obtain first limb movement image data;

Step S12: collecting second limb movement data at a preset second threshold angle data through an image sensor to obtain second limb movement image data;

Step S13: performing image stitching on the first limb motion image data and the second limb motion image data to obtain limb motion image data.

In the present invention, by using an image sensor, the user's limb movement data can be collected in real time and accurately. The first and second threshold angle data can be adjusted according to the specific situation of the user to adapt to the athletic ability of different users, so that the collected data is more operational and accurate. By setting different threshold angles, limb movement data can be collected from different angles, so that the user's movement posture can be observed and analyzed from multiple perspectives, and the state and characteristics of limb movement can be more accurately evaluated. The first and second limb movement image data are spliced to increase the information richness of the image, and the spliced image can present a more complete and comprehensive limb movement state, which helps to improve the accuracy and stability of subsequent posture estimation and key point detection.

Preferably, step S13 is specifically:

Step S131: performing first limb angle detection on the first limb motion image data according to preset first threshold angle data to obtain first limb angle detection data;

Step S132: performing second limb angle detection on the second limb motion image data according to preset second threshold angle data to obtain second limb angle detection data;

Step S133: performing image correction on the first limb motion image data according to the first limb angle detection data to obtain first limb motion image correction data, and performing image correction on the second limb motion image data according to the second limb angle detection data to obtain second limb motion image correction data;

Step S134: extracting feature points from the first limb motion image correction data and the second limb motion image correction data to obtain first limb motion image feature point data and second limb motion image feature point data;

Step S135: performing feature matching on the first limb motion image feature point data and the second limb motion image feature point data to obtain limb motion image feature matching data;

Step S136: performing perspective transformation on the first limb movement image data and the second limb movement image data according to the limb movement image feature matching data to obtain limb movement image perspective transformation data;

Step S137: performing image fusion on the first limb motion image perspective transformation data and the second limb motion image perspective transformation data to obtain limb motion image fusion data;

Step S138: Perform edge repair on the limb motion image fusion data to obtain limb motion image data.

In the present invention, steps S131 and S132 use angle detection technology to accurately detect the angle information of the first and second limbs. Through image correction (step S133), the collected image data can be corrected to reduce the error introduced by angle deviation and improve the accuracy of subsequent processing. Steps S134 and S135 use feature point extraction and matching technology to extract key feature points from the corrected image and associate them through feature matching, which helps to accurately capture the key information of limb movement and maintain consistency. Steps S136 and S137 use perspective transformation and image fusion technology to synthesize the two corrected images to obtain more comprehensive and complete limb movement image data, improve the information richness and quality of the image, and thus more accurately analyze limb movement. Step S138 performs edge repair on the image, which can repair the edge problems introduced in the image due to correction, transformation and other processes, making the image clearer and more accurate, and improving the effect of subsequent processing.

Preferably, step S134 is specifically as follows:

Step S101: performing cluster calculation on the first limb motion image correction data and the second limb motion image correction data to obtain first limb motion image cluster data and second limb motion image cluster data;

Step S102: extracting cluster centers of the first limb movement image cluster data and the second limb movement image cluster data to obtain first image cluster center data and second image cluster center data;

Step S103: extracting cluster radius from the first limb movement image cluster data and the second limb movement image cluster data to obtain first cluster radius data and second cluster radius data;

Step S104: performing weighted calculation on the first cluster radius data using the first limb motion image cluster data to obtain first cluster radius weighted data, and performing weighted calculation on the second cluster radius data using the second limb motion image cluster data to obtain second cluster radius weighted data;

Step S105: performing domain pixel selection on the first image cluster center data using the first cluster radius weighted data to obtain first limb motion image domain pixel data, and performing domain pixel selection on the second image cluster center data using the second cluster radius weighted data to obtain second limb motion image domain pixel data;

Step S106: Perform dynamic pixel grayscale judgment on the pixel data in the first limb motion image area and the pixel data in the second limb motion image area to obtain the first limb motion image feature point data and the second limb motion image feature point data.

In step S134 of the present invention, the image data can be processed and analyzed in a refined manner, which helps to more accurately capture the feature information in the image and improve the accuracy and reliability of subsequent processing. In steps S104 and S105, weighted calculation and field pixel selection technology are used to perform weighted processing and pixel selection on the image data according to clustering data and cluster radius data, highlight the key information in the image, and improve the accuracy and stability of feature point extraction. In step S106, dynamic pixel grayscale judgment is performed, and feature points can be screened and judged according to the grayscale information of the image, reducing the sensitivity to noise and interference, and improving the recognition and extraction effect of feature points. The method analyzes image data at multiple levels and angles to improve the accuracy and stability of feature point extraction of limb movement images. Traditional feature point extraction often uses feature point acquisition based on a fixed threshold. The present invention performs adaptive adjustment through the image's own characteristics, thereby improving the extraction effect of feature points and the quality of image data, and overcoming the problems of insufficient accuracy and unstable extraction effect in the prior art.

Preferably, step S106 is specifically:

Obtain light angle data and light intensity data through light sensors;

Performing angle calculation on preset first threshold angle data and preset second threshold angle data according to the illumination angle data to obtain first shooting illumination angle data and second shooting illumination angle data;

Generating a grayscale threshold according to the first shooting illumination angle data and the illumination intensity data to obtain first grayscale threshold data, and generating a grayscale threshold according to the second shooting illumination angle data and the illumination intensity data to obtain second grayscale threshold data;

The pixel data in the first limb motion image field is subjected to pixel grayscale judgment using the first grayscale threshold data to obtain the first limb motion image feature point data, and the pixel data in the second limb motion image field is subjected to pixel grayscale judgment using the second grayscale threshold data to obtain the second limb motion image feature point data.

In the present invention, by using the illumination angle data obtained by the illumination sensor, the influence of the illumination direction on the image can be considered. Calculating the grayscale threshold according to the illumination angle data helps to adaptively adjust the image according to different illumination conditions, thereby improving the stability and robustness of image processing. Step S106 dynamically generates the grayscale threshold according to the illumination angle data and the illumination intensity data, and can adjust the grayscale threshold of the image according to the real-time illumination conditions, thereby avoiding the problem of unstable or misjudgment of image processing caused by different illumination conditions, and improving the accuracy of extracting image feature points. By performing grayscale judgment on image pixels according to the grayscale threshold generated by the illumination angle data, adaptive pixel grayscale judgment can be achieved, which helps to accurately extract the feature points of the image under different illumination conditions, and overcomes the problem of unstable feature extraction effect of the prior art when the illumination changes greatly. Step S106 calculates the grayscale threshold and performs pixel grayscale judgment according to the illumination angle data, thereby improving the robustness of image processing. For images under different illumination conditions, feature points can be extracted more accurately, thereby improving the effect and reliability of subsequent processing.

Preferably, step S22 is specifically:

Perform branch network detection selection according to the posture estimation data to obtain branch network detection data;

Perform key point detection on specific parts of the posture estimation data according to the branch network detection data to obtain key point detection data of specific parts;

The key point detection data of specific parts corresponding to the detection data of different branch networks are feature fused to obtain the first limb movement key point detection data.

In the present invention, by introducing multiple branch networks, each branch network is responsible for detecting the key points of a specific part, which can more accurately capture the motion information of different parts in limb movement, thereby improving the accuracy and comprehensiveness of detection. On the basis of the branch network detection selection, key point detection is performed for each part separately, so that the detection results are more detailed and specific, which helps to more accurately analyze and evaluate the user's motion state in the process. Feature fusion of the key point data of specific parts detected by different branch networks can take into account the motion information of different parts, thereby obtaining more accurate limb movement key point detection data, which helps to improve the rehabilitation assistance system's understanding and analysis capabilities of the user's motion state.

Preferably, step S3 specifically includes:

Extracting features from the limb movement key point detection data to obtain limb movement key point feature data;

Performing feature selection on the key point feature data of limb movement to obtain key point feature selection data;

The key point feature selection data is used for action recognition to obtain limb movement action recognition data.

In the present invention, through the feature extraction in step S3, key point feature data representing the characteristics of limb movement can be extracted from the key point detection data of limb movement, which is helpful to convert complex limb movement into more specific and easier to analyze feature data. The feature selection process in step S3 can select the most representative and discriminative features from the extracted key point feature data to reduce the feature dimension and improve the accuracy of recognition, which helps to reduce the complexity of data processing, while retaining important information about limb movement and improving the effect of action recognition. The data extracted and selected by the features can better reflect the characteristics of limb movement, so that the action recognition is more accurate. By selecting representative features and combining appropriate recognition algorithms, different limb movement actions can be more reliably identified, providing more accurate data support for intelligent limb rehabilitation assistance. Through feature selection, the calculation amount of the recognition algorithm can be reduced and the calculation efficiency can be improved. The carefully selected features can better reflect the essential characteristics of limb movement, thereby reducing unnecessary calculation overhead and speeding up the speed of action recognition. Due to the reduction of feature dimensions and calculation amount, the system can process limb movement data more quickly and improve real-time performance.

Preferably, the present application also provides an intelligent limb rehabilitation assisting device for executing the intelligent limb rehabilitation assisting method as described above, the intelligent limb rehabilitation assisting device comprising:

A limb movement data acquisition module is used to acquire limb movement data through an image sensor to obtain limb movement image data;

A limb movement key point detection module is used to perform posture estimation on limb movement image data to obtain posture estimation data, and perform key point detection on the posture estimation data to obtain limb movement key point detection data;

The action recognition module is used to perform action recognition on the limb movement key point detection data to obtain limb movement action recognition data;

The intelligent limb rehabilitation auxiliary operation module is used to generate action parameters for limb movement key point detection data according to limb movement action recognition data, and obtain limb movement parameter data for intelligent limb rehabilitation auxiliary operations.

The beneficial effects of the present invention are: using image sensors to collect limb movement data avoids invasive detection or operation of users, and improves user comfort and acceptance. The data collected by the image sensor can be processed and analyzed in real time, so that rehabilitation assistance operations can be adjusted and fed back in a timely manner according to the actual situation of the user. By performing posture estimation and key point detection on limb movement image data, the user's movement state and key point information can be accurately obtained, thereby achieving parameter formulation, generation and monitoring that is more in line with the user's actual situation. The method uses motion recognition technology to analyze limb movement key point detection data, and can automatically identify the user's movement without manual intervention. And the corresponding motion parameters are automatically generated according to the recognition results, further simplifying the operation process of rehabilitation assistance operations. Compared with traditional rehabilitation assistance methods, this method does not require the use of expensive equipment or complex operations, has a lower cost, and is easy to implement and promote, which helps to improve the coverage and popularity of rehabilitation assistance services.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting implementations made with reference to the following drawings:

FIG1 is a flowchart showing the steps of an intelligent limb rehabilitation assisting method according to an embodiment;

FIG2 is a flowchart showing a method for collecting limb movement data according to an embodiment;

FIG3 is a flowchart showing a method for splicing limb motion images according to an embodiment;

FIG. 4 shows a flowchart of the steps of a method for extracting feature points from a limb movement image according to an embodiment of the present invention.

Detailed ways

The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor methods and/or microcontroller methods.

It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and/or" used herein includes any and all combinations of one or more of the listed associated items.

Please refer to Figures 1 to 4. The present application provides an intelligent limb rehabilitation assisting method, comprising the following steps:

Step S1: collecting limb movement data through an image sensor to obtain limb movement image data;

Specifically, an RGB camera or a depth camera is used to shoot the user to obtain image data of limb movement. For example, a camera can be installed on the wall or ceiling of a rehabilitation training room to capture the limb movement of the user during rehabilitation training.

Step S2: performing posture estimation on the limb motion image data to obtain posture estimation data, and performing key point detection on the posture estimation data to obtain limb motion key point detection data;

Specifically, a deep learning model, such as OpenPose or PoseNet, is used to estimate the posture of the limb motion image, thereby obtaining the user's posture estimation data. Then, key points, such as the joint positions of the elbow, wrist, knee, etc., are detected from the posture estimation data to obtain key point detection data of limb motion.

Specifically, OpenPose is used to perform posture estimation on the limb motion images to obtain posture estimation data. The posture estimation data contains the coordinates and confidence of each joint point, of which there are 18 key points in total. Key point information is extracted from the posture estimation data, including joint positions such as shoulders, elbows, wrists, hips, knees and ankles. The position information of 6 key points, including shoulders, elbows, wrists, hips, knees and ankles, is extracted from the posture estimation data.

Step S3: performing action recognition on the limb movement key point detection data to obtain limb movement action recognition data;

Specifically, machine learning or deep learning algorithms are used to process and analyze the limb movement key point detection data to identify the specific actions performed by the user. For example, a recurrent neural network (RNN) or a convolutional neural network (CNN) can be used to classify sequence data to identify the user's actions, such as raising a hand, bending a knee, etc.

Specifically, for each frame of the image, the key point data must first be converted into a feature vector, including the relative position, angle information, speed information, etc. between the joints. And several consecutive frames of data in the time series are merged into a sequence to capture the dynamic characteristics of the action. Use a deep learning model, such as a recurrent neural network (RNN) or a convolutional neural network (CNN), to train the extracted features. The training data set includes the labeled limb movement key point detection data and the corresponding action labels, such as raising hands and bending over. For new limb movement key point detection data, it is input into the trained model for prediction to obtain the corresponding action label on each frame of the image. The final limb movement action recognition data contains the actions performed on each frame of the image, as well as the corresponding time information.

Step S4: Generate motion parameters for the limb motion key point detection data according to the limb motion action recognition data to obtain limb motion parameter data for performing intelligent limb rehabilitation auxiliary operations.

Specifically, based on the action recognition data and key point detection data, the user's limb movement parameters, such as angle, speed, acceleration, etc., are calculated to reflect the user's movement state and the quality of the action. For example, limb movement parameter data can be generated through the analysis of joint angle changes and movement trajectories, and personalized rehabilitation assistance guidance and feedback can be provided to users based on these parameter data.

Specifically, the user is photographed by an RGB camera or a depth camera to capture image data of limb movement. Assume that the resolution of the image is 1920x1080 pixels. Use a deep learning model for posture estimation and key point detection, such as OpenPose. Process the image data to obtain posture estimation data and key point detection data. Assume that the coordinate position and posture data detected by the image data of each limb movement have 100 key points. The coordinate position of each key point is represented by (x, y) coordinates, and the posture data is represented by an angle value. Action recognition is performed on the key point detection data, and a recurrent neural network (RNN) is used to classify the key point sequence. For example, the position coordinates and posture data of each key point can be used as sequence input, and the classification result of the action can be obtained after being processed by the RNN model. Assume that there are 10 different action classifications in total. According to the action recognition data and key point detection data, various parameters of limb movement are calculated, for example, the change of joint angle, movement speed, acceleration, etc. can be calculated. Assume that for each joint, the device calculates its angle change, speed and acceleration values. Taking the elbow joint as an example, assume that the angle of the elbow joint ranges from 0 to 180 degrees, the velocity ranges from 0 to 100 pixels/second, and the acceleration ranges from -10 to 10 pixels/second².

The present invention uses image sensors to collect limb movement data, avoiding invasive detection or operation of users and improving user comfort and acceptance. The data collected by the image sensor can be processed and analyzed in real time, so that rehabilitation assistance operations can be adjusted and fed back in a timely manner according to the actual situation of the user. By performing posture estimation and key point detection on limb movement image data, the user's movement state and key point information can be accurately obtained, thereby achieving parameter formulation, generation and monitoring that is more in line with the user's actual situation. The method uses motion recognition technology to analyze limb movement key point detection data, and can automatically identify the user's movement without manual intervention. And the corresponding motion parameters are automatically generated according to the recognition results, further simplifying the operation process of rehabilitation assistance operations. Compared with traditional rehabilitation assistance methods, this method does not require the use of expensive equipment or complex operations, has a lower cost, and is easy to implement and promote, which helps to improve the coverage and popularity of rehabilitation services.

Preferably, step S1 specifically includes:

Step S11: collecting first limb movement data at a preset first threshold angle data through an image sensor to obtain first limb movement image data;

Specifically, an RGB camera or a depth camera is used to photograph the user, and a preset first threshold angle is set, such as setting the maximum extension angle of the arm. Then, the arm movement is monitored during the acquisition process. When the arm reaches the preset angle, the frame image data is recorded as the first limb movement image data.

Specifically, the user is photographed through an RGB camera, and the preset first threshold angle is set to 120 degrees, that is, the maximum extension angle of the arm. During the acquisition process, the system monitors the movement of the arm in real time. When the extension angle of the arm reaches the preset 120 degrees, the frame image data is recorded as the first limb movement image data. When the user is performing rehabilitation training, the system will continuously capture images of the arm and calculate the extension angle of the arm in real time through an image processing algorithm. When the angle of the arm reaches or exceeds 120 degrees, the system will automatically save the frame image data as the first limb movement image data.

Specifically, in another case, the preset first threshold angle data is the angle threshold set when collecting limb movement data, which is used to determine the shooting angle of the image sensor. For example, if the set first threshold angle is 60 degrees, the image sensor will start collecting limb movement data when the shooting angle is 60 degrees.

Step S12: collecting second limb movement data at a preset second threshold angle data through an image sensor to obtain second limb movement image data;

Specifically, an RGB camera or a depth camera is also used to shoot the user, and a preset second threshold angle is set, such as setting the maximum bending angle of the arm. Then, the arm movement is monitored during the acquisition process. When the arm reaches the preset angle, the frame image data is recorded as the second limb motion image data.

Specifically, the user is also photographed through the RGB camera, and the preset second threshold angle is set to 60 degrees, that is, the maximum bending angle of the arm. During the acquisition process, the system monitors the movement of the arm in real time. When the bending angle of the arm reaches the preset 60 degrees, the frame image data is recorded as the second limb motion image data. When the user performs rehabilitation training, the system will continuously capture images of the arm and calculate the bending angle of the arm in real time. When the angle of the arm reaches or exceeds 60 degrees, the system will automatically save the frame image data as the second limb motion image data.

Specifically, in another case, the preset second threshold angle data is the angle threshold set when collecting limb movement data, which is used to determine the shooting angle of the image sensor. For example, if the set second threshold angle is 120 degrees, the image sensor will start collecting limb movement data when the shooting angle is 120 degrees.

Step S13: performing image stitching on the first limb motion image data and the second limb motion image data to obtain limb motion image data.

Specifically, the first limb motion image data and the second limb motion image data are spliced, and image processing software or programming can be used to achieve image splicing. For example, the two images can be superimposed or collaged in a specific manner. For example, based on the known shooting angle and camera parameters (preset first threshold angle data and preset second threshold angle data), a perspective transformation matrix can be calculated to align the two images in space. The perspective transformation adjusts the geometric shape and spatial position of the image so that the two images are visually aligned. The adjusted image is cropped or filled to ensure that they have the same size and border. Alternatively, a feature point matching algorithm, including SIFT (scale-invariant feature transform) or SURF (speeded up robust features) is used to detect and match feature points in the two images, and then the image is adjusted in position based on the matching results. The first limb motion image and the second limb motion image are fused (image fusion based on a hybrid model, pixel-level blending or gradient blending) so that the two images can smoothly transition and merge together, and a gradient-based edge alignment or edge filling technique is used to adjust the position of the image edge so that the two images present continuous edges at the splicing point to generate an image containing complete limb motion information. The obtained image data is the final limb motion image data for processing and analysis in subsequent steps.

In the present invention, by using an image sensor, the user's limb movement data can be collected in real time and accurately. The first and second threshold angle data can be adjusted according to the specific situation of the user to adapt to the athletic ability of different users, so that the collected data is more operational and accurate. By setting different threshold angles, limb movement data can be collected from different angles, so that the user's movement posture can be observed and analyzed from multiple perspectives, and the state and characteristics of limb movement can be more accurately evaluated. The first and second limb movement image data are spliced to increase the information richness of the image, and the spliced image can present a more complete and comprehensive limb movement state, which helps to improve the accuracy and stability of subsequent posture estimation and key point detection.

Preferably, step S13 is specifically:

Step S131: performing first limb angle detection on the first limb motion image data according to preset first threshold angle data to obtain first limb angle detection data;

Specifically, according to the preset first threshold angle data, the first limb motion image data is deformed, such as by affine transformation or perspective transformation, and then the contour or key points of the limb are extracted using an edge detection algorithm or a feature point detection algorithm, and the angle of the limb is calculated based on this information.

Step S132: performing second limb angle detection on the second limb motion image data according to preset second threshold angle data to obtain second limb angle detection data;

Specifically, according to the preset second threshold angle data, the second limb motion image data is deformed, such as by affine transformation or perspective transformation, and then the second limb image is angle detected using a method similar to step S131 to obtain angle detection data of the second limb.

Step S133: performing image correction on the first limb motion image data according to the first limb angle detection data to obtain first limb motion image correction data, and performing image correction on the second limb motion image data according to the second limb angle detection data to obtain second limb motion image correction data;

Specifically, the first limb motion image data and the second limb motion image data are corrected according to the first limb angle detection data and the second limb angle detection data. For example, the image can be rotated, scaled or cropped according to the detected angle information so that the position and proportion of the limb in the image meet expectations.

Step S134: extracting feature points from the first limb motion image correction data and the second limb motion image correction data to obtain first limb motion image feature point data and second limb motion image feature point data;

Specifically, for the first limb motion image correction data and the second limb motion image correction data, key feature points are extracted from the images using a feature point extraction algorithm, such as SIFT (Scale Invariant Feature Transform) or SURF (Speeded Up Robust Features), etc. These feature points are usually prominent locations such as corners and edges in the image.

Step S135: performing feature matching on the first limb motion image feature point data and the second limb motion image feature point data to obtain limb motion image feature matching data;

Specifically, feature matching is performed on the feature point data of the first limb motion image and the feature point data of the second limb motion image, and a matching algorithm (such as a nearest neighbor-based method) is used to match the feature points in the first image with the feature points in the second image to establish a corresponding relationship between them.

Step S136: performing perspective transformation on the first limb movement image data and the second limb movement image data according to the limb movement image feature matching data to obtain limb movement image perspective transformation data;

Specifically, based on the feature matching data, the first limb motion image data and the second limb motion image data are corrected and aligned using perspective transformation technology. Perspective transformation can correct the distortion and distortion in the image so that the two images are aligned and consistent in space.

Step S137: performing image fusion on the first limb motion image perspective transformation data and the second limb motion image perspective transformation data to obtain limb motion image fusion data;

Specifically, the first limb motion image data and the second limb motion image data after perspective transformation are fused by image superposition, blending or weighted averaging techniques to retain important information of the two images and eliminate overlap and inconsistency.

Step S138: Perform edge repair on the limb motion image fusion data to obtain limb motion image data.

Specifically, edge restoration is performed on the image fusion data. The purpose of edge restoration is to fill in the missing parts of the image or repair the discontinuity at the edge to ensure the integrity and continuity of the image, which is achieved through methods such as interpolation algorithms or pixel filling based on edge information.

In the present invention, steps S131 and S132 use angle detection technology to accurately detect the angle information of the first and second limbs. Through image correction (step S133), the collected image data can be corrected to reduce the error introduced by angle deviation and improve the accuracy of subsequent processing. Steps S134 and S135 use feature point extraction and matching technology to extract key feature points from the corrected image and associate them through feature matching, which helps to accurately capture the key information of limb movement and maintain consistency. Steps S136 and S137 use perspective transformation and image fusion technology to synthesize the two corrected images to obtain more comprehensive and complete limb movement image data, improve the information richness and quality of the image, and thus more accurately analyze limb movement. Step S138 performs edge repair on the image, which can repair the edge problems introduced in the image due to correction, transformation and other processes, making the image clearer and more accurate, and improving the effect of subsequent processing.

Preferably, step S134 is specifically as follows:

Step S101: performing cluster calculation on the first limb motion image correction data and the second limb motion image correction data to obtain first limb motion image cluster data and second limb motion image cluster data;

Specifically, a clustering algorithm, such as K-means clustering or hierarchical clustering, is used to perform clustering calculation on the first limb motion image correction data and the second limb motion image correction data. Each cluster represents a set of similar pixel points in the image, thereby obtaining the first limb motion image clustering data and the second limb motion image clustering data.

K-means clustering calculation is performed on the first limb motion image correction data and the second limb motion image correction data respectively. Assume that the first limb motion image clustering data obtained contains 5 clusters, and the pixel points of each cluster are: cluster 1: contains 30 pixels, cluster 2: contains 25 pixels, cluster 3: contains 20 pixels, cluster 4: contains 15 pixels, cluster 5: contains 10 pixels, and the clustering of the second limb motion image clustering data is similar, also containing 5 clusters, and the number of pixels in each cluster is different.

Step S102: extracting cluster centers of the first limb movement image cluster data and the second limb movement image cluster data to obtain first image cluster center data and second image cluster center data;

Specifically, a representative pixel point is selected from each cluster as the cluster center, and a pixel point with the smallest average internal distance in the cluster can be selected as the cluster center to obtain first image cluster center data and second image cluster center data for subsequent processing.

Representative pixels are extracted from each cluster as cluster centers. Assume that the cluster centers extracted from the first limb motion image cluster data are: Cluster 1 Center: (10, 20), Cluster 2 Center: (30, 40), Cluster 3 Center: (50, 60), Cluster 4 Center: (70, 80), Cluster 5 Center: (90, 100). Similarly, the cluster centers extracted from the second limb motion image cluster data also have corresponding coordinates.

Step S103: extracting cluster radius from the first limb motion image cluster data and the second limb motion image cluster data to obtain first cluster radius data and second cluster radius data;

Specifically, the average distance from all pixels in each cluster to the cluster center is calculated as the cluster radius of the cluster, and the cluster radius data of the first limb motion image cluster data and the cluster radius data of the second limb motion image cluster data are obtained.

The first limb motion image cluster data and the second limb motion image cluster data are clustered and the average radius of each cluster is calculated. It is assumed that the cluster radius data of the first limb motion image cluster data are 15, 20, 25, 30, 35, and the cluster radius data of the second limb motion image cluster data are 18, 22, 27, 32, 38.

Step S104: performing weighted calculation on the first cluster radius data using the first limb motion image cluster data to obtain first cluster radius weighted data, and performing weighted calculation on the second cluster radius data using the second limb motion image cluster data to obtain second cluster radius weighted data;

Specifically, the first limb motion image cluster data is analyzed and processed to obtain the first cluster radius data. According to the distribution and characteristics of the first limb motion image cluster data, the first cluster radius data is weighted using a weighted calculation method. For example, weighted calculation can be performed based on factors such as the size, density, and shape of each cluster to obtain the first cluster radius weighted data. Similarly, the second limb motion image cluster data is analyzed and processed to obtain the second cluster radius data, and the weighted calculation method is used to obtain the second cluster radius weighted data.

Specifically, the cluster radius data are weighted using a linear weighted calculation method, and the weights are 0.2, 0.3, 0.4, 0.5, and 0.6 respectively. For the first limb motion image, the weighted cluster radius data are 3.0, 6.0, 10.0, 15.0, and 21.0, while for the second limb motion image, the weighted cluster radius data are 3.6, 6.6, 10.8, 16.0, and 22.8.

Step S105: performing domain pixel selection on the first image cluster center data using the first cluster radius weighted data to obtain first limb motion image domain pixel data, and performing domain pixel selection on the second image cluster center data using the second cluster radius weighted data to obtain second limb motion image domain pixel data;

Specifically, the first cluster radius weighted data is used to analyze the first image cluster center data to determine the domain range of each cluster center. According to the first cluster radius weighted data, the domain pixel data associated with each cluster center is selected. The size and shape of the domain can be determined according to the weighted degree to ensure that the selected pixels are representative and effective. Similarly, the second cluster radius weighted data is used to analyze the second image cluster center data and select domain pixels.

Specifically, the domain range of each cluster center is determined according to the weighted cluster radius data, and the corresponding pixels are selected as domain pixel data. Assume that in the first limb motion image, 100 pixels around each cluster center are selected as domain pixel data. The same method is adopted to select domain pixel data for the second limb motion image.

Step S106: Perform dynamic pixel grayscale judgment on the pixel data in the first limb motion image area and the pixel data in the second limb motion image area to obtain the first limb motion image feature point data and the second limb motion image feature point data.

Specifically, dynamic pixel grayscale judgment is performed on the pixel data in the field of the first limb motion image and the pixel data in the field of the second limb motion image. Dynamic pixel grayscale judgment is to classify or filter the pixels in the image according to the grayscale value and motion characteristics of the pixels. For example, the pixels can be classified according to the brightness, contrast, color and other characteristics of the pixels, combined with the information such as the motion direction and speed, so as to filter out representative feature point data. Finally, the feature point data of the first limb motion image and the feature point data of the second limb motion image are obtained for subsequent motion parameter generation and rehabilitation auxiliary work.

Specifically, for the selected domain pixel data, the representative feature point data is screened out according to the gray value and motion characteristics of the pixel. Assume that 50 feature point data are finally screened out from each group of domain pixel data according to the gray value and motion characteristics of the pixel for subsequent analysis and processing.

Specifically, the gray value of a pixel is , where/> Indicates a row, /> The goal of dynamic pixel grayscale judgment is to determine whether the pixel belongs to an action feature point. First, define the dynamic pixel grayscale judgment function/> , represents the pixel gray value/> The judgment result of the action feature point is:

;

Next, whether the pixel is an action feature point can be determined by calculating the difference between the grayscale value of the pixel and the grayscale values of surrounding pixels. Indicates pixel point/> The surrounding neighborhood pixel set, the dynamic pixel grayscale judgment function can be defined as:

;

in is the dynamic pixel grayscale judgment threshold data, which is used to control the sensitivity of dynamic pixel grayscale judgment. The gray value of the pixel and its surrounding pixels (/> Indicates/> The range of variation, such as/> = (1,1) means/> Pixels within the surrounding range of 1, such as/> 、/> 、/> , 、/> 、/> 、/> 、/> , ) has a difference in average grayscale value greater than the threshold /> , then the pixel is judged to be an action feature point (i.e. ), otherwise not (ie/> ).

In step S134 of the present invention, the image data can be processed and analyzed in a refined manner, which helps to more accurately capture the feature information in the image and improve the accuracy and reliability of subsequent processing. In steps S104 and S105, weighted calculation and domain pixel selection technology are used to perform weighted processing and pixel selection on the image data according to clustering data and cluster radius data, highlight the key information in the image, and improve the accuracy and stability of feature point extraction. In step S106, dynamic pixel grayscale judgment is performed, and feature points can be screened and judged according to the grayscale information of the image, reducing the sensitivity to noise and interference, and improving the recognition and extraction effect of feature points. The method analyzes image data at multiple levels and angles to improve the accuracy and stability of feature point extraction of limb movement images. Traditional feature point extraction often uses feature point acquisition based on a fixed threshold. The present invention performs adaptive adjustment through the image's own characteristics, thereby improving the extraction effect of feature points and the quality of image data, and overcoming the problems of insufficient accuracy and unstable extraction effect in the prior art.

Preferably, step S106 is specifically:

Obtain light angle data and light intensity data through light sensors;

Specifically, a light sensor, such as a photoresistor or a photodiode, is used to measure the light angle and light intensity in the environment. To measure the light angle, a mechanical or electronic steering method can be used to make the photoresistor or photodiode face the direction of interest. Then, the light angle is determined by rotating or moving the sensor and measuring the light intensity at different positions. Multiple photoresistors or photodiodes are configured in the device and arranged into an array. By placing multiple sensors (photoresistors or photodiodes) in the array, light is captured from different directions. Based on the measurement results of these sensors, the angle information of the light can be inferred. Specifically, if the light from a specific direction is strong, the corresponding sensor will show a higher signal intensity, while light from other directions will result in a lower signal intensity. By comparing the signal strengths of different sensors, the incident angle of the light is determined. These sensors can be installed on the device to obtain ambient light information in real time during motion image data acquisition.

Performing angle calculation on preset first threshold angle data and preset second threshold angle data according to the illumination angle data to obtain first shooting illumination angle data and second shooting illumination angle data;

Specifically, the shooting angle data of each image is obtained. Now, the device compares these shooting angles with the preset first threshold angle and second threshold angle to calculate the actual illumination angle data. For example, if the illumination angle data is , the preset first threshold angle data is/> , the preset second threshold angle data is/> , the first shooting illumination angle data is/> , the second shooting illumination angle data is/> .

;

;

A grayscale threshold is generated according to the first shooting illumination angle data and the illumination intensity data to obtain first grayscale threshold data, and a grayscale threshold is generated according to the second shooting illumination angle data and the illumination intensity data to obtain second grayscale threshold data.

The pixel data in the first limb motion image field is subjected to pixel grayscale judgment using the first grayscale threshold data to obtain the first limb motion image feature point data, and the pixel data in the second limb motion image field is subjected to pixel grayscale judgment using the second grayscale threshold data to obtain the second limb motion image feature point data.

Specifically, the image used is a grayscale image (or if it is a non-grayscale image, it is first converted to a grayscale image to determine the feature points), and the feature points are determined according to the grayscale value of the pixel. Set the first grayscale threshold to 100 and the second grayscale threshold to 200. For the pixel data in the first limb motion image field: traverse each pixel and obtain its grayscale value. If the grayscale value of the pixel is greater than or equal to the first grayscale threshold, mark the pixel as a feature point and add its coordinates to the first limb motion image feature point data. For the pixel data in the second limb motion image field: Similarly, traverse each pixel and obtain its grayscale value. If the grayscale value of the pixel is greater than or equal to the second grayscale threshold, mark the pixel as a feature point and add its coordinates to the second limb motion image feature point data.

For the first limb motion image field pixel data: Assume that the grayscale value of a pixel selected by the device is 120, and the first grayscale threshold is 100. Since 120 is greater than or equal to 100, the device marks the pixel as a feature point and adds its coordinates to the first limb motion image feature point data. For the second limb motion image field pixel data: Similarly, assume that the grayscale value of another pixel selected by the device is 180, and the second grayscale threshold is 200. Since 180 is less than 200, the device does not mark the pixel as a feature point.

In the present invention, by using the illumination angle data obtained by the illumination sensor, the influence of the illumination direction on the image can be considered. Calculating the grayscale threshold according to the illumination angle data helps to adaptively adjust the image according to different illumination conditions, thereby improving the stability and robustness of image processing. Step S106 dynamically generates the grayscale threshold according to the illumination angle data and the illumination intensity data, and can adjust the grayscale threshold of the image according to the real-time illumination conditions, thereby avoiding the problem of unstable or misjudgment of image processing caused by different illumination conditions, and improving the accuracy of extracting image feature points. By performing grayscale judgment on image pixels according to the grayscale threshold generated by the illumination angle data, adaptive pixel grayscale judgment can be achieved, which helps to accurately extract the feature points of the image under different illumination conditions, thereby overcoming the problem that the feature extraction effect of the prior art is unstable when the illumination changes greatly. Step S106 calculates the grayscale threshold and performs pixel grayscale judgment according to the illumination angle data, thereby improving the robustness of image processing. For images under different illumination conditions, feature points can be extracted more accurately, thereby improving the effect and reliability of subsequent processing.

Preferably, the limb movement key point detection data includes first limb movement key point detection data and second limb movement key point detection data, and step S2 is specifically:

Step S21: performing posture estimation on the limb motion image data to obtain posture estimation data;

Specifically, a deep learning model or a traditional computer vision algorithm is used to process the limb motion image data to obtain the limb posture information. For example, a posture estimation model based on a convolutional neural network (CNN), such as OpenPose, can be used to detect the positions of key points of human posture.

Step S22: performing local key point detection on the posture estimation data to obtain first limb movement key point detection data;

Specifically, based on the pose estimation data, by defining a region of interest (ROI) around a specific body part or joint, a specific keypoint detection algorithm (such as a single keypoint detector based on deep learning) is used to detect local keypoints. For example, for hand motion keypoint detection, a deep learning model for hand keypoint detection, such as the Hand Keypoint Detection model, can be used.

Step S23: Perform global key point detection on the posture estimation data to obtain second limb movement key point detection data.

Specifically, the posture estimation data is used to perform comprehensive key point detection on the entire human posture to obtain global key point information. This is achieved by applying a global key point detection algorithm to the entire image, such as object detection algorithms such as YOLO or Faster R-CNN, which can simultaneously detect multiple key points and provide their position information, thereby obtaining global key point detection data.

In the present invention, through the posture estimation of step S21, the overall posture information of the limb can be obtained, which helps to better understand the overall motion state of the limb and provide an important reference for subsequent key point detection. Step S22 performs local key point detection on the posture estimation data, which can finely identify the local key points of the limb. These local key points can represent the important parts and motion characteristics of the limb, which helps to more accurately analyze and understand the details of the limb movement. Step S23 performs global key point detection on the posture estimation data, which can capture the key point information of the limb as a whole. The global key points can reflect the overall structure and motion state of the limb, which helps to more comprehensively analyze and understand the overall motion characteristics of the limb. The present invention can improve the accuracy and stability of key point detection, help to more accurately identify and locate the key points of the limb, and provide reliable data support for subsequent action recognition and parameter generation.

Preferably, step S22 is specifically:

Perform branch network detection selection according to the posture estimation data to obtain branch network detection data;

Specifically, a multi-branch network architecture is designed, where each branch is responsible for detecting key points of a specific body part. For example, one branch can be designed to detect key points of the head, another branch can be designed to detect key points of the arm, another branch can be designed to detect key points of the leg, etc. These branches can be separate convolutional neural networks or network structures with shared parameters.

Perform key point detection on specific parts of the posture estimation data according to the branch network detection data to obtain key point detection data of specific parts;

Specifically, based on the results detected by each branch network, the key points of a specific body part are located in the pose estimation data. For example, if a branch network is specifically used to detect arm key points, the position of the arm can be determined in the pose estimation data based on the detection results of the branch, and the arm key point information can be extracted.

The key point detection data of specific parts corresponding to the detection data of different branch networks are feature fused to obtain the first limb movement key point detection data.

Specifically, the key point detection data of specific body parts detected by each branch network is feature fused to comprehensively reflect the key point information of the overall limb movement, which is achieved through simple feature splicing, weighted averaging, etc. For example, the key point data of arms, legs, etc. detected by different branches can be fused to obtain complete limb movement key point detection data.

A convolutional neural network (CNN) with multiple branches is designed, each branch is responsible for detecting key points of a specific part of the human body, such as input layer: the input is posture estimation data, which contains the location information of the human posture key points. Multi-branch convolutional neural network: head branch: input layer: posture estimation data, convolution layer: multiple convolution layers for feature extraction, pooling layer: reduce the size of the feature map, retain the main features, convolution layer: further extract features, fully connected layer: map the features to the coordinates of the head key points, output layer: the coordinates of the head key points. Shoulder branch: input layer: posture estimation data, convolution layer: multiple convolution layers for feature extraction, pooling layer: reduce the size of the feature map, retain the main features, convolution layer: further extract features, fully connected layer: map the features to the coordinates of the shoulder key points, output layer: the coordinates of the shoulder key points. Arm branch: input layer: posture estimation data, convolution layer: multiple convolution layers for feature extraction, pooling layer: reduce the size of the feature map, retain the main features, convolution layer: further extract features, fully connected layer: map the features to the coordinates of the arm key points, output layer: the coordinates of the arm key points. Knee branch: Input layer: pose estimation data, Convolution layer: multiple convolution layers for feature extraction, Pooling layer: reduce the size of feature map, retain the main features, Convolution layer: further extract features, Fully connected layer: map features to the coordinates of knee key points, Output layer: coordinates of knee key points. Ankle branch: Input layer: pose estimation data, Convolution layer: multiple convolution layers for feature extraction, Pooling layer: reduce the size of feature map, retain the main features, Convolution layer: further extract features, Fully connected layer: map features to the coordinates of ankle key points, Output layer: coordinates of ankle key points. In order to train such a network, by obtaining pose estimation training data and training the model/network based on the pose estimation training data, the loss function of the key point detection task is adopted, such as Mean Squared Error (MSE) or Smooth L1 Loss. For each key point position , the loss function is expressed as:

;

is the model training loss value, /> The number of training data for posture estimation, /> is the sequence item data of posture estimation training data,/> For the first/> The network prediction data of the pose estimation training data, /> For the first/> The real position data of the pose estimation training data. The optimization function can choose Stochastic Gradient Descent (SGD) or its variants, such as Adam or RMSProp. Input: pose estimation data, which contains the position information of the key points of the human body pose. Output: the coordinates of the key points of each specific part of the human body, such as (x, y) coordinate pairs. For example: the first branch is used to detect the key points of the head. The second branch is used to detect the key points of the arm. The third branch is used to detect the key points of the leg. Each branch network obtains the key point detection data of the corresponding part by processing the pose estimation data. For example: the head branch detects the position of the head key point. The arm branch detects the position of the arm key point. The leg branch detects the position of the leg key point. According to the results detected by each branch network, the key point position of a specific body part is determined in the pose estimation data. For example: the detection result of the head branch can be used to locate the head position in the pose estimation data and extract the head key point information. The detection result of the arm branch can be used to locate the arm position in the pose estimation data and extract the arm key point information. The detection result of the leg branch can be used to locate the leg position in the pose estimation data and extract the leg key point information. The key point detection data of specific body parts detected by different branches are feature fused to obtain complete limb movement key point detection data. For example, the key point data detected by the head, arms, legs and other branches can be simply feature spliced or weighted averaged to comprehensively reflect the key point information of the overall limb movement.

In the present invention, by introducing multiple branch networks, each branch network is responsible for detecting the key points of a specific part, which can more accurately capture the motion information of different parts in limb movement, thereby improving the accuracy and comprehensiveness of detection. On the basis of the branch network detection selection, key point detection is performed for each part separately, so that the detection results are more detailed and specific, which helps to more accurately analyze and evaluate the user's motion state in the process. Feature fusion of the key point data of specific parts detected by different branch networks can take into account the motion information of different parts, thereby obtaining more accurate limb movement key point detection data, which helps to improve the rehabilitation assistance system's understanding and analysis capabilities of the user's motion state.

Preferably, step S3 specifically includes:

Extracting features from the limb movement key point detection data to obtain limb movement key point feature data;

Specifically, meaningful features are extracted from limb motion key point detection data through various feature extraction methods, such as using deep neural networks (such as convolutional neural networks) to extract image features around key points, or using manually designed feature descriptors (such as SIFT, HOG, etc.) to describe the appearance and shape characteristics of key points.

Performing feature selection on the key point feature data of limb movement to obtain key point feature selection data;

Specifically, the most representative and discriminative features are selected through feature selection algorithms to reduce data dimensions and redundant information. For example, principal component analysis (PCA), linear discriminant analysis (LDA), information gain and other methods can be used for feature selection to select the most critical feature subset for action recognition.

The key point feature selection data is used for action recognition to obtain limb movement action recognition data.

Specifically, a machine learning or deep learning algorithm is used to perform action recognition on the data after feature selection. For example, classifiers such as support vector machine (SVM), random forest (Random Forest), and convolutional neural network (CNN) can be used for action recognition. These models are trained to distinguish different limb movement actions, and these models are used to classify new action data during testing, thereby obtaining limb movement action recognition data.

In the present invention, through the feature extraction in step S3, key point feature data representing the characteristics of limb movement can be extracted from the key point detection data of limb movement, which is helpful to convert complex limb movement into more specific and easier to analyze feature data. The feature selection process in step S3 can select the most representative and discriminative features from the extracted key point feature data to reduce the feature dimension and improve the accuracy of recognition, which helps to reduce the complexity of data processing, while retaining important information about limb movement and improving the effect of action recognition. The data extracted and selected by the features can better reflect the characteristics of limb movement, so that the action recognition is more accurate. By selecting representative features and combining appropriate recognition algorithms, different limb movement actions can be more reliably identified, providing more accurate data support for intelligent limb rehabilitation assistance. Through feature selection, the calculation amount of the recognition algorithm can be reduced and the calculation efficiency can be improved. The carefully selected features can better reflect the essential characteristics of limb movement, thereby reducing unnecessary calculation overhead and speeding up the speed of action recognition. Due to the reduction of feature dimensions and calculation amount, the system can process limb movement data more quickly and improve real-time performance.

Preferably, step S4 is specifically:

Perform action classification mapping according to the limb movement action recognition data to obtain action classification mapping data;

Specifically, an action recognition model is established, such as input layer: the input is the limb movement action recognition data, which can be the key point coordinates of the time series or the image sequence. Convolutional neural network (CNN) part: The CNN part is responsible for extracting spatial features from the input data. Multi-layer convolutional layers and pooling layers can be used to gradually extract features. Recurrent neural network (RNN) part: The RNN part is responsible for processing time series data and capturing the time evolution information of the action. The network can be constructed using RNN units such as LSTM or GRU. The output sequence of the CNN part is used as the input sequence of the RNN. Fully connected layer: A fully connected layer is added to the output of the RNN part to convert the sequence information into a fixed-length vector representation. A Dropout layer is added to reduce overfitting. Output layer: The output layer uses the softmax function for classification, and each category represents a specific action. The number of nodes in the output layer is equal to the number of action categories. Loss function and optimization function: The loss function selects the cross entropy loss function (CrossEntropy Loss) for multi-category classification tasks. The optimization function selects optimization algorithms such as Adam and SGD. Input and output: Input: limb movement action recognition data, which can be the key point coordinates of the time series or the image sequence. Output: Action classification result, that is, the probability distribution of which action category it belongs to, or the action recognition model is a set sequence mapping array. The limb movement action recognition data is input into the model for classification. Each category represents a specific action. According to the output of the model, each action is mapped to the corresponding category to obtain the action classification mapping data.

Perform action timing division according to the limb movement key point detection data and the action classification mapping data to obtain action timing division data;

Specifically, the start and end time of the action are determined based on the key point detection data, and the action is divided into time series. Using a time window-based method, the key point data is divided into several segments in time, each segment represents an action stage. The specific action category of each action stage is determined based on the action classification mapping data.

Extract motion features according to the motion time sequence division data to obtain motion feature data, wherein the motion feature data includes motion duration data, motion speed data, motion acceleration data and motion angle change data;

Specifically, features related to the action are extracted in each action stage. For example, features such as the duration, average speed, maximum acceleration, and angle changes between key points of each action stage can be calculated.

Motion parameters are generated according to the motion classification mapping data, motion timing division data and motion feature data to obtain limb motion parameter data for performing intelligent limb rehabilitation auxiliary operations.

Specifically, based on the action timing division data and action feature data, combined with the previously established action classification mapping relationship, action parameters describing each action stage are generated, including action type, start time, end time, duration, speed, acceleration, angle change and other information, which are used for the execution and monitoring of intelligent limb rehabilitation assisted tasks.

Specifically, the action recognition model can recognize the following three actions: lifting an object, keeping the object lifted, and putting the object down. When the model recognizes an action, the device maps it to the corresponding category based on its output result, for example: 1 means lifting an object, 2 means keeping the object lifted, and 3 means putting the object down. Assume that the action timing division algorithm of the device determines the following moments based on the key point detection data: the start moment of the action: t_start=10s, the end moment of the action: t_end=30s, and the device divides this time period into three stages: Stage 1: t_start to t_1 (15s), representing the process of lifting an object. Stage 2: t_1 to t_2 (20s), representing the process of keeping the object lifted. Stage 3: t_2 to t_end (30s), representing the process of putting the object down. For each stage, the device extracts the following action feature data: Duration: 5 seconds for stage 1, 5 seconds for stage 2, and 10 seconds for stage 3. Speed: The average speed of each stage is calculated based on the key point data. Acceleration: The average acceleration of each stage is calculated based on the speed data. Angle change: Calculate the angle change between key points in each stage based on key point data. Based on the action classification mapping data, action timing division data and action feature data, the device generates the following limb action parameter data: The type of each stage is determined according to the mapping relationship, for example, stage 1 is to lift the object, stage 2 is to keep the lifting state, and stage 3 is to put the object down. Start time and end time: Determined according to the timing division. The generated limb action parameter data includes information such as duration, speed, acceleration, angle change, etc.

In the present invention, by performing action classification mapping according to the limb movement action recognition data, the motion data can be mapped to a specific action category, thereby achieving classification and recognition of different actions, which helps the system to better understand the user's movement behavior and provide effective data support for subsequent rehabilitation assistance. According to the limb movement key point detection data and the action classification mapping data, the action timing division can be performed, and the entire movement process can be decomposed into different time periods or stages, which helps to analyze the movement process more carefully and accurately, and provides a more reliable basis for subsequent feature extraction and parameter generation. According to the action timing division data, action feature extraction can be performed, and information about the movement can be obtained from multiple aspects, such as duration, speed, acceleration, and angle change, etc. These feature data can reflect various aspects of the movement, and provide more reference basis for subsequent parameter generation and rehabilitation assistance. According to the action classification mapping data, the action timing division data, and the action feature data, the action parameter generation can be performed, and the type, timing, and feature information of the action can be comprehensively considered, so as to obtain more accurate limb movement parameter data.

Preferably, the present application also provides an intelligent limb rehabilitation assisting device for executing the intelligent limb rehabilitation assisting method as described above, the intelligent limb rehabilitation assisting device comprising:

A limb movement data acquisition module is used to acquire limb movement data through an image sensor to obtain limb movement image data;

A limb movement key point detection module is used to perform posture estimation on limb movement image data to obtain posture estimation data, and perform key point detection on the posture estimation data to obtain limb movement key point detection data;

The action recognition module is used to perform action recognition on the limb movement key point detection data to obtain limb movement action recognition data;

The intelligent limb rehabilitation auxiliary operation module is used to generate action parameters for limb movement key point detection data according to limb movement action recognition data, and obtain limb movement parameter data for intelligent limb rehabilitation auxiliary operations.

The beneficial effects of the present invention are: using image sensors to collect limb movement data avoids invasive detection or operation of users, and improves user comfort and acceptance. The data collected by the image sensor can be processed and analyzed in real time, so that rehabilitation assistance operations can be adjusted and fed back in a timely manner according to the actual situation of the user. By performing posture estimation and key point detection on limb movement image data, the user's movement state and key point information can be accurately obtained, thereby achieving parameter formulation, generation and monitoring that is more in line with the user's actual situation. The method uses motion recognition technology to analyze limb movement key point detection data, and can automatically identify the user's movement without manual intervention. And the corresponding motion parameters are automatically generated according to the recognition results, further simplifying the operation process of rehabilitation assistance operations. Compared with traditional rehabilitation assistance methods, this method does not require the use of expensive equipment or complex operations, has a lower cost, and is easy to implement and promote, which helps to improve the coverage and popularity of rehabilitation assistance services.

Therefore, from any point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the attached application documents rather than the above description, and it is intended that all changes within the meaning and scope of equivalent elements of the application documents are included in the present invention.

The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims (4)

1. An intelligent limb rehabilitation assisting method is characterized by comprising the following steps of:

Step S1, acquiring limb movement data through an image sensor to obtain limb movement image data, wherein the step comprises the following steps of;

Step S11: acquiring first limb movement data by an image sensor according to preset first threshold angle data to obtain first limb movement image data;

Step S12: acquiring second limb movement data by using an image sensor according to preset second threshold angle data to obtain second limb movement image data;

Step S13, including:

Step S131: performing first limb angle detection on the first limb moving image data according to preset first threshold angle data to obtain first limb angle detection data;

Step S132: performing second limb angle detection on second limb moving image data according to preset second threshold angle data to obtain second limb angle detection data;

Step S133: performing image correction on the first limb moving image data according to the first limb angle detection data to obtain first limb moving image correction data, and performing image correction on the second limb moving image data according to the second limb angle detection data to obtain second limb moving image correction data;

Step S134: extracting characteristic points of the first limb moving image correction data and the second limb moving image correction data to obtain first limb moving image characteristic point data and second limb moving image characteristic point data;

Step S135: performing feature matching on the first limb moving image feature point data and the second limb moving image feature point data to obtain limb moving image feature matching data;

step S136: performing perspective transformation on the first limb moving image data and the second limb moving image data according to the limb moving image feature matching data to obtain limb moving image perspective transformation data;

step S137: performing image fusion on the first limb moving image perspective transformation data and the second limb moving image perspective transformation data to obtain limb moving image fusion data;

Step S138: performing edge restoration on the limb moving image fusion data to obtain limb moving image data;

step S2, including:

step S21: carrying out gesture estimation on the limb moving image data to obtain gesture estimation data;

Step S22: detecting local key points of the gesture estimation data to obtain first limb movement key point detection data;

Step S23: performing global key point detection on the gesture estimation data to obtain second limb movement key point detection data;

Step S3: performing motion recognition on limb movement key point detection data to obtain limb movement motion recognition data, wherein the limb movement key point detection data comprise first limb movement key point detection data and second limb movement key point detection data;

Step S4, including:

performing action classification mapping according to the limb movement identification data to obtain action classification mapping data;

performing action time sequence division according to limb movement key point detection data and action classification mapping data to obtain action time sequence division data;

Extracting action characteristics according to the action time sequence dividing data to obtain action characteristic data, wherein the action characteristic data comprises action duration time data, action speed data, action acceleration data and action angle change data;

Generating action parameters according to the action classification mapping data, the action time sequence division data and the action characteristic data to obtain limb action parameter data so as to perform intelligent limb rehabilitation auxiliary operation;

The step S134 specifically includes:

Step S101: clustering calculation is carried out on the first limb moving image correction data and the second limb moving image correction data to obtain first limb moving image clustering data and second limb moving image clustering data;

step S102: extracting a clustering center of the first limb moving image clustering data and the second limb moving image clustering data to obtain first image clustering center data and second image clustering center data;

step S103: cluster radius extraction is carried out on the first limb moving image cluster data and the second limb moving image cluster data to respectively obtain first cluster radius data and second cluster radius data;

step S104: weighting calculation is carried out on the first cluster radius data by using the first limb moving image clustering data to obtain first cluster radius weighting data, and weighting calculation is carried out on the second cluster radius data by using the second limb moving image clustering data to obtain second cluster radius weighting data;

step S105: performing field pixel selection on the first image clustering center data by using the first cluster radius weighting data to obtain first limb moving image field pixel data, and performing field pixel selection on the second image clustering center data by using the second cluster radius weighting data to obtain second limb moving image field pixel data;

Step S106: performing dynamic pixel gray scale judgment on the first limb moving image field pixel data and the second limb moving image field pixel data to obtain first limb moving image feature point data and second limb moving image feature point data;

The step S106 specifically includes:

acquiring illumination angle data and illumination intensity data through an illumination sensor;

Performing angle calculation on preset first threshold angle data and preset second threshold angle data according to the illumination angle data to obtain first shooting illumination angle data and second shooting illumination angle data;

gray threshold generation is carried out according to the first shooting illumination angle data and the illumination intensity data to obtain first gray threshold data, gray threshold generation is carried out according to the second shooting illumination angle data and the illumination intensity data to obtain second gray threshold data;

And performing pixel gray judgment on the first limb moving image field pixel data through the first gray threshold data to obtain first limb moving image feature point data, and performing pixel gray judgment on the second limb moving image field pixel data through the second gray threshold data to obtain second limb moving image feature point data.

2. The method according to claim 1, wherein step S22 is specifically:

according to the gesture estimation data, branch network detection selection is carried out to obtain branch network detection data;

performing specific position key point detection on the gesture estimation data according to the branch network detection data to obtain specific position key point detection data;

and carrying out feature fusion on the specific position key point detection data corresponding to the different branch network detection data to obtain first limb movement key point detection data.

3. The method according to claim 1, wherein step S3 is specifically:

extracting features of the limb movement key point detection data to obtain limb movement key point feature data;

Feature selection is carried out on the feature data of the key points of the limb movement, so that feature selection data of the key points are obtained;

And performing action recognition on the key point feature selection data to obtain limb movement action recognition data.

4. A smart limb rehabilitation assistance device for performing the smart limb rehabilitation assistance method of claim 1, the smart limb rehabilitation assistance device comprising:

The limb movement data acquisition module is used for acquiring limb movement data through the image sensor to obtain limb movement image data;

The limb movement key point detection module is used for carrying out gesture estimation on limb movement image data to obtain gesture estimation data, and carrying out key point detection on the gesture estimation data to obtain limb movement key point detection data;

the motion recognition module is used for performing motion recognition on the limb motion key point detection data to obtain limb motion recognition data;

and the intelligent limb rehabilitation auxiliary operation module is used for generating action parameters of limb movement key point detection data according to limb movement action identification data to obtain limb action parameter data so as to perform intelligent limb rehabilitation auxiliary operation.