CN110544302A - Human motion reconstruction system, method and motion training system based on multi-eye vision - Google Patents

Abstract

The present application discloses a human motion reconstruction system, method and motion training system based on multi-eye vision. The corrected motion image of a target human body is captured based on at least two monocular cameras, and the motion image is subjected to a two-dimensional image of the human body. Two-dimensional joint point recognition, and then reconstruct the three-dimensional action of the target human body based on the two-dimensional joint points of the human body, without the need for the user to wear a special optical capture suit or assemble the corresponding sensor, which solves the existing depth camera human body three-dimensional action recognition method, suitable for applications Scenario requirements are high, and in complex application scenarios, technical problems with low recognition accuracy are recognized.

Description

Human motion reconstruction system, method and motion training system based on multi-eye vision

technical field

The present application relates to the technical field of motion recognition, and in particular, to a system, method and motion training system for human motion reconstruction based on multi-eye vision.

Background technique

With the development of artificial intelligence, human gesture recognition technology has made important breakthroughs based on the larger data sets and powerful computing power available in the era of big data.

The existing 3D human action recognition method is to use optical capture technology to capture the 3D action of the target human body for action recognition. The user wears a special optical capture suit with optical markings or a corresponding detection sensor to obtain the corresponding marked position to generate the relative spatial position, thereby constructing a 3D model of human body for action recognition. The method of using optical capture technology to capture the three-dimensional movement of the target human body for action recognition requires the user to wear a special optical capture suit or assemble the corresponding sensor, which is troublesome to wear, and the user has a large load, which is inconvenient for movement training and affects the user experience.

SUMMARY OF THE INVENTION

The purpose of this application is to provide a human motion reconstruction system, method and motion training system based on multi-eye vision, which is used to solve the problem that the existing three-dimensional human motion recognition method requires the user to wear a special optical capture suit or assemble a corresponding sensor. It is troublesome, and the user has a large load, which causes inconvenience to the movement training and technical problems that affect the user experience.

A first aspect of the present application provides a multi-vision-based human motion reconstruction system, including:

a camera calibration module, configured to perform monocular camera calibration according to the collected calibration point data, and the number of the monocular cameras is at least two;

a motion acquisition module, configured to acquire all motion image sequences of the target human body collected by the monocular camera, and send the motion image sequences to the two-dimensional human motion recognition module;

The two-dimensional human action recognition module is used to identify the key limb joint parts of the human body in the action image sequence, and extract the two-dimensional joint points belonging to the same target human body in each frame of the action image sequence, so as to facilitate the reconstruction of all body parts. The human skeleton of the target human body is stored, and the human skeleton information of each frame is stored and sent to the three-dimensional motion reconstruction module;

The three-dimensional motion reconstruction module is configured to restore the real positions of the two-dimensional joint points of the target human body in the three-dimensional space based on the human skeleton information in each frame, and reconstruct the three-dimensional motion of the target human body.

Optionally, it also includes: an imaging point error correction module;

The imaging point error correction module is used to correct the pixel coordinates of the two-dimensional joint points in the two-dimensional human motion recognition module when the monocular camera is offset, so as to facilitate the two-dimensional human motion The recognition module reconstructs the human skeleton of the target human body according to the two-dimensional joint points after correcting the pixel coordinates, and stores and sends the human skeleton information of each frame to the three-dimensional motion reconstruction module.

Optionally, the three-dimensional motion reconstruction module specifically includes:

The first solving sub-module is used to solve, based on the monocular camera parameters calibrated by the monocular camera, the projection imaging point perpendicular to the focus of the monocular camera projected on the imaging plane of the monocular camera at the preset value. the first real three-dimensional coordinate in the three-dimensional coordinate system;

a second solving sub-module, configured to solve the rotation matrix and translation vector of the first real three-dimensional coordinates based on the first real three-dimensional coordinates and the imaging plane parameters of the monocular camera;

a third solving submodule, configured to solve the second real three-dimensional coordinates of the two-dimensional joint points on the imaging plane in the preset three-dimensional coordinate system based on the rotation matrix and the translation vector;

The joint point reconstruction sub-module is used to make the three-dimensional coordinate point at which the minimum distance is obtained between the optical center coordinates of all the monocular cameras and the corresponding second real three-dimensional coordinate line in the preset three-dimensional coordinate system, as the The real three-dimensional joint points of the two-dimensional joint points in the preset three-dimensional coordinate system are used to reconstruct the three-dimensional motion of the target human body.

Optionally, the joint submodule is specifically used for:

Based on the least squares method of overdetermined equations, solve the straight line formed by the optical center coordinates of all the monocular cameras and the corresponding second real three-dimensional coordinates to obtain the three-dimensional coordinate point with the minimum distance, as the two-dimensional joint point in the The real three-dimensional joint points in the preset three-dimensional coordinate system are used to reconstruct the three-dimensional motion of the target human body.

Optionally, the two-dimensional human motion recognition module is specifically used for:

Based on the preset convolutional neural network, the key limb joint parts of the human body in the action image sequence are identified, and 25 joint points belonging to the same target human body in each frame of the action image sequence are extracted to facilitate the reconstruction of the target. The human body skeleton of the human body, the information of the human body skeleton of each frame is stored and sent to the three-dimensional motion reconstruction module.

A second aspect of the present application also provides a three-dimensional human motion reconstruction method, including:

Perform monocular camera calibration based on the collected calibration point data, and the number of the monocular cameras is at least two;

Acquiring at least two action image sequences of the target human body captured by the monocular camera;

Identify the key limb joint parts of the human body in the action image sequence, extract the two-dimensional joint points belonging to the same target human body in each frame of the action image sequence, so as to reconstruct the human skeleton of the target human body, and obtain each frame of the human skeleton information;

Based on the human skeleton information in each frame, the real position of the target human body in the joint point in the three-dimensional space is restored, and the three-dimensional action of the target human body is reconstructed.

Optionally, restoring the real position of the target human body in the three-dimensional space of the joint points based on the human skeleton information in each frame, and reconstructing the three-dimensional action of the target human body, specifically includes:

Based on the monocular camera parameters calibrated by the monocular camera, the first real value of the projected imaging point perpendicular to the focal point of the monocular camera on the imaging plane of the monocular camera in the preset three-dimensional coordinate system is obtained. three-dimensional coordinates;

Based on the first real three-dimensional coordinates and the imaging plane parameters of the monocular camera, solve the rotation matrix and translation vector of the first real three-dimensional coordinates;

Solving the second real three-dimensional coordinates of the two-dimensional joint points on the imaging plane in the preset three-dimensional coordinate system based on the rotation matrix and the translation vector;

In the preset three-dimensional coordinate system, the three-dimensional coordinate points that make the optical center coordinates of all the monocular cameras and the corresponding second real three-dimensional coordinate lines obtain the minimum distance are taken as the two-dimensional joint points in the The real three-dimensional joint points in the three-dimensional coordinate system are preset to reconstruct the three-dimensional motion of the target human body.

A third aspect of the present application provides an action training system, including any of the three-dimensional human action reconstruction systems described in the first aspect, and further including an action evaluation module;

The motion evaluation module is configured to perform a difference comparison between the three-dimensional motion of the target human body obtained from the three-dimensional motion reconstruction module and a preset standard motion, and output a motion evaluation result corresponding to the difference comparison result.

Optionally, the action evaluation module is specifically used for:

The three-dimensional motion of the target human body obtained from the three-dimensional motion reconstruction module and the preset standard motion are judged on the similarity of the joint point combination angle, the average curvature comparison of the joint point combined motion trajectory, and the human body joint point combined motion amount comparison;

Obtain the first action evaluation result corresponding to the result of the joint point combination angle similarity judgment, the second action evaluation result corresponding to the result of the average curvature comparison of the joint point combination motion trajectory, and the human body joint point combination exercise amount comparison. The third action evaluation result corresponding to the result;

Perform weighting processing on the first action evaluation result, the second action evaluation result and the third action evaluation result, and output the action evaluation result obtained after the weighting process.

Optionally, it also includes: a music rhythm fit module;

The music rhythm fit module is used for extracting the music features of the music being broadcast based on the audio extraction algorithm, and in units of beats, it is judged that the human body movements of the target human body in continuous frames and the preset standard movements are in the series of movements that match the rhythm. Matching degree, output beat matching result.

The present application provides a human motion reconstruction system based on multi-eye vision, including: a camera calibration module for performing monocular camera calibration according to the collected calibration point data, and the number of monocular cameras is at least two; In order to obtain the action image sequence of the target human body collected by all monocular cameras, the action image sequence is sent to the two-dimensional human action recognition module; the two-dimensional human action recognition module is used to identify the key human limb joints in the action image sequence Parts, extract the two-dimensional joint points belonging to the same target human body in each frame of the action image sequence, so as to reconstruct the human skeleton of the target human body, store and send the human skeleton information of each frame to the three-dimensional action reconstruction module; three-dimensional action reconstruction The module is used to restore the real position of the two-dimensional joint points of the target human body in the three-dimensional space based on the human skeleton information in each frame, and reconstruct the three-dimensional action of the target human body.

The human motion reconstruction based on multi-eye vision provided by the present application captures the motion image of the target human body based on at least two monocular cameras after correction, and performs two-dimensional joint point recognition of the human body on the two-dimensional image for the motion image, and then based on the human body The 2D joint points reconstruct the 3D action of the target human body, without the need for the user to wear a special optical capture suit or assemble the corresponding sensor, which solves the existing 3D action recognition method of the human body by the depth camera, which has high requirements for application scenarios and is used in complex applications. In scenarios, identify technical problems with low accuracy.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only It is an embodiment of the present application. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without any creative effort.

FIG. 1 is a schematic structural diagram of an embodiment of a multi-vision-based human motion reconstruction system provided by an embodiment of the application;

FIG. 2 is another schematic structural diagram of a multi-vision-based human motion reconstruction system provided by an embodiment of the present application;

FIG. 3 is a schematic flowchart of an embodiment of a three-dimensional human motion reconstruction method provided by an embodiment of the present application;

FIG. 4 is another schematic flowchart of an embodiment of a three-dimensional human motion reconstruction method provided by an embodiment of the present application

5 is a schematic structural diagram of an action training system provided in an embodiment of the application;

6 is a schematic diagram of an imaging point error correction of an imaging point error correction module provided in an embodiment of the present application;

FIG. 7 is a three-dimensional space theoretical layout diagram of three monocular cameras (A, B, and C in the figure) provided in an embodiment of the application.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Please refer to FIG. 1. FIG. 1 is a schematic structural diagram of a multi-vision-based human motion reconstruction system provided by an embodiment of the present application. The multi-vision-based human motion reconstruction system provided in the embodiment of the present application includes:

The camera calibration module 101 is configured to perform monocular camera calibration according to the collected calibration point data, and the number of monocular cameras is at least two.

It should be noted that, in the embodiment of the present application, at least two monocular cameras are used to obtain the action images of the target human body, and the monocular cameras used to obtain the action images of the target human body in the system need to be calibrated, and all monocular cameras are used for calibration. The collection of calibration points may be checkerboard calibration points or two-dimensional code calibration points, and the calibration method of the calibration points is not limited here. The calibration point collection only needs to be collected when the monocular camera is used for the first time, and there is no need to repeat the calibration point collection in the future. The collected calibration point data is stored in the storage module for subsequent recall. After calibrating the monocular camera according to the calibration point data, the calibration parameters of the monocular camera can be obtained.

In the embodiment of the present application, when the calibration point data is collected, the number of images collected by each monocular camera is 25 calibration point images, which can improve calibration accuracy and stability for each monocular camera.

The motion acquisition module 102 is configured to acquire the motion image sequence of the target human body collected by all monocular cameras, and send the motion image sequence to the two-dimensional human motion recognition module 103 .

It should be noted that the action image sequence of the target human body is collected by the calibrated monocular camera, and the action image sequence is sent to the two-dimensional human action recognition module 103 for two-dimensional image processing.

The two-dimensional human action recognition module 103 is used to identify the key limb joint parts of the human body in the action image sequence, and extract the two-dimensional joint points belonging to the same target human body in each frame of the action image sequence, so as to reconstruct the human skeleton of the target human body, The human skeleton information of each frame is stored and sent to the three-dimensional motion reconstruction module 104 .

It should be noted that the joint points belonging to the same person in each frame of the action image sequence are extracted to form a complete human skeleton, and the skeleton information of each frame is cached in a json format file to the database, and further provided to the 3D action Rebuild module 104 is used. In the embodiment of the present application, the two-dimensional human motion recognition module 103 uses the open-source multi-person two-dimensional pose neural network openpose to realize two-dimensional human motion recognition and reconstruct the human skeleton of the target human body.

The three-dimensional motion reconstruction module 104 is configured to restore the real position of the target human body in the three-dimensional space of the joint points based on the human skeleton information in each frame, and reconstruct the three-dimensional motion of the target human body.

It should be noted that the motion image sequence of the target human body captured by the corrected monocular camera is input into the above-mentioned two-dimensional human motion recognition module 103 to obtain the skeleton information represented by the two-dimensional joint point information. The human skeleton information in each frame restores the real position of the target human body in the three-dimensional space of the joint points, and reconstructs the three-dimensional action of the target human body.

The three-dimensional human motion reconstruction system provided in the embodiment of the present application captures the corrected motion image of the target human body based on at least two monocular cameras, performs 2D human body joint point recognition on the two-dimensional image for the motion image, and The 2D joint points reconstruct the 3D action of the target human body, without the need for the user to wear a special optical capture suit or assemble the corresponding sensor, which solves the existing 3D action recognition method of the human body by the depth camera, which has high requirements for application scenarios and is used in complex applications. In scenarios, identify technical problems with low accuracy.

As a further improvement to the human motion reconstruction system based on polycular vision in the embodiment of the present application, as shown in FIG. 2 , the human motion reconstruction system based on polyocular vision in the embodiment of the present application further includes: an imaging point error correction module 105 ;

The imaging point error correction module 105 is used to correct the pixel coordinates of the two-dimensional joint points in the two-dimensional human motion recognition module when the monocular camera is offset, so that the two-dimensional human motion recognition module can correct the pixel coordinates according to the two-dimensional The 3D joint points reconstruct the human skeleton of the target human body, and store and send the human skeleton information of each frame to the 3D motion reconstruction module.

It should be noted that, as shown in Figure 6, Figure 6 is a schematic diagram of the imaging point error correction of the imaging point error correction module. In the actual application scenario, during the installation and use of the monocular camera, there is a high probability that a monocular camera will appear. The camera shakes or shifts, which will cause imaging point errors, thereby affecting the recognition results. To solve this technical problem, an imaging point error correction module 105 is provided in this embodiment of the present application. The pixel coordinates of the two-dimensional joint points in the two-dimensional human motion recognition module 103, so that the two-dimensional human motion recognition module 103 reconstructs the human skeleton of the target human body according to the two-dimensional joint points after correcting the pixel coordinates, and converts the human skeleton of each frame The information is stored and sent to the 3D motion reconstruction module 104 . The working principle of the imaging point error correction module 105 is explained below with reference to FIG. 6 :

The imaging plane EFHG is the original imaging plane of the monocular camera A (that is, the normal imaging plane before shaking or offset), and the imaging plane BCJK is the imaging plane of the monocular camera A after shaking or offset. The image recognition algorithm that comes with opencv can recognize that the image pixel coordinates of the marker point D before and after the offset are (u _i ,v _i ) and ( _{u i} ',vi '), respectively. According to the focal length of the calibrated monocular camera f performs the following operations to correct the pixel coordinates (u _w ', v _w ') of the error imaging point to (u _w , v _w ). The operation formula is:

where Δθ and are the elevation angle difference and the direction angle difference of the imaging point of the marker point D on the spherical coordinate system before and after the imaging plane is shifted; θ' and are the elevation angle and direction angle of any error imaging point on the spherical coordinate system after the imaging plane is shifted.

According to the above formula, the error correction can be performed on the pixel coordinates of any imaging point in the monocular camera A, which avoids the technical problem of poor recognition results caused by imaging point errors caused by the shaking or offset of the monocular camera.

As a further improvement, the two-dimensional human motion recognition module 103 in the embodiment of the present application is specifically used for:

Based on the preset convolutional neural network, the key limb joint parts of the human body in the action image sequence are identified, and 25 joint points belonging to the same target human body in each frame of the action image sequence are extracted, so as to reconstruct the human skeleton of the target human body , and the human skeleton information of each frame is stored and sent to the three-dimensional motion reconstruction module 104 .

It should be noted that the preset convolutional neural network adopts the non-parametric representation method Part AffinityFields to match the key limb joint parts in the two-dimensional image with the corresponding individuals and reconstruct the human joint skeleton. The main idea is to use the bottom-up parsing steps of the greedy algorithm to achieve high accuracy and real-time performance. Body part localization and association are performed simultaneously on two branches. It can identify the positions of 25 joint points of the human body in two-dimensional images with high accuracy and real-time speed.

The process of using a preset convolutional neural network to identify a two-dimensional human pose is as follows: input a two-dimensional human image video sequence with a pixel scale of w×h, locate the key points of the individuals appearing in each frame of the image video sequence, and then Using a CNN neural network model with a two-branch multi-stage architecture for prediction, the first branch of CNN predicts a set of confidence maps S of human body parts and a set of two-dimensional limb vector fields J (each body part has a corresponding confidence map, Each human limb corresponds to a vector), and after parsing the confidence map and PAF through greedy reasoning, the 2D skeleton key point information of each person on each frame of the image video sequence is output, and there are a total of 25 human body joint points.

As a further improvement, the three-dimensional motion reconstruction module 104 in this embodiment of the present application specifically includes:

The first solving sub-module 1041 is used to solve the projection imaging point in the preset three-dimensional coordinate system that is perpendicular to the focus of the monocular camera on the imaging plane of the monocular camera based on the monocular camera parameters calibrated by the monocular camera. The first real three-dimensional coordinates.

The second solving sub-module 1042 is configured to solve the rotation matrix and translation vector of the first real three-dimensional coordinates based on the first real three-dimensional coordinates and the imaging plane parameters of the monocular camera.

The third solving sub-module 1043 is configured to solve the second real three-dimensional coordinates of the two-dimensional joint points on the imaging plane in the preset three-dimensional coordinate system based on the rotation matrix and the translation vector.

The joint point reconstruction sub-module 1044 is used to set the three-dimensional coordinate point with the minimum distance between the optical center coordinates of all monocular cameras and the corresponding second real three-dimensional coordinate line in the preset three-dimensional coordinate system, as the two-dimensional joint point in the preset three-dimensional coordinate point. Set the real 3D joint points in the 3D coordinate system to reconstruct the 3D action of the target body.

Specifically, the joint sub-module 1044 is specifically used for:

Based on the least squares method of overdetermined equations, solve the straight line formed by the optical center coordinates of all monocular cameras and the corresponding second real three-dimensional coordinates to obtain the three-dimensional coordinate point with the minimum distance, as the two-dimensional joint point in the preset three-dimensional coordinate system. Real 3D joint points to reconstruct the 3D action of the target body.

It should be noted that FIG. 7 is a three-dimensional space theoretical layout diagram of three monocular cameras (A, B, and C in the figure) provided in the embodiment of the application. To simplify the description, only A, The two monocular cameras B are used as the research objects. Through the calibration of the monocular cameras, the focal length f (in centimeters) of the two monocular cameras A and B, and the two-dimensional image coordinates of the two focal points (I, N) can be known ( In pixels), the coordinates of the focal points I and N of the two monocular cameras A and B are I(u _i , v _i ) and N(u _n , v _n ), respectively.

Taking the monocular camera A as an example, the real three-dimensional coordinates of the solution focus I are (x _i , y _i , z _i ), and D is the intersection of the extension line from point A to focus I and the extension line from point B to focus N, then The distance from point A to point D can be expressed as:

A vector of points A and D vector with points A and I The relationship is:

The above formula can be transformed into:

From this, the real three-dimensional coordinates (x _i , y _i , z _i ) of the point I can be obtained, and the real three-dimensional coordinates of the imaging points E' and F' of the other two marker points on the A imaging plane EFHG can be obtained correspondingly according to the above relationship. Coordinates (x _e ', y _e ', z _e ') and (x _f ', y _f ', z _f '), in the same way, the real three-dimensional coordinates of the focus N in the monocular camera B can be obtained (x _n ,y _n ,z _n ).

To solve the rotation matrix a and translation vector t of the monocular camera, according to the coordinates of point I, point A, point E' and point F' determined above, the rotation matrix a and translation vector T can be obtained, which can be expressed as:

The rotation matrix a and the translation vector t are respectively:

Solve the real three-dimensional coordinates of the imaging points W and V on the A and B imaging surfaces:

Substitute the coordinates (u _w , v _w , f) of the monocular camera coordinate system of the point W, and the coordinates (x _w , y _w , z _w ) in the real three-dimensional coordinate system of the point W can be obtained by the following equation:

Similarly, for the B monocular camera, the corresponding transformation matrix B can also be obtained, and the real three-dimensional coordinates (x _v , y _v , z _v ) of the point V can be obtained. After obtaining the respective rotation matrix a and translation vector t of the two monocular cameras, the pixel coordinates of the monocular camera coordinates of the imaging point of any point in the space on the corresponding imaging plane can be converted into real three-dimensional coordinates.

Taking the two cameras A and B as the research object, the straight lines AT and BT intersect at a point T in the space, and the real three-dimensional coordinates (x, y, z) of any point T in the space can be solved by the following equation:

It is known that two monocular cameras can obtain the three-dimensional coordinates of a point in space. However, due to the production problem of the monocular camera, it cannot be guaranteed that BT and AT intersect at point T in the process. In order to reduce the error, the method of adding a monocular camera can be taken to improve the accuracy of the solution. There are many disjoint straight lines in the space, and the least squares method of the overdetermined equation system can be used to solve the least squares solution between the straight lines. The real 3D joint points of the 3D joint points in the preset 3D coordinate system are used to reconstruct the 3D movements of the target human body.

For ease of understanding, please refer to FIG. 3 , an embodiment of a three-dimensional human motion reconstruction method is also provided in this application, including:

Step S1 , perform monocular camera calibration based on the collected calibration point data, and the number of monocular cameras is at least two.

Step S2, acquiring a sequence of motion images of the target human body captured by at least two monocular cameras.

Step S3: Identify the key limb joints of the human body in the action image sequence, and extract the two-dimensional joint points belonging to the same target human body in each frame of the action image sequence, so as to reconstruct the human skeleton of the target human body and obtain the human skeleton of each frame. information.

Step S4, based on the human skeleton information in each frame, restore the real position of the target human body at the joint point in the three-dimensional space, and reconstruct the three-dimensional action of the target human body.

As a further improvement, please refer to Figure 4. Step S4 specifically includes the following steps:

Step S41 , based on the monocular camera parameters calibrated by the monocular camera, obtain the first real three-dimensional coordinates in the preset three-dimensional coordinate system of the projected imaging point perpendicular to the focus of the monocular camera on the imaging plane of the monocular camera.

Step S42 , based on the first real three-dimensional coordinates and the imaging plane parameters of the monocular camera, solve the rotation matrix and translation vector of the first real three-dimensional coordinates.

Step S43 , solving the second real three-dimensional coordinates of the two-dimensional joint points on the imaging plane in the preset three-dimensional coordinate system based on the rotation matrix and the translation vector.

Step S44, in the preset three-dimensional coordinate system, the optical center coordinates of all monocular cameras and the corresponding second real three-dimensional coordinate line obtain the three-dimensional coordinate point of the minimum distance, as the two-dimensional joint point in the preset three-dimensional coordinate system. Real 3D joint points to reconstruct the 3D action of the target body.

For ease of understanding, please refer to FIG. 3 , an embodiment of an action training system is provided in this application, including the camera calibration module 101 , the action acquisition module 102 , the two-dimensional human action recognition module 103 , the three-dimensional action reconstruction in the foregoing embodiment module 104 and imaging point error correction module 105, and also includes an action evaluation module 106;

The action evaluation module 106 is configured to compare the difference between the three-dimensional action of the target human body obtained from the three-dimensional action reconstruction module and the preset standard action, and output an action evaluation result corresponding to the difference comparison result.

Specifically, the action evaluation module 106 is used for:

The three-dimensional motion of the target body obtained from the three-dimensional motion reconstruction module and the preset standard motion are used to judge the similarity of the joint point combination angle, the average curvature of the joint point combined motion trajectory and the human body joint point combined motion amount comparison;

Obtain the first action evaluation result corresponding to the result of the joint point combination angle similarity judgment, the second action evaluation result corresponding to the result of the average curvature comparison of the joint point combination motion trajectory and the third action corresponding to the human body joint point combination motion amount comparison result evaluation result;

Perform weighting processing on the first action evaluation result, the second action evaluation result and the third action evaluation result, and output the action evaluation result obtained after the weighting process.

It should be noted that after obtaining the three-dimensional dance movements of the user's human body, the differences between the movements and the standard movements are evaluated. The basis for the difference evaluation is divided into the following parts:

(1) Judgment of the similarity of the joint point combination angle. The evaluation method of angle similarity analysis can derive the cosine of the corresponding joint angle by using the Euclidean dot product formula (A and B are two vectors composed of three joint points, that is, three joint points are combined into a group of joint point combinations). Then convert it into an angle θ, which is convenient for users to see the difference more intuitively

And the average difference between a set of joint points combined in a set of continuous actions It can be expressed as:

Compared with the standard motion data, generally the corresponding joint angle can be regarded as good when it is less than 10°, and it can be regarded as normal when it is greater than 10° and less than 20°.

(2) The average curvature comparison of the combined motion trajectories of joint points. Human movements are dynamically changing movements, and the overall degree of movement of the joints will affect the aesthetics of the dance. The similarity of the motion trajectories is judged by calculating the average curvature of the combined motion trajectories of the joint points of each part of the human body and comparing it with the average curvature of the combined motion trajectories of the standard action joint points. First, fit the motion trajectory of the joint point into a curve, define the spatial distance of the joint point movement between the two frames as Δs, and the corresponding rotated angle as Δα, Represents the average curvature of the motion of each joint point, and the joint point combined average curvature It can be expressed as

by judgment The size of the two can compare the similarity of the combined motion trajectory of the corresponding joint points of the two.

(3) Comparison of combined motion of human joint points. The target human movement (such as dance movement) is the movement of the whole human body. In addition to the evaluation of the single joint situation, it is necessary to judge the swing range of the whole human body. Since each human body is different, in order to unify the standard, first normalize each node of the human body with the distance between the two shoulders of the human body, and then add the normalized forward and backward movement of the joints to obtain the human body at a certain moment. The amount of human exercise and then compared with the standard action.

①Because there are different degrees of differences in the human body, it is necessary to use the three-dimensional space coordinate distance of the human body's shoulders to normalize the coordinates of each joint point of the human body for data processing

x*=(xx _min )/(x _max -x _min )

y*=(yy _min )/(y _max -y _min )

z*=(zz _min )/(z _max -z _min )

② After normalization, the sum of the normalized relative coordinate displacement of each joint point before and after a certain corresponding time interval calculation interval is Wp, and the total displacement W of the joint point combination composed of joint points is used as the joint point. Combined amount of exercise

③ Compare the amount of human body movement at a certain moment with the standard action, and judge the difference between the two by ΔW.

The values obtained from the three parts are weighted, but at the same time the quality level determined by each body part depends on the difficulty of the exercise. To set the percentage range suitable for the quality level results, get the final dance movement score, and provide the above three parts of data to the user through the central processing unit and visualized on the display screen.

As a further improvement, the action training system in the embodiment of the present application further includes: a music rhythm fit module 107;

The music rhythm fit module 107 is used for extracting the music features of the music being broadcast based on the audio extraction algorithm, and using the beat as a unit, to determine the matching degree of the human body movements of the target human body in continuous frames and the preset standard movements in the series of movements matching the rhythm, Output beat matching results.

It should be noted that an audio extraction algorithm is used to extract the musical features of the dance music, and the beat is used as the unit to determine whether the user's actions in consecutive frames are consistent with the series of actions of the standard action in the beat, so as to determine whether it is in tune. In the event of a snap shot or a slow shot, the system records the first half of the in-beat score and then resets the beat alignment to match the actual motion of the user and provide the motion evaluation module 106 with accurate motion data comparison. Through the combination of the action evaluation module 106 and the music rhythm fit module 107, each beat score and training report are generated and visually displayed to the user.

A data storage module 109 can also be set. In the data storage module 109, the user can choose to save the score and training report and training recording and comparison video data to the hard disk, or partially upload it to the cloud platform through the network for the recommendation algorithm to match more suitable users. Individual dance styles and training patterns. At the same time, the data storage module 109 also stores camera parameters, which are used for three-dimensional motion reconstruction of the human body.

As a further improvement, the action training system in the embodiment of the present application further includes: a health guard module 108;

The health guard module 108 is used to calculate the user's physical fitness level, suitable dance style and current training exercise volume through the cloud platform algorithm according to the user information of the target human body, and recommend the most suitable action set for the user according to the calculation result with high quality, so as to avoid the user The training intensity is too weak or too high, and the user's training time and training intensity are recorded at the same time, and the user is prompted to take a break through voice broadcast and/or display of a pop-up window.

It should be noted that a mode switching module 110 can also be set: the module provides mode selection for dance training. Three functions can be selected under this module, namely learning module, scoring mode and special training mode. Among them, the learning module and scoring mode can be performed by multiple people at the same time, and the special training mode is single-player mode by default.

1) In the learning mode, you can choose to download the standard dance video pre-recorded by professional dancers from the cloud platform for imitation learning, and the standard video will provide a positive version and a mirror inversion version for users to choose to display on the monitor, and the system will Differences in standard movements or music rhythms, divide learning steps and identify difficult and difficult movements for users' attention to strengthen learning.

2) The scoring mode will call the dance evaluation module, including evaluating the fit between the user's dance movements and the music rhythm, and visually displaying it to the user through text special effects.

3) In the special training module, the cloud platform will judge the user's dance defects according to the user's series of course learning scores and store data, and provide corresponding local special training according to the defects.

The motion training system provided in the embodiment of the present application is based on the principle of binocular stereo vision. The monocular camera does not need to add other expensive sensors, and only needs to be calibrated to achieve high accuracy, which is basically close to the recognition accuracy of the Kinect depth camera. The implementation cost is low, and there is no need to worry about camera shake and complex application scenarios. Users can dynamically change the position of the camera according to their needs to obtain a better human observation angle. The binocular imaging based on parallax triangulation can be used even in a cluttered room. It will not affect its accuracy and has strong adaptability. At the same time, dance learners do not need to wear any sensing equipment or special clothing for collecting human movements, which is more convenient for dance learners to stretch their dance postures and reduce weight. At the same time, it can also be expanded to adapt to the recognition of multi-person dance movements.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: The technical solutions described in the embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (10)

1. a human action reconstruction system based on multi-eye vision, is characterized in that, comprises: a camera calibration module, configured to perform monocular camera calibration according to the collected calibration point data, and the number of the monocular cameras is at least two; a motion acquisition module, configured to acquire all motion image sequences of the target human body collected by the monocular camera, and send the motion image sequences to the two-dimensional human motion recognition module; The two-dimensional human action recognition module is used to identify the key limb joint parts of the human body in the action image sequence, and extract the two-dimensional joint points belonging to the same target human body in each frame of the action image sequence, so as to facilitate the reconstruction of all body parts. The human skeleton of the target human body is stored, and the human skeleton information of each frame is stored and sent to the three-dimensional motion reconstruction module; The three-dimensional motion reconstruction module is configured to restore the real positions of the two-dimensional joint points of the target human body in the three-dimensional space based on the human skeleton information in each frame, and reconstruct the three-dimensional motion of the target human body. 2. The multi-vision based human motion reconstruction system according to claim 1, further comprising: an imaging point error correction module; The imaging point error correction module is used to correct the pixel coordinates of the two-dimensional joint points in the two-dimensional human motion recognition module when the monocular camera is offset, so as to facilitate the two-dimensional human motion The recognition module reconstructs the human skeleton of the target human body according to the two-dimensional joint points after correcting the pixel coordinates, and stores and sends the human skeleton information of each frame to the three-dimensional motion reconstruction module. 3. The multi-vision based human motion reconstruction system according to claim 1, wherein the three-dimensional motion reconstruction module specifically comprises: The first solving sub-module is used to solve, based on the monocular camera parameters calibrated by the monocular camera, the projection imaging point perpendicular to the focus of the monocular camera projected on the imaging plane of the monocular camera at the preset value. the first real three-dimensional coordinate in the three-dimensional coordinate system; a second solving sub-module, configured to solve the rotation matrix and translation vector of the first real three-dimensional coordinates based on the first real three-dimensional coordinates and the imaging plane parameters of the monocular camera; a third solving submodule, configured to solve the second real three-dimensional coordinates of the two-dimensional joint points on the imaging plane in the preset three-dimensional coordinate system based on the rotation matrix and the translation vector; The joint point reconstruction sub-module is used to set the three-dimensional coordinate point at which the minimum distance is obtained between the optical center coordinates of all the monocular cameras and the corresponding second real three-dimensional coordinate line in the preset three-dimensional coordinate system, as the The real three-dimensional joint points of the two-dimensional joint points in the preset three-dimensional coordinate system are used to reconstruct the three-dimensional motion of the target human body. 4. The human motion reconstruction system based on multi-eye vision according to claim 3, wherein the joint sub-module is specifically used for: Based on the least squares method of overdetermined equations, solve the straight line formed by the optical center coordinates of all the monocular cameras and the corresponding second real three-dimensional coordinates to obtain the three-dimensional coordinate point with the minimum distance, as the two-dimensional joint point in the The real three-dimensional joint points in the preset three-dimensional coordinate system are used to reconstruct the three-dimensional motion of the target human body. 5. The human motion reconstruction system based on multi-eye vision according to claim 1, wherein the two-dimensional human motion recognition module is specifically used for: Based on the preset convolutional neural network, the key limb joint parts of the human body in the action image sequence are identified, and 25 joint points belonging to the same target human body in each frame of the action image sequence are extracted to facilitate the reconstruction of the target. The human body skeleton of the human body, the information of the human body skeleton of each frame is stored and sent to the three-dimensional motion reconstruction module. 6. A human action reconstruction method based on multi-eye vision, is characterized in that, comprises: Perform monocular camera calibration based on the collected calibration point data, and the number of the monocular cameras is at least two; Acquiring at least two action image sequences of the target human body captured by the monocular camera; Identify the key limb joint parts of the human body in the action image sequence, extract the two-dimensional joint points belonging to the same target human body in each frame of the action image sequence, so as to reconstruct the human skeleton of the target human body, and obtain each frame of the human skeleton information; Based on the human skeleton information in each frame, the real position of the target human body in the joint point in the three-dimensional space is restored, and the three-dimensional action of the target human body is reconstructed. 7 . The method for reconstructing human motions based on multi-eye vision according to claim 6 , wherein the restoration of the reality of the target human body at joint points in three-dimensional space based on the human skeleton information in each frame. 8 . position to reconstruct the three-dimensional motion of the target human body, including: Based on the monocular camera parameters calibrated by the monocular camera, the first real value of the projected imaging point perpendicular to the focal point of the monocular camera on the imaging plane of the monocular camera in the preset three-dimensional coordinate system is obtained. three-dimensional coordinates; Based on the first real three-dimensional coordinates and the imaging plane parameters of the monocular camera, solve the rotation matrix and translation vector of the first real three-dimensional coordinates; Solving the second real three-dimensional coordinates of the two-dimensional joint points on the imaging plane in the preset three-dimensional coordinate system based on the rotation matrix and the translation vector; In the preset three-dimensional coordinate system, the three-dimensional coordinate points that make the optical center coordinates of all the monocular cameras and the corresponding second real three-dimensional coordinate lines obtain the minimum distance are taken as the two-dimensional joint points in the The real three-dimensional joint points in the three-dimensional coordinate system are preset to reconstruct the three-dimensional motion of the target human body. 8. An action training system, characterized in that, comprising the multi-vision based human action reconstruction system according to any one of claims 1-5, and also comprising an action evaluation module; The motion evaluation module is configured to perform a difference comparison between the three-dimensional motion of the target human body obtained from the three-dimensional motion reconstruction module and a preset standard motion, and output a motion evaluation result corresponding to the difference comparison result. 9. The motion training system according to claim 8, wherein the motion evaluation module is specifically used for: The three-dimensional motion of the target human body obtained from the three-dimensional motion reconstruction module and the preset standard motion are judged on the similarity of the joint point combination angle, the average curvature comparison of the joint point combined motion trajectory, and the human body joint point combined motion amount comparison; Obtain the first action evaluation result corresponding to the result of the joint point combination angle similarity judgment, the second action evaluation result corresponding to the result of the average curvature comparison of the joint point combination motion trajectory, and the human body joint point combination exercise amount comparison. The third action evaluation result corresponding to the result; Perform weighting processing on the first action evaluation result, the second action evaluation result and the third action evaluation result, and output the action evaluation result obtained after the weighting process. 10. The motion training system according to claim 9, further comprising: a music rhythm fit module; The music rhythm fit module is used for extracting the music features of the music being broadcast based on the audio extraction algorithm, and in units of beats, it is judged that the human body movements of the target human body in continuous frames and the preset standard movements are in the series of movements that match the beats. Matching degree, output beat matching result.