JP7067709B1 - Programs, devices and methods for statistically analyzing skeleton-based body length from skin models - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program for statistically analyzing a skin model in which a plurality of joint points are embedded in a polygon mesh composed of a plurality of vertices. SOLUTION: The program comprises a bone length vector extraction means for constructing a skeleton by connecting joint points with bones by a parent-child relationship and extracting all bone lengths as a bone length vector, and a teacher data group as a training stage. The computer functions as a statistical learning engine that constructs a statistical learning model so as to input a bone length vector in a plurality of skin models and output component variables. The bone length vector extraction means extracts as a bone length vector consisting of all bones, with the absolute difference | b | between the position of the child joint point and the position of the parent joint point as the bone length. [Selection diagram] Fig. 2

Description

The present invention relates to a technique for statistically analyzing a skin model. Especially suitable as a skin model for a three-dimensional human body model.

There is a three-dimensional body scanner technique for acquiring human body shape data (see, for example, Non-Patent Document 1). According to this technique, a three-dimensional human body model is measured by non-contact optical surveying. This human body model is expressed as a "skin" composed of polygons consisting of a plurality of vertices in a mesh shape. The number of vertices can be measured at an ultra-high density of, for example, about 1.5 million points.

Conventionally, there is a technique of easily generating a three-dimensional model from only measured values and reducing the data capacity by dimensional compression (see, for example, Patent Document 1). According to this technique, using a teacher data group in which measurement values of the number of dimensions n corresponding to a plurality of measurement points are associated with each three-dimensional model, from the measurement values of the number of dimensions n of one body, 3 A dimensional model can be generated. Specifically, a statistical learning engine that outputs a dimensionally compressed component variable of the dimension number m from a plurality of three-dimensional models of the teacher data group, and constructs a statistical learning model, and a measurement value of the dimension number n. It has a correlation learning engine for constructing a correlation learning model with a component variable having a number of dimensions m. Further, the encoding unit that generates the component variable of the dimension number m from the measured value of one dimension number n as the target data using the correlation learning engine, and the component variable of the dimension number m using the statistical learning engine. It has a decoding unit that generates a three-dimensional model from.

Japanese Patent No. 6424309

"Smart & try", Wacoal Co., Ltd., [online], [Search on November 20, 3rd year of Reiwa], Internet <URL: https://www.wacoal.jp/smart_try/> Tomohiko Mukai, Katsuaki Kawachi, Yoichiro Miyake, "Mathematical and System of Character Animation, Physical Movement Generation and Artificial Intelligence in -Dimensional Games-", [online], [Search on November 20, 3rd year of Reiwa], Internet < URL: https://www.coronasha.co.jp/np/isbn/9784339029093/>

According to the technique described in Patent Document 1, in the three-dimensional model, each vertex is represented by three dimensions (x, y, z) of the world coordinate system (Global coordinate system). For example, when the number of vertices for each body is N = 1.5 million, one three-dimensional model is represented by a vector of 3N (= 4.5 million) dimensions. Then, for example, about 6000 three-dimensional models can be input to the statistical learning engine to obtain component variables. The statistical learning engine is specifically based on Principal Component Analysis.

In order to acquire a three-dimensional human body model, a person must actually enter the three-dimensional body scanner and take a picture, but at this time, the posture varies from person to person. In the case of the world coordinate system, even if the person is the same person, different three-dimensional models will be acquired under the influence of the posture of the person.
The posture depends on the position of the joint point. For example, even if the person is exactly the same, the case where both legs are opened and the case where both legs are closed will be acquired as a human body model that is significantly different especially in the foot portion. In addition, when measuring with a three-dimensional scanner, it is very difficult to force everyone who is a living person to have the same posture.

On the other hand, when the human body is measured by a three-dimensional scanner, the inventors of the present application have "body shape" based on the flesh of the human body, "body length" based on the length of the human body, and "body length" based on the position of the joints of the human body. It was noted that the data was a mixture of "posture". Here, I wondered if it would be possible to extract only the "body length" from which the "flesh" and "posture" were separated from the 3D model of the human body measured by the 3D scanner.

Here, the "body length" is assumed to be based on the skeletal length of the human body. Originally, the skeletal length of such a human body is not affected by "posture" or "flesh".
On the other hand, the "posture" changes depending on the position of the skeletal joint point in the person. Even among people with the same body shape, their appearance impressions differ greatly depending on changes in posture such as leaning forward, leaning backward, stooping, and rolled shoulders. In addition, "flesh" also changes every day even if it is the same person.

On the other hand, the inventors of the present application wondered if it would be possible to statistically analyze multiple 3D models from only the data of "body length based on the skeleton" with "flesh" and "posture" separated. .. In particular, I thought that it would be an element for analyzing the essential appearance characteristics of human beings.

Therefore, it is an object of the present invention to provide a program, an apparatus and a method for generating a statistical shape space capable of analyzing a body length based on a skeleton from a skin model.

According to the present invention, it is a program for operating a computer mounted on a device for statistically analyzing a skin model in which a plurality of joint points are embedded in a polygon mesh composed of a plurality of vertices.
A bone length vector extraction means that constructs a skeleton by connecting joint points with bones according to a parent-child relationship and extracts all bone lengths as bone length vectors.
As a training stage, it is characterized in that the computer functions as a statistical learning engine that constructs a statistical learning model so as to input a bone length vector in a plurality of skin models that are teacher data groups and output component variables.

According to other embodiments in the program of the invention
It is also preferable that the bone length vector extraction means has a computer function so that the absolute difference | b | between the position of the child joint point and the position of the parent joint point is defined as the bone length and the bone length vector is extracted as a bone length vector consisting of all bones. ..

According to other embodiments in the program of the invention
It is also preferable to further function a computer as a means for estimating the joint point from one polygon mesh to estimate the position of the joint point.

According to other embodiments in the program of the invention
The means for estimating the joint point is the joint point learning engine.
As a training stage, input each of multiple polygon meshes of the teacher data group, construct a joint point learning model so that the position of the joint point is output, and build a joint point learning model.
As an estimation step, it is also preferable to input a polygon mesh to be estimated and make a computer function to output the position of a joint point.

According to other embodiments in the program of the invention
As the first estimation step,
It is also preferable that the statistical learning engine operates the computer to input the bone length vector in one polygon mesh to be estimated and output the component variable.

According to other embodiments in the program of the invention
As a second estimation step,
It is also preferable that the statistical learning engine operates a computer to input a component variable to be estimated and output a bone length vector for each joint point in one skin model.

According to other embodiments in the program of the invention
The skin model is a human body model,
The skeleton is the skeleton of the human body,
It is also preferable to make the computer function as if the bone is a bone between the joint points.

According to other embodiments in the program of the invention
It is also preferable that the statistical learning engine makes the computer function as if it were based on Principal Component Analysis or AutoEncoder.

According to the present invention, it is an analyzer that statistically analyzes a skin model in which a plurality of joint points are embedded in a polygon mesh composed of a plurality of vertices.
A bone length vector extraction means that constructs a skeleton by connecting joint points with bones according to a parent-child relationship and extracts all bone lengths as bone length vectors.
As a training stage, it is characterized by having a statistical learning engine that inputs a bone length vector in a plurality of skin models that are teacher data groups and constructs a statistical learning model so as to output component variables.

According to the present invention, it is an analysis method of an apparatus for statistically analyzing a skin model in which a plurality of joint points are embedded in a polygon mesh composed of a plurality of vertices.
The device is
The first step of constructing a skeleton by connecting joint points with bones according to a parent-child relationship and extracting all bone lengths as bone length vectors,
As a training step, a second step of inputting a bone length vector in a plurality of skin models to be a teacher data group and constructing a statistical learning model so as to output a component variable is performed.

According to the program, apparatus and method of the present invention, it is possible to generate a statistical shape space from which the body length based on the skeleton can be analyzed from the skin model.

It is explanatory drawing which shows the relationship between the joint point and the apex. It is a basic functional block diagram of the analyzer in this invention. It is explanatory drawing which shows the bone connecting the joint points in this invention. It is explanatory drawing which shows the vector space of a 3D model in a statistical learning engine. It is explanatory drawing which shows the statistical shape space in a statistical learning engine. It is a simple code that represents the principal component analysis in the statistical learning engine. It is a functional block diagram of the analyzer further having a joint point estimation part and a skeleton deformation part.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is an explanatory diagram showing the relationship between a joint point and a vertex.

According to the embodiments of the present invention, the terms are defined as follows.
"Skin": Polygon mesh consisting of multiple vertices
(For example, a polygon mesh of the human body taken with a 3D scanner)
"Skin model": A model in which multiple joint points are embedded in the skin "Joint points": Joints that deform the skin (for example, joints of the human body)
"Bone": A line segment connecting the joint points (for example, the bone of the human body)
"Skeleton": A structure in which joint points are connected by bones by a parent-child relationship.
(For example, the skeleton of the human body)
"Skin weight": The degree of influence of the joint points on each vertex of the skin.

According to FIG. 1, the human body is represented by, for example, a polygon mesh having 1.5 million vertices. Further, the number of joint points is, for example, the following 15 points.
"Cervical spine"
"Central chest"
"Left shoulder""Leftelbow""Leftwrist"
"Rightshoulder""Rightelbow""Rightwrist"
"Centralwaist"
"Lefthip""Leftknee""Leftankle"
"Righthip""Rightknee""Rightankle"

FIG. 2 is a basic functional configuration diagram of the analyzer according to the present invention.

The analyzer 1 statistically analyzes a skin model in which a plurality of joint points are embedded in a polygon mesh composed of a plurality of vertices. Generate a statistical shape space that enables analysis of "body length based on the skeleton" from the skin model.
According to FIG. 2, the analyzer 1 has a bone length vector extraction unit 11 and a statistical learning engine 12. These functional components are realized by executing a program that makes a computer mounted on the analyzer function. In addition, the processing flow of these functional components can be understood as an analysis method.

The analyzer 1 prepares a large number of skin models as teacher data for statistical learning. As mentioned above, the skin model is a skin in which a plurality of joint points are embedded. The joint points are connected by a bone in a parent-child relationship. When the skin model is a human body, the polygon mesh of the human body contains a skeleton connecting the joint points. That is, a skin model of the human body acquired from an unspecified number of people is prepared.
It should be noted that the joint points and skeletons assumed by the present invention are not anatomical ones of the human body, but merely the basic axes of movement (motion).

[Bone length vector extraction unit 11]
The bone length vector extraction unit 11 constitutes a skeleton by connecting joint points with bones according to a parent-child relationship, and extracts all bone lengths as bone length vectors.

FIG. 3 is an explanatory diagram showing a bone connecting joint points in the present invention.

<Position vector for each joint point>
The joint point is expressed with reference to the three-dimensional world coordinate system. In the world coordinate system, the coordinate system is basically such that the positive direction of the y-axis is the vertical upward direction and the xz plane is the horizontal plane. Then, the position vector for each joint point is expressed as follows.
Position vector of joint point: t = (tx ty tz)

<Bone length vector for each joint point>
The human skeleton is represented as a tree structure connecting joint points.
The bone length vector extraction unit 11 extracts the absolute difference | b | between the position of the child joint point and the position of the parent joint point as the bone length vector r consisting of all bones.
The bone length vector r is represented by the following 14-dimensional vector, for example, according to FIG. 1 described above.
{
Bone length of "cervical spine-central chest" Bone length of "central chest-left shoulder" Bone length of "left shoulder-left elbow" Bone length of "left elbow-left wrist" Bone length of "central chest-right shoulder" Bone length "right shoulder" -Bone length of "right elbow" Bone length of "right elbow-right wrist" Bone length of "central chest-central waist" Bone length of "central waist-left waist" Bone length of "left waist-left knee" Bone length "left knee" -Bone length of "left ankle" Bone length of "center waist-right waist" Bone length of "right waist-right knee" Bone length of "right knee-right ankle"}

The bone length vector r is an essential cosmetic feature that identifies the person and is an immutable element. That is, even if a person's "flesh" and "posture" change, the "skeleton-based body length" does not change.

[Statistical learning engine 12]
The statistical learning engine 12 functions as a training stage as follows.
<Training stage>
As a training step, the statistical learning engine 12 inputs a bone length vector in a plurality of skin models serving as a teacher data group, and constructs a statistical learning model so as to output a component variable.

FIG. 4 is an explanatory diagram showing the vector space of the three-dimensional model in the statistical learning engine.

According to FIG. 4, as the teacher data, for example, 6000 human bodies having various body shapes are assumed.
According to FIG. 4A, one three-dimensional model is represented by a D (= 14) -dimensional vector which is a bone length vector.
According to FIG. 4B, each human body of the three-dimensional model is represented by one point in the D-dimensional space.

FIG. 5 is an explanatory diagram showing a statistical shape space in a statistical learning engine.
FIG. 6 is a simple code representing the principal component analysis in the statistical learning engine.

Specifically, the statistical learning engine 12 may be based on Principal Component Analysis.
By "principal component analysis", the principal components (component variables) of the dimension (for example, 14 dimensions) that are uncorrelated with each other and well represent the overall variation are derived from 6000 points in the correlated D-dimensional space. The variance of the first principal component is maximized, and the subsequent principal components are selected so as to maximize the variance under the constraint of no correlation with the previously determined principal components. By maximizing the variance of the principal components, the principal components have the ability to explain changes in observed values as much as possible. The main axis that gives the principal component is the orthogonal basis of the group of 6000 points in the D-th order space. The orthogonality of the main axis is derived from the fact that the main axis is the eigenvector of the covariance matrix and the covariance matrix is a real symmetric matrix.

The statistical learning engine 12 constructs a statistical learning model that projects a component variable based on principal component analysis onto a statistical shape space (for example, 14 dimensions) whose number of dimensions is a D-dimensional space.
According to the present invention, each three-dimensional model in the D (= 14) dimensional space is projected onto, for example, a 14-dimensional (component variable) space. The transformation that gives the principal component is represented by the singular value decomposition of a matrix consisting of a set of observed values, and the singular value decomposition of a rectangular matrix X consisting of a group of 6000 points in the D-dimensional space is expressed by the following equation.
X = U * Σ * V ^T
X: Matrix consisting of 6000 points in D-dimensional space (6000 rows x D columns)
U: Square matrix of n (6000) × n (6000) (Orthogonal matrix of n-dimensional unit vector)
Σ: N (6000) × p (D) rectangular diagonal matrix (diagonal component is a singular value of X)
V: square matrix of p (D) × p (D) (orthogonal matrix of p-dimensional unit vector)
Here, the linear transformation by the matrix V gives the principal component of X.
V: D-dimensional space-> Matrix representing transformation to statistical shape (14-dimensional) space
V ^-1 : Statistical shape (14-dimensional) Matrix-> Matrix representing transformation to D-dimensional space Note that the superscript -1 of the matrix is not a symbol indicating the inverse matrix, but for the transformation of the vector defined by the matrix. , Is used as an abstract symbol meaning its inverse transformation. Here, V ^-1 is equal to the transposed V ^T of V.

As is clear from FIG. 6, it is possible to associate the D-dimensional space with the statistical shape space for each human body of the three-dimensional model by the linear transformation by the matrix V or V ^-1 .
s = x * V
x = s * V ^-1
s: Vector in statistical shape space
x: Vector in D-dimensional space
V: Statistical learning model The origin in the statistical shape space is the average of the bone lengths of all human body models trained as teacher data. That is, the farther away from the average origin, the more characteristic the bone length at that point is.

Further, the statistical learning engine 12 may be based on an autoencoder.
An autoencoder is a type of neural network that outputs an abstract concept (feature amount) from the amount of information in the input data. Specifically, the input data is dimensionally compressed by a hidden layer and returned to its original size at the time of output.
Similar to principal component analysis, the autoencoder also derives 14-dimensional component variables that are uncorrelated with each other and best represent the overall variation from 6000 points in the correlated D-dimensional space.

According to FIG. 2 described above, the statistical learning engine 12 functions as two estimation stages as follows.
<Estimation stage as an encoder>
The statistical learning engine 12 inputs bone length vectors of all joint points in one polygon mesh to be estimated, and outputs component variables.
The component variable is, for example, a 14-dimensional vector and is the main component of the "posture (rotation of all joint points)" of the skin model (polygon mesh + skeleton) to be estimated.
<Estimation stage as a decoder>
The statistical learning engine 12 inputs component variables to be estimated and outputs "all bone lengths (= bone length vectors)" in one skin model.
For example, by inputting the component variable of the 14-dimensional vector to be estimated, all the bone length vectors of one body can be output. All bone length vectors represent the invariant visual features of the human body.

Here, the features of the present invention will be described in detail again.
According to the prior art, in skinning, a posture or movement is generally expressed as a rotation of a human body part centered on a joint point. The skinning process is also called "Skeleton Subspace Deformation" and describes the deformation by the coordinate system of each joint point in the skeleton. By displacing the skeleton, the polygon mesh is deformed. Further, according to the general use of animation, the setting for creating the skeleton is manually operated by the operator.
On the other hand, in the present invention, only the bone length vector for each joint point in the skeleton is extracted and statistically analyzed separately from the deformation of the 3D polygon mesh. Such a technique does not exist as an existing technique. According to the present invention, the decomposition of the bone length vector of the joint point is that the bone length vector as the intermediate data is extracted and trained by the statistical learning engine without calculating the skinning algorithm to the end.

<Posture judgment application>
For example, in the case of clothes, the design is based on the size of the person (length of the skeleton) and the circumference, but the posture of the person (rotation of the joint point) that suits the clothes is not taken into consideration. On the other hand, the user who purchases clothes also considers his / her size and body shape, but does not consider his / her posture at all. It does not consider how far the user's own posture is from the average or ideal posture.
Although various uses are expected as a posture determination application, it is possible to numerically acquire the difference in posture between different human bodies. The difference in posture can be recognized as a distance between points in a statistical shape space based on principal component analysis, for example.

FIG. 7 is a functional configuration diagram of an analyzer further including a joint point estimation unit and a skeleton deformation unit.

According to FIG. 7, as compared with FIG. 2, it further has a joint point estimation unit 13 and a skeleton deformation unit 14. These functional components are also realized by executing a program that makes the computer mounted on the analyzer function. In addition, the processing flow of these functional components can be understood as an analysis method.

When the human body is photographed by an optical scanner, the three-dimensional human body model is a skin (polygon mesh). That is, the skin does not include a skeleton (skeleton) like the skin model. Therefore, according to FIG. 7, in the previous stage of the bone length vector extraction unit 11, the joint points are extracted from the skin, and the skeleton in which the joint points are connected by the bone by the parent-child relationship is estimated.

[Portrait estimation unit 13]
The joint point estimation unit 13 estimates the position of the joint point from one polygon mesh.
The joint point estimation unit 13 may be a joint point learning engine and functions as follows.
(Training stage) A polygon mesh (explanatory variable) of multiple objects of the teacher data group is input, and a joint point learning model is constructed so as to output the position of the joint point (objective variable).
(Estimation stage) Input the polygon mesh (explanatory variable) to be estimated, and output the position of the joint point (objective variable).

As the joint point learning engine, a regression model that predicts the joint point of motion capture from the vertex position of the polygon mesh can also be used for estimating the joint point. For the training of the regression model, it is assumed that the human body shape is almost convex around the joint. The convex shape means that the line segment connecting any two points does not go out of the polygon (internal angle is 180 ° or less).
Of course, instead of machine learning as in the regression model, the joint position may be calculated directly from the mesh by designating the mesh vertices corresponding to the mark points reflected in the frame image based on the motion capture.

Further, the joint position may be estimated from the motion capture data. For example, VICON (registered trademark) may be used to estimate the joint position from each frame of motion capture. For example, it is possible to estimate the displacement of the joint position based on the walking motion from each frame of the motion capture in which the walking motion of a large number of people is captured.

As a feature of the present invention, the posture can be analyzed only from the skin (polygon mesh) containing no skeleton. This makes it possible to use any type of 3D scanner on the market and the shooting environment. That is, even polygon meshes taken by various three-dimensional scanners can estimate joint positions and associate them with the statistical shape space as long as they are homologously modeled. In this way, by separating only the posture from the skin of the human body, it becomes possible to handle different models of the three-dimensional scanner and the shooting environment in a unified manner.

[Skeleton deformation part 14]
The skeleton deforming portion 14 reproduces one skeleton from the bone length vectors of all the joint points of one body.
Since only the bone length vector is output from the statistical learning engine 12, a skeleton template is prepared in advance. The template is made to correspond to the bone length vector (rotation axis, angle of rotation) output from the statistical learning engine 12, and the skeleton model of the template is deformed (rotation of the joint point).
In this way, the skeleton deforming portion 14 can visually recognize the posture based on the skeleton.

As described in detail above, according to the program, apparatus and method of the present invention, it is possible to generate a statistical shape space in which the body length based on the skeleton can be analyzed from the skin model.

Various changes, modifications and omissions of the technical idea and the scope of view of the present invention can be easily carried out by those skilled in the art with respect to the various embodiments of the present invention described above. The above explanation is just an example and does not attempt to limit anything. The present invention is limited only to the scope of claims and their equivalents.

1 Analytical device 11 Bone length vector extraction part 12 Statistical learning engine 13 Joint point estimation part 14 Skeleton deformation part

Claims (10)

A program that makes a computer function on a device that statistically analyzes a skin model that has multiple joint points in a polygon mesh consisting of multiple vertices.
A bone length vector extraction means that constructs a skeleton by connecting joint points with bones according to a parent-child relationship and extracts all bone lengths as bone length vectors.
As a training stage, a program characterized by operating a computer as a statistical learning engine that constructs a statistical learning model so as to input bone length vectors in multiple skin models that are teacher data groups and output component variables. The bone length vector extraction means is characterized in that the computer functions to extract as a bone length vector consisting of all bones, with the absolute difference | b | between the position of the child joint point and the position of the parent joint point as the bone length. The program according to claim 1. The program according to claim 1 or 2, wherein the computer further functions as a joint point estimation means for estimating the position of the joint point from one polygon mesh. The means for estimating the joint point is the joint point learning engine.
As a training stage, input each of multiple polygon meshes of the teacher data group, construct a joint point learning model so that the position of the joint point is output, and build a joint point learning model.
The program according to claim 3, wherein as an estimation step, a polygon mesh to be estimated is input and a computer is made to function to output the position of a joint point. As the first estimation step,
The invention according to any one of claims 1 to 4, wherein the statistical learning engine inputs a bone length vector in one polygon mesh to be estimated and causes a computer to function to output a component variable. Program. As a second estimation step,
Any of claims 1 to 4, wherein the statistical learning engine inputs a component variable to be estimated and causes a computer to function to output a bone length vector for each joint point in one skin model. The program described in Section 1. The skin model is a human body model,
The skeleton is the skeleton of the human body,
The program according to any one of claims 1 to 6, wherein the bone is a computer functioning as if it were a bone between joint points. The program according to any one of claims 1 to 7, wherein the statistical learning engine operates a computer so as to be based on Principal Component Analysis or AutoEncoder. An analyzer that statistically analyzes a skin model that has multiple joint points in a polygon mesh consisting of multiple vertices.
A bone length vector extraction means that constructs a skeleton by connecting joint points with bones according to a parent-child relationship and extracts all bone lengths as bone length vectors.
As a training stage, an analyzer characterized by having a statistical learning engine that inputs a bone length vector in a plurality of skin models serving as a teacher data group and constructs a statistical learning model so as to output component variables. It is an analysis method of a device that statistically analyzes a skin model that has multiple joint points in a polygon mesh consisting of multiple vertices.
The device is
The first step of constructing a skeleton by connecting joint points with bones according to a parent-child relationship and extracting all bone lengths as bone length vectors,
As a training step, an analysis method characterized by inputting a bone length vector in a plurality of skin models to be a teacher data group and performing a second step of constructing a statistical learning model so as to output component variables. ..

Images