KR20240121550A - Device, method and computer program for 3d pose estimation - Google Patents

Abstract

According to an embodiment of the present invention, a 3D pose estimation device includes: a receiving unit which receives an image including a person; a 2D pose estimation unit which analyzes the image to extract types and positions of joints according to the pose of the person, and generates a skeleton graph based on the types and positions of the extracted joints; and a 3D pose estimation unit which estimates a 3D pose corresponding to the skeleton graph using an adjacency matrix; wherein, in the adjacency matrix, different adjacency matrices are used corresponding to each repeated process in the process of deriving the k-th (0<k<K, where K is a natural number greater than or equal to 2) updated feature matrix.

Description

3D pose estimation device, method and computer program {DEVICE, METHOD AND COMPUTER PROGRAM FOR 3D POSE ESTIMATION}

The present invention relates to a device, method and computer program for analyzing an input image and estimating a pose of a person in the image.

Computer vision is a field of artificial intelligence that uses computers to reproduce the general visual recognition ability of humans. Among these, pose estimation recognizes the body posture or certain attitude of a person in an image, and mainly aims to find the position of the person's joints.

And 3D pose estimation (HPE) aims to derive the 3D position of human body joints within the camera coordinate system from a single image or multiple images such as videos. 3D pose estimation plays an important role in fields such as human-computer interaction (HCI), gesture recognition, and human action recognition.

However, in pose estimation, if the person in the image is partially obscured by another person or object, causing occlusion, there is a problem in that it is difficult to make an accurate estimation due to the low probability value for the joint position.

Recently, 3D pose estimation has achieved better performance than conventional methods due to the development of deep neural networks. Past 3D pose estimation methods used an end-to-end pipeline through a convolutional neural network on the image or a two-stage pipeline.

Recently, there has been a trend toward utilizing graph convolutional neural networks (GCNs), which represent the joint positions of a person in an image in the form of a skeleton graph and apply a convolutional neural network to estimate 3D poses.

However, in the case of conventional graph convolutional neural networks, there is a problem in that learning is performed mainly on the features of nodes adjacent to a random node in the process of updating the features of each node in the graph, and long-range dependencies for nodes that are relatively far away are not sufficiently modeled, which reduces the accuracy of 3D pose estimation.

Korean Patent Publication No. 10-1812379 (Registered on December 19, 2017)

The present invention is intended to solve the problems of the above-mentioned prior art, and to provide a 3D pose estimation device, method, and computer program capable of estimating not only a 2D pose of a person expressed in an image but also a 3D pose.

In addition, the present invention aims to provide a 3D pose estimation device, method, and computer program that efficiently reflect long-distance dependency in the process of estimating a 3D pose by analyzing the pose of a person expressed in an image, thereby improving the accuracy of pose estimation.

However, the technical tasks that this embodiment seeks to accomplish are not limited to the technical tasks described above, and other technical tasks may exist.

As a means for achieving the above-described technical task, one embodiment of the present invention includes a receiving unit which receives an image including a person; a 2D pose estimation unit which analyzes the image to extract types and positions of joints according to the posture of the person, and generates a skeleton graph based on the types and positions of the extracted joints; and a 3D pose estimation unit which estimates a 3D pose corresponding to the skeleton graph using an adjacency matrix; wherein the adjacency matrix can provide a 3D pose estimation device in which different adjacency matrices are used corresponding to each repeated process in a process of deriving a k-th (0<k<K, where K is a natural number greater than or equal to 2) updated feature matrix.

Another embodiment of the present invention provides a 3D pose estimation method performed by a 3D pose estimation device, comprising: a receiving step of inputting an image including a person; a 2D pose estimation step of analyzing the image to extract types and positions of joints according to the pose of the person, and generating a skeleton graph based on the types and positions of the extracted joints; and a 3D pose estimation step of estimating a 3D pose corresponding to the skeleton graph using an adjacency matrix; wherein, in the adjacency matrix, different adjacency matrices may be used corresponding to each repeated process in the process of deriving the k-th (0<k<K, where K is a natural number greater than or equal to 2) updated feature matrix.

Another embodiment of the present invention is a computer program stored on a computer-readable recording medium including a sequence of commands for estimating a 3D pose from an image, the computer program including the sequence of commands for receiving an image including a person, analyzing the image to extract types and positions of joints according to the pose of the person, estimating a 2D pose by generating a skeleton graph based on the types and positions of the extracted joints, and estimating a 3D pose corresponding to the skeleton graph using an adjacency matrix, wherein the adjacency matrix may include a sequence of commands for using different adjacency matrices corresponding to each repeated process in the process of deriving a k-th (0<k<K, where K is a natural number greater than or equal to 2) updated feature matrix.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the problem solving means of the present invention described above, a 3D pose estimation device, method and computer program capable of estimating not only a 2D pose of a person expressed in an image but also a 3D pose can be provided.

In addition, the present invention can provide a 3D pose estimation device, method, and computer program that efficiently reflect long-distance dependency in the process of estimating a 3D pose by analyzing the pose of a person expressed in an image, thereby improving the accuracy of pose estimation.

FIG. 1 is a configuration diagram of a 3D pose estimation device according to one embodiment of the present invention.
FIGS. 2A to 2C are drawings for explaining an example of a skeleton graph according to one embodiment of the present invention.
FIGS. 3A to 3C are drawings for explaining the configuration of an adjacency matrix according to one embodiment of the present invention.
FIG. 4 is a drawing for explaining the configuration of a lifting network according to one embodiment of the present invention.
FIG. 5 is a drawing for explaining an example of a 3D pose estimated from an image according to one embodiment of the present invention.
FIG. 6 is a flowchart of a 3D pose estimation method according to one embodiment of the present invention.

Hereinafter, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are assigned similar drawing reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element in between. Also, when a part is said to "include" a certain component, this should be understood to mean that it may further include other components, unless specifically stated to the contrary, and does not preclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In this specification, the term 'unit' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be realized using two or more pieces of hardware, and two or more units may be realized by one piece of hardware.

Some of the operations or functions described as being performed by a terminal or device in this specification may instead be performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by a server may also be performed by a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

FIG. 1 is a configuration diagram of a 3D attitude estimation device according to one embodiment of the present invention. Referring to FIG. 1, a 3D attitude estimation device (200) may include a receiving unit (210), a 2D attitude estimation unit (220), and a 3D attitude estimation unit (230). However, the above components (210, 220, 230) are only exemplary components that can be controlled by the 3D attitude estimation device (200).

The components of the 3D pose estimation device (200) of Fig. 1 are generally connected via a network. For example, as shown in Fig. 2, the receiving unit (210), the 2D pose estimation unit (220), and the 3D pose estimation unit (230) may be connected simultaneously or at time intervals.

A network is a connection structure that enables information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), the Internet (WWW: World Wide Web), wired and wireless data communication networks, telephone networks, and wired and wireless television communication networks. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, visible light communication (VLC), LiFi, etc.

The receiving unit (210) of the 3D posture estimation device (200) can receive an image (100) including a person. Here, the image (100) is composed of a plurality of pixels, and each pixel can have characteristics according to the ratio of RGB. The image (100) can be a single image or one of a plurality of frames included in an image that is played back in time series. When the 3D posture estimation device (200) receives continuous images, it can estimate the person's behavior according to changes in the 3D posture.

A person may be expressed in an image (100) received by a 3D pose estimation device (200), and the 3D pose estimation device (200) may analyze the position and shape of the person in the image (100) to estimate a 2D pose, and analyze the estimated 2D pose to output an estimated 3D pose (300), which is the final result.

The 2D pose estimation unit (220) analyzes the image to extract the type and location of joints according to the pose of the person in the image, and can create a skeleton graph based on the type and location of the extracted joints.

In the process of estimating a 2D pose from an input image by the 2D pose estimation unit (220), various techniques that have been disclosed in the past can be used. For example, a bottom-up method can be used in which all joints of a person are estimated in an image and then grouped into a specific pose or a pose of a single human object, or a top-down method can be used in which a human object is extracted from an image using a human detector and then joints are estimated from the extracted human object.

As another example, a classical method for estimating 2D poses can be exemplified by a hand-crafted method in which engineers directly extract features from each part of a person, and a recently popular deep learning-based 2D pose estimation can be used. 2D pose estimation based on deep learning can be used by deriving a bounding box corresponding to the area where a person is located in an image, and then finding key points of a person or object within the bounding box. Here, the key points can mean joint information such as a person's elbow, knee, and wrist. Representative pose estimation models such as OpenPose, CPN, AlphaPose, and HRNet can be utilized as 2D pose estimation methods that utilize deep learning.

The 2D pose estimation unit (220) can output joint information corresponding to key points of a person in the image as a result of estimating a 2D pose from an image. Here, the joint information can include coordinate values where the corresponding joint is located in the image or information on whether the corresponding joint is visible in the image, and such joint information can be utilized as feature information of each node on the skeleton graph.

Here, the skeleton graph refers to a graph in which the main joints (e.g., head, shoulder, elbow, hand, pelvis, knee, foot) necessary for posture estimation among the joints that make up the body skeletal structure are set as nodes on the graph, and the joints that are connected to each other in the human body are set as edges on the graph.

FIGS. 2A to 2C are drawings for explaining skeleton graphs according to an embodiment of the present invention. Referring to FIGS. 2A to 2C, a graph composed of 17 nodes from node 0 to node 16 is illustrated, and each node may correspond to a major joint of the human body. For example, node 0 may correspond to a node corresponding to the pelvis, nodes 1 and 4 may correspond to the left and right hip joints, respectively, and node 7 may correspond to the waist. In addition, the number of each node may be set in the order in which the node corresponding to the pelvis position is searched in a tree or graph with the node corresponding to the pelvis position as the root in a DFS (Depth First Search) manner. However, this is set as an example for convenience of explanation, and different node numbers may be assigned depending on the designer's intention or the model learning environment.

Figure 2a shows nodes (nodes 1, 4, and 7) that are 1 hop away from the reference node 0, Figure 2b shows nodes (nodes 2, 5, and 8) that are 2 hops away from the reference node 0, and Figure 2c shows nodes (nodes 3, 6, 9, 11, and 14) that are 3 hops away from the reference node 0.

Returning to FIG. 1 again, the 3D pose estimation unit (230) can estimate a 3D pose corresponding to the skeleton graph output from the 2D pose estimation unit (220) using an adjacency matrix, a feature matrix, and a weight matrix. Here, the adjacency matrix is based on the connection relationship between nodes in the skeleton graph, the feature matrix is based on the features included in each node of the skeleton graph, and the weight matrix can be based on the filter value applied to each feature of the node.

More specifically, the adjacency matrix may be a matrix having a size of N x N and having 0 or 1 as an element when the number of nodes constituting the skeleton graph is N. The feature matrix is a matrix including information on the feature of each node and may include information on the coordinate values of each joint estimated by the 2D pose estimation unit (220) within the image or whether the coordinates within the image are displayed. When the dimension value or the number of features for the feature of each node is D, the feature matrix may have a size of D x N. In addition, the weight matrix may be a matrix having a size of D' x D assuming that the feature dimension is converted from D to D' as a matrix for weights that can be learned through a convolutional neural network.

In the process of updating the features of each node, conventional graphical convolutional neural networks use an adjacency matrix to reflect the relationship between nodes connected to arbitrary nodes. An adjacency matrix is a matrix with as many rows and columns as the number of nodes to express the structure of a graph. For example, if nodes i and j in a graph are connected via an edge, the elements of row i and column j of the adjacency matrix are set to 1, otherwise they are set to 0. The adjacency matrix used in conventional graph convolutional neural networks reflects the self-connection of an arbitrary node to itself, and sets the diagonal elements of the matrix to 1. Through this, in the process of updating the features of a node, the features of the node itself and the features of other nodes adjacent to the node are reflected together. Here, an adjacent node is a node that has an edge that is directly connected to a reference node, and means a node that is 1 hop away from the reference node. In addition, if the edges of the graph have no direction, the adjacency matrix has the form of a symmetric matrix that is symmetrical to each other based on the diagonal elements.

However, if the features are updated by reflecting only the information about the reference node and the nodes adjacent to the reference node as above, the information about nodes at a further hop distance cannot be properly reflected, which causes problems related to long-range dependency, reducing the accuracy of detail estimation.

More specifically, for conventional graph convolutional neural networks, feature updates are performed using the following formula.

<Formula 1>

Here, H' represents the updated feature matrix, σ represents the activation function exemplified by ReLU, and W represents the learnable weight matrix. can mean a symmetric normalized adjacency matrix that includes self-connection for the adjacency matrix of the skeleton graph. And H can mean an unupdated feature matrix.

As mentioned above, when updating the feature matrix using a symmetrically normalized adjacency matrix to include self-connection, there is a problem that it makes it difficult to learn various relationship patterns because the weight matrix W is shared equally by each node.

To solve this problem, the 3D pose estimation unit (230) can use different adjacency matrices corresponding to each repeated process in the process of deriving the updated feature matrix of the kth adjacency matrix (0<k<K, where K is a natural number greater than or equal to 2 that can be arbitrarily set by the designer and k is a natural number smaller than K) when updating the feature matrix.

More specifically, the different adjacency matrices can be set so that, when obtaining the kth updated feature matrix, the adjacency matrix has a value of 1 only for nodes that are k hops away on the skeleton graph. Here, the adjacency matrix is a symmetric matrix with diagonal elements of 0 and can be normalized.

Figures 3a to 3c are drawings for explaining the configuration of an adjacency matrix according to one embodiment of the present invention. First, referring to Figure 3a, when the k value is 1, i.e., the adjacency matrix used when obtaining the first updated feature matrix _1. You can see the composition of the adjacency matrix. The configuration of ₁ reflects the connection status between nodes in the graphs of Figures 2a to 2c examined above.

More specifically, each row in Fig. 3a is represented by ₁ (n) (n is a natural number), n can mean the number of joints, and the elements of the corresponding row can mean the neighboring relationship for the joints corresponding to the numbers of each column. For example, ₁ (0) can mean a neighboring joint 1 hop away from joint 0. Therefore, in Fig. 3a, The row corresponding to ₁ (0) can have the elements in columns 1, 4, and 7 set to 1, and the remaining elements can be set to 0 because they are not neighboring joints within 1 hop.

Likewise, in Fig. 3a ₁ (1) may mean a neighboring joint that is 1 hop away from joint 1. Therefore, in Fig. 3a, The row corresponding to ₁ (1) can have the elements in columns 0 and 2 set to 1 and the remaining elements set to 0 because they are not neighboring joints within 1 hop.

As described above, it can be seen that the adjacency matrix used according to the embodiment of the present invention sets only the nodes connected to the k-hop distance to 1 when updating the k-th feature matrix, and sets the other nodes to 0, and that the diagonal elements are also 0. For convenience of explanation, FIGS. 3A to 3C exemplarily represent the first and second rows of the adjacency matrix, and information on the other rows is omitted.

Figure 3b is an adjacency matrix used when the above k value is 2. ₂ is an example. It can be seen that the adjacency matrix of Fig. 3b sets the values for nodes 2, 5, and 8, which are located 2 hops away from node 0 in the graph of Fig. 2b, to 1, and the values for nodes 1, 4, and 7, which are located 1 hop away, to 0.

Figure 3c is an adjacency matrix used when the above k value is 3. ₃ is an example. It can be seen that the adjacency matrix of Fig. 3c sets the values for nodes 3, 6, 9, 11, and 14 located 3 hops away from node 0 in the graph of Fig. 2c to 1, and the values for nodes located 1 or 2 hops away are set to 0.

Since the adjacency matrix used in the embodiment of the present invention is a symmetric matrix with diagonal elements of 0, the self-connection of the node cannot be reflected in the process of updating the feature matrix, so it is necessary to calculate the updated feature matrix as in <Equation 2> in order to additionally reflect the feature matrix for the node itself in the calculation of the updated feature matrix.

<Formula 2>

Here, H' represents the updated feature matrix, σ represents the activation function, W(0) represents the weight matrix corresponding to the node itself, W(1) represents the weight matrix corresponding to the neighboring transformation, and H represents the non-updated feature matrix. can mean the above adjacency matrix. That is, can be a normalized symmetric matrix with diagonal elements of 0. As in <Formula 2>, by separating the self-connection corresponding to the diagonal elements of the adjacency matrix and the connection between adjacent nodes and calculating and adding the feature matrix for each, the accuracy of 3D pose estimation can be further improved.

The adjacency matrix according to the embodiment of the present invention can be set to 1 only for connections of nodes located at a distance of k hops. Since the proposed adjacency matrix sets connections of nodes at a distance of less than k hops to 0 in representing relationships between neighboring joints, nodes at a distance of less than k hops have less correlation, and through this, long-distance dependencies between joints can be efficiently reflected, thereby increasing the accuracy of 3D pose estimation.

In another embodiment of the present invention, a modulation method may be used to additionally merge the aggregated features of each hop in the previous embodiment to further improve the accuracy of 3D pose estimation.

<Formula 3>

Here, H' represents the updated feature matrix, σ represents the activation function, and λ _k represents the aggregated features (WH) of each k-hop distance. ) and the features merged up to k+1 hop distance (C _k+1 ) are modeled as a learnable modulation matrix, W is a weighting matrix, H is a feature matrix, is an adjacency matrix for the k-hop distance, and may refer to the adjacency matrix described above through FIGS. 3a to 3c. And C _k+1 may refer to features merged up to the k+1-hop distance.

That is, in order to reflect the long-distance dependency more efficiently in the learning model, the feature representation of the k-hop distance can be affected by the merged features C _k+1 up to the k+1-hop distance, and for this, the modulation matrix λ _k can be used. When k satisfies the range of 0<k<K and the K value is defined by the user, C _k means the feature aggregated up to the k-hop distance. For example, when the value of K is set to 4 and the range of k is from 1 to 3, C ₂ can be calculated by the combination of the features corresponding to the 3-hop distance and the features corresponding to the 2-hop distance. In more detail, C _k can be calculated by aggregating the features through a learnable matrix that modulates the features calculated for each hop distance, such as λ _k . Through this, it can be made to have a smaller weight as the hop distance from the neighboring joint is farther.

Fig. 4 is a drawing for explaining the configuration of a lifting network according to one embodiment of the present invention. The lifting network of Fig. 4 may be a model used in the process of estimating a 3D posture from a skeleton graph by a 3D posture estimation unit (230).

The lifting network includes multiple blocks, and each of the multiple blocks may include a convolution layer, a batch normalization layer, and a ReLU activation function layer (BatchNorm & ReLU layer).

Except for the last block, the output of the convolution layer in the remaining blocks can pass through the batch normalization and ReLU activation function layers. The network structure exemplified in Fig. 4 has the number of channels set to 128, which can be changed according to the user's design intention.

And, the lifting network of the present invention can configure at least two blocks as one residual block, and the lifting network can include a plurality of such residual blocks.

In addition, the lifting network may be provided to a third block by merging input data of a first block among the blocks constituting the lifting network with output data of a second block through a skip connection. Here, the second block may be a block in the next order of the first block, and the third block may be a block in the next order of the second block. More specifically, the first block and the second block may constitute one residual block, and the third block may be included in the residual block in the next order.

The convolutional layer of the last block does not apply any additional modules and outputs the 3D position values of each joint as 3D pose information.

FIG. 5 is a drawing for explaining an example of a 3D pose estimated from an image according to one embodiment of the present invention.

FIG. 5 is an example of a result of 3D pose estimation for a dataset according to an embodiment of the present invention, showing a result value (MM-GCN) and a true value (Ground truth) of 3D pose information estimated by a 3D pose estimation device (200) for an input image (Input). Through a comparison of the result value (MM-GCN) and the true value (Ground truth), it can be seen that 3D poses of different people performing various activities are accurately estimated. The Human3.6M dataset can be used as a dataset for measuring the accuracy of the lifting model.

FIG. 6 is a flowchart for explaining a 3D pose estimation method according to an embodiment of the present invention. The 3D pose estimation method illustrated in FIG. 6 includes steps that are processed in time series according to the embodiments illustrated in FIGS. 1 to 5. Therefore, even if there is any omitted content below, it is also applied to a method for estimating a 3D pose of a person in a 3D pose estimation device according to the embodiments illustrated in FIGS. 1 to 5.

Referring to FIG. 6, a 3D pose estimation method performed by a 3D pose estimation device may include a receiving step (S100), a 2D pose estimation step (S200), and a 3D pose estimation step (S300).

The 3D receiving step (S100) may be a step of receiving an image including a person. In addition, the 2D pose estimation step (S200) may be a step of analyzing the image to extract the types and positions of joints according to the person's pose, and generating a skeleton graph based on the types and positions of the extracted joints, and the 3D pose estimation step (S300) may be a step of estimating a 3D pose corresponding to the skeleton graph using an adjacency matrix. Here, the adjacency matrix may use different adjacency matrices corresponding to each repeated process in the process of deriving the k-th (0<k<K, where K is a natural number greater than or equal to 2 that can be arbitrarily set by the designer) updated feature matrix.

In the above description, steps S100 to S300 may be further divided into additional steps or combined into fewer steps, depending on the implementation example of the present invention. In addition, some steps may be omitted as needed, or the order between the steps may be switched.

The method for estimating a 3D posture of a person in the 3D posture estimation device described through FIGS. 1 to 6 can also be implemented in the form of a computer program stored in a computer-readable recording medium executed by a computer or a recording medium including commands executable by a computer. In addition, the method for estimating a 3D posture of a person in the 3D posture estimation device described through FIGS. 1 to 6 can also be implemented in the form of a computer program stored in a computer-readable recording medium executed by a computer.

Computer-readable recording media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Additionally, computer-readable recording media can include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential characteristics of the present invention. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and likewise, components described as distributed may be implemented in a combined manner.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

100: Input image
200: 3D pose estimation device
210: Receiver
220: 2D pose estimation unit
230: 3D pose estimation unit
300: Estimated 3D pose information

Claims (20)

A receiving unit for receiving an image containing a person;
A 2D pose estimation unit that analyzes the image above to extract the types and locations of joints according to the person's pose, and generates a skeleton graph based on the types and locations of the extracted joints; and
A 3D pose estimation unit that estimates a 3D pose corresponding to the skeleton graph using an adjacency matrix;
Including,
A 3D pose estimation device, characterized in that different adjacency matrices are used corresponding to each repeated process in the process of deriving the kth (0<k<K, where K is a natural number greater than or equal to 2) updated feature matrix.
In the first paragraph, the 3D pose estimation unit
By updating the feature matrix using the feature matrix and weight matrix together in the above adjacency matrix, the 3D pose corresponding to the skeleton graph is estimated,
The above adjacency matrix is based on the connection relationship between nodes in the above skeleton graph,
The above feature matrix is based on the features contained in each node,
A 3D pose estimation device, characterized in that the above weighting matrix is based on filter values applied to each of the above features.
In the second paragraph,
The above 3D pose estimation unit updates the feature matrix of each node constituting the skeleton graph.
An updated feature matrix is derived by multiplying the above feature matrix by the above adjacency matrix and the above weight matrix,
A 3D pose estimation device characterized in that the 3D pose is estimated by repeating the process of obtaining a further updated feature matrix by multiplying the adjacency matrix and the weight matrix for the updated feature matrix.
In the third paragraph,
A 3D pose estimation device, characterized in that when obtaining the kth updated feature matrix, the adjacency matrix has a value of 1 only for nodes that are k hops away on the skeleton graph.
In paragraph 4,
A 3D pose estimation device characterized in that the above adjacency matrix is a symmetric matrix with diagonal elements of 0 and is normalized.
In paragraph 4,
The above updated feature matrix is calculated as follows:

Here, H' represents the updated feature matrix,
σ represents the activation function,
λ _k denotes a learnable modulation matrix to model the relationship between the aggregated features at each k-hop distance and the merged features up to k+1-hop distance,
W represents the above weighted matrix,
H represents the above feature matrix,
denotes the adjacency matrix for the above k-hop distance,
A 3D pose estimation device characterized in that C _k+1 denotes features merged up to k+1 hops distance.
In the first paragraph,
The above 3D pose estimation unit estimates the 3D pose from the skeleton graph through a lifting network,
The above lifting network comprises a plurality of blocks,
A 3D pose estimation device, wherein each of the plurality of above blocks includes a convolutional layer, a batch normalization layer, and a ReLU activation function layer.
In Article 7,
The above lifting network
A 3D pose estimation device, characterized in that input data of a first block is combined with output data of a second block and provided to a third block, wherein the second block is a block in the next order of the first block, and the third block is a block in the next order of the second block.
In Article 8,
A 3D pose estimation device characterized in that the last block of the lifting network does not include the batch normalization and ReLU activation function layers and outputs the 3D position values of each joint.
In a 3D pose estimation method performed by a 3D pose estimation device,
A receiving step for receiving an image containing a person;
A 2D pose estimation step of analyzing the image to extract the type and location of joints according to the person's pose, and generating a skeleton graph based on the type and location of the extracted joints; and
A 3D pose estimation step for estimating a 3D pose corresponding to the skeleton graph using an adjacency matrix; including;
A 3D pose estimation method, characterized in that different adjacency matrices are used corresponding to each repeated process in the process of deriving the kth (0<k<K, where K is a natural number greater than or equal to 2) updated feature matrix.
In the 10th paragraph, the 3D pose estimation step
This is a step of estimating a 3D pose corresponding to the skeleton graph by updating the feature matrix using the feature matrix and weight matrix together in the above adjacency matrix.
The above adjacency matrix is based on the connection relationship between nodes in the above skeleton graph,
The above feature matrix is based on the features contained in each node,
A 3D pose estimation method, characterized in that the above weighting matrix is based on filter values applied to each of the above features.
In Article 11,
The above 3D pose estimation step updates the feature matrix of each node constituting the skeleton graph.
An updated feature matrix is derived by multiplying the above feature matrix by the above adjacency matrix and the above weight matrix,
A 3D pose estimation method characterized by comprising a step of estimating the 3D pose by repeating the process of obtaining a further updated feature matrix by multiplying the adjacency matrix and the weight matrix for the updated feature matrix.
In Article 12,
A 3D pose estimation method, characterized in that when obtaining the kth updated feature matrix, the adjacency matrix has a value of 1 only for nodes that are k hops away on the skeleton graph.
In Article 13,
A 3D pose estimation method characterized in that the above adjacency matrix is a symmetric matrix with diagonal elements of 0 and is normalized.
In Article 13,
The above updated feature matrix is calculated as follows:

Here, H' represents the updated feature matrix,
σ represents the activation function,
λ _k denotes a learnable modulation matrix to model the relationship between the aggregated features at each k-hop distance and the merged features up to k+1-hop distance,
W represents the above weighted matrix,
H represents the above feature matrix,
denotes the adjacency matrix for the above k-hop distance,
A 3D pose estimation method characterized in that C _k+1 means the merged features up to k+1 hop distance.
In Article 10,
The above 3D pose estimation step is a step of estimating the 3D pose from the skeleton graph through a lifting network.
The above lifting network comprises a plurality of blocks,
A 3D pose estimation method, wherein each of the plurality of above blocks includes a convolutional layer, a batch normalization layer, and a ReLU activation function layer.
In Article 16,
The above lifting network
A 3D pose estimation device, characterized in that input data of a first block is combined with output data of a second block and provided to a third block, wherein the second block is a block in the next order of the first block, and the third block is a block in the next order of the second block.
In Article 16,
A 3D pose estimation method, characterized in that the last block of the lifting network does not include the batch normalization and ReLU activation function layers and outputs the 3D position values of each joint.
A computer program stored on a computer-readable recording medium comprising a sequence of commands for estimating a 3D pose from an image,
Receives an image containing a person as input,
2D pose estimation is performed by analyzing the image above to extract the type and location of joints according to the person's posture, and generating a skeleton graph based on the type and location of the extracted joints.
Estimate the 3D pose corresponding to the above skeleton graph using the adjacency matrix.
A computer program stored on a computer-readable recording medium, which includes a sequence of commands that cause different adjacency matrices to be used corresponding to each repeated process in the process of deriving the kth (0<k<K, where K is a natural number greater than or equal to 2) updated feature matrix.
In Article 19,
When estimating the above 3D pose, the 3D pose corresponding to the skeleton graph is estimated by updating the feature matrix by using the feature matrix and the weight matrix together in the above adjacency matrix.
The above adjacency matrix is based on the connection relationship between nodes in the above skeleton graph,
The above feature matrix is based on the features contained in each node,
A computer program stored on a computer-readable recording medium, wherein the weighting matrix includes a sequence of commands based on filter values applied to each of the features.