JP2019016164A - Learning data generation device, estimation device, estimation method, and computer program - Google Patents

Abstract

To provide a learning data generation device, an estimation device, an estimation method, and a computer program that can easily generate a plurality of learning data including three-dimensional joint information.SOLUTION: A leaning data generation device comprises: a silhouette image rendering unit for obtaining three-dimensional model images that are computer graphic images expressing the three-dimensional model of a subject having joints, for each perspective defined in the periphery of the three-dimensional model, and generating the silhouette images of the three-dimensional model for each perspective by carrying out rendering processing on the three-dimensional model images; a camera parameter unit for obtaining camera parameters for each perspective; a shape information restoration unit for, based on the camera parameters of each perspective, from the silhouette images for each perspective, restoring the three-dimensional shape information of the three-dimensional model; and a joint information voxelization unit for generating the three-dimensional joint information of the three-dimensional model in a voxel space that is equal to the voxel space of the three-dimensional shape information of the three-dimensional model.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a learning data generation device, an estimation device, an estimation method, and a computer program.

  By applying a technique for measuring the movement of a person's joints, it is possible to give a lively action to a computer graphics character that imitates a person appearing in a movie or the like. Therefore, a technique for measuring the movement of a person's joint is an indispensable technique for improving the quality of the entire content. In addition to the entertainment field as described above, techniques for measuring the movement of a person's joints are used in various fields. For example, in the medical field, it is important as information for grasping the patient's condition.

  Hereinafter, the number assigned to each joint of the three-dimensional model of the subject is referred to as “joint part information”. Hereinafter, information indicating the position of each joint of the three-dimensional model of the subject is referred to as “joint position information”. Hereinafter, information including joint part information and joint position information is referred to as “joint information”.

  As described above, joint information is important information in various fields. However, acquiring joint information involves a great deal of labor. There is a data acquisition technique using motion capture as a technique for acquiring joint information. In motion capture, it is necessary to have a person to be measured wear a special suit, and it is necessary to calibrate the space in advance, which requires complicated work. Other technologies also have various problems, such as those that require special equipment and can only be used in limited environments.

  In recent years, a technique for robustly estimating the joint position of a subject captured in an image using deep learning has been announced (see Non-Patent Document 1). With this technique, even if there are a plurality of persons in the image, robust estimation is possible.

L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. Gehler, B. Schiele, “DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016).

  However, in the conventional method, the estimated joint information is only the two-dimensional joint position on the image. Therefore, it is insufficient as joint information used for animation generation. On the other hand, when trying to estimate three-dimensional joint information using machine learning such as deep learning, it is necessary to obtain a large amount of learning data including three-dimensional joint information in advance, which is difficult.

  In view of the above circumstances, an object of the present invention is to provide a learning data generation device, an estimation device, an estimation method, and a computer program that can more easily generate a plurality of learning data including three-dimensional joint information. It is said.

  According to one aspect of the present invention, a 3D model image, which is a computer graphics image representing a 3D model of a subject having a joint, is acquired for each viewpoint defined around the 3D model, and the 3D model image is acquired. Based on the camera parameters for each viewpoint, a silhouette image rendering unit that generates a silhouette image of the three-dimensional model for each viewpoint by performing rendering processing, a camera parameter unit that acquires camera parameters for each viewpoint, and A three-dimensional shape information restoring unit that restores the three-dimensional shape information of the three-dimensional model from the silhouette image of each viewpoint, and the three-dimensional shape information in the same voxel space as the three-dimensional shape information of the three-dimensional model. A learning data generation device comprising a joint information voxelization unit for generating three-dimensional joint information of a model A.

  One aspect of the present invention is the learning data generation device described above, further including a learning unit that learns parameters of a deep neural network that outputs the joint information according to the shape information, and an output layer of the deep neural network Has a number of channels corresponding to the number of joints represented by the joint information.

  According to one aspect of the present invention, there is provided a deep neural network that is trained to output three-dimensional joint information of the three-dimensional model according to the three-dimensional shape information of the three-dimensional model generated by the learning data generation device. The estimation apparatus includes an analysis unit that estimates the three-dimensional joint information of the subject related to the three-dimensional model by using the three-dimensional shape information of the subject related to the three-dimensional model as an input of the deep neural network. .

  One aspect of the present invention is the learning data generation device described above, further including a learning unit that learns parameters of a deep neural network that outputs the joint information according to the shape information, and the learning unit includes a plurality of learning units. The shape information set is generated by combining the shape information, and whether or not the shape information set is used for learning of parameters of the deep neural network is determined based on a predetermined condition.

  One aspect of the present invention is the above estimation apparatus, further comprising a storage unit that stores a configuration and parameters of a deep neural network that outputs the joint information according to the shape information, and the parameters of the deep neural network are A parameter based on a learning result using the set of shape information that satisfies a predetermined condition among the set of shape information generated by collecting a plurality of pieces of the shape information.

  One aspect of the present invention is an estimation method executed by an estimation device, and the three-dimensional joint information of the three-dimensional model according to the three-dimensional shape information of the three-dimensional model generated by the learning data generation device. Is estimated using a deep neural network learned to output the three-dimensional joint information of the subject related to the three-dimensional model.

  One embodiment of the present invention is a computer program for causing a computer to function as the learning data generation device.

  One embodiment of the present invention is a computer program for causing a computer to function as the above estimation device.

  According to the present invention, it is possible to more easily generate a plurality of learning data including three-dimensional joint information.

It is a figure which shows the example of a structure of the estimation system in 1st Embodiment. It is a figure which shows the example of a structure of the learning data generation part in 1st Embodiment. It is a figure which shows the example of a structure of the learning part in 1st Embodiment. It is a figure which shows the example of a structure of the deep neural network in 1st Embodiment. It is a figure which shows the example of a structure of the input data generation part in 1st Embodiment. It is a figure which shows the example of a structure of the analysis part in 1st Embodiment. It is a flowchart which shows the example of operation | movement of the estimation apparatus in 1st Embodiment. It is a flowchart which shows the example of operation | movement of the learning data generation part in 1st Embodiment. It is a flowchart which shows the example of operation | movement of the learning part in 1st Embodiment. It is a flowchart which shows the example of operation | movement of the input data generation part in 1st Embodiment. It is a flowchart which shows the example of operation | movement of the analysis part in 1st Embodiment. It is a figure which shows the example of a structure of the estimation system in 2nd Embodiment.

Embodiments of the present invention will be described in detail with reference to the drawings.
(First embodiment)
In the following, the subject is a living organism or object having a joint, such as a person, an animal, an insect, or a robot. In the following, the subject is a person as an example.

  FIG. 1 is a diagram illustrating an example of the configuration of the estimation system 1a. The estimation system 1a is a system including the estimation device 10. The estimation device 10 is an information processing device that estimates a three-dimensional joint position of a subject. The estimation device 10 is, for example, a server device, a personal computer device, a tablet terminal, a smartphone terminal, or the like. For example, the estimation device 10 may execute the estimation process by using a plurality of information processing devices that are distributed. The estimation device 10 includes a learning device 11, an input data generation unit 14, and an analysis unit 15.

  The learning device 11 includes a learning data generation unit 12 and a learning unit 13. The learning data generation unit 12 may be provided in a learning data generation device that is different from the learning device 11. In this case, the learning data generated by the learning data generation device may be given to the learning device 11 via a network or a storage medium. The learning data generation unit 12 as the learning data generation device may include a learning unit 13. The estimation device 10 may be separate from the learning device 11. That is, the estimation system 1a may include the estimation device 10 and the learning device 11 separately.

  The estimation device 10 may further include a storage unit. The storage unit is a storage device having a nonvolatile recording medium (non-temporary recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit may store various data related to machine learning, images, and computer programs. The various types of data related to machine learning are, for example, data representing an algorithm, data representing a configuration and parameters of a deep neural network (DNN).

  Part or all of each functional unit is realized by, for example, a processor such as a CPU (Central Processing Unit) executing a computer program stored in a predetermined storage unit. Part or all of each functional unit may be realized by using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

  The learning data generation unit 12 (learning data generation device) generates learning data used for machine learning. The learning data generated by the learning data generation unit 12 can be used for machine learning other than the deep neural network. A three-dimensional model image, which is an image representing a three-dimensional model of a subject, is generated in advance by computer graphics. The learning data generation unit 12 acquires a three-dimensional model image to which joint information is added. The learning data generation unit 12 uses a three-dimensional model silhouette image (hereinafter referred to as a “multi-viewpoint model silhouette image”) captured from a plurality of cameras (multi-viewpoints) around the three-dimensional model as a third-order computer graphics. Generate from the original model image.

  In the following, the number of joints (joint part information) of the subject is 16 as an example. For example, when the three-dimensional joint information of the fingertip is estimated, the number of joints of the subject is increased. Also, the number of channels in the output layer of the deep neural network is increased according to the number of joints of the subject.

  The learning data generation unit 12 acquires camera parameters for each multi-viewpoint model silhouette image. The learning data generation unit 12 generates a multi-view model silhouette image by performing rendering processing on the three-dimensional model image for a plurality of viewpoints. The learning data generation unit 12 restores the three-dimensional shape information (voxel shape information) of the three-dimensional model from the multi-viewpoint model silhouette image based on the camera parameters of each viewpoint. That is, the learning data generation unit 12 restores the three-dimensional shape information of the three-dimensional model based on the camera parameters of each viewpoint and the multi-viewpoint model silhouette image. The three-dimensional shape information of the three-dimensional model includes binary information. The numerical value of the 3D shape information of the 3D model is 1 for the shape portion of the 3D model and 0 for the shape portion other than the 3D model. The learning data generation unit 12 outputs the three-dimensional shape information of the three-dimensional model to the learning unit 13 as learning data (learning teacher data) used for machine learning.

  The learning data generation unit 12 extracts joint information from the three-dimensional model image to which the joint information is given. The learning data generation unit 12 voxels joint information in the same voxel space as the three-dimensional shape information of the three-dimensional model. That is, the learning data generation unit 12 generates the three-dimensional joint information (voxel joint information) of the three-dimensional model in the same voxel space as when the three-dimensional shape information of the three-dimensional model is restored.

  When the learning data generation unit 12 generates a plurality of learning data, the learning data generation unit 12 changes the three-dimensional shape of the subject generated by computer graphics and changes the three-dimensional shape information of the three-dimensional model. change. When the learning data generation unit 12 generates a plurality of learning data, the learning data generation unit 12 performs the same voxelization process on a plurality of three-dimensional models having different shape information.

  The learning unit 13 performs machine learning using the learning data. The learning unit 13 does not need to perform machine learning each time the estimation apparatus 10 performs the estimation process. For example, the learning unit 13 may complete the machine learning before the estimation apparatus 10 executes the estimation process.

  The learning unit 13 is not limited to the deep neural network, and may execute machine learning such as genetic programming or clustering, for example. In the first embodiment, the learning unit 13 executes a deep neural network learning process as an example of machine learning. The input of the deep neural network is 3D shape information of the 3D model. The output of the deep neural network is three-dimensional joint information. The output channel of the deep neural network is associated with joint part information constituting three-dimensional joint information.

  When the learning data generation unit 12 generates a plurality of learning data, the learning unit 13 learns the parameters of the deep neural network based on the plurality of learning data generated by the learning data generation unit 12. The learning unit 13 outputs information representing the configuration and parameters of the learned deep neural network to the analysis unit 15.

  Hereinafter, a real image obtained by imaging a subject from a plurality of cameras (multi-viewpoints) around the subject is referred to as a “multi-view real image”. The input data generation unit 14 acquires a multi-view photographed image in which a subject as an object for which joint information is estimated is captured. The input data generation unit 14 acquires camera parameters of the multi-viewpoint photographed image.

  The input data generation unit 14 applies the voxelization process similar to the voxelization process performed on the multi-view model image by the learning data generation unit 12 to restore the three-dimensional shape information of the live-action subject. To do. The three-dimensional shape information of the subject includes binary information. The numerical value of the 3D shape information of the subject is 1 in the shape portion of the 3D model, and 0 in the shape portion other than the shape portion of the 3D model.

  The analysis unit 15 acquires the parameters of the learned deep neural network from the learning unit 13. The analysis unit 15 uses the three-dimensional shape information of the subject restored by the input data generation unit 14 as the input of the learned deep neural network. The analysis unit 15 uses the three-dimensional joint position information of the subject as an output for each channel of the learned deep neural network. Thereby, the analysis unit 15 can estimate the three-dimensional joint information of the subject.

Next, details of each functional unit will be described.
FIG. 2 is a diagram illustrating an example of the configuration of the learning data generation unit 12. The learning data generation unit 12 includes a silhouette image rendering unit 121, a camera parameter output unit 122, a shape information restoration unit 123, a joint information output unit 124, and a joint information voxelization unit 125.

  The silhouette image rendering unit 121 acquires a three-dimensional model image to which joint information is added for each viewpoint (camera position). The silhouette image rendering unit 121 generates a silhouette image of the 3D model of the subject for each viewpoint by rendering the 3D model image for a plurality of viewpoints. That is, the silhouette image rendering unit 121 generates a multi-viewpoint model silhouette image by performing rendering processing on the three-dimensional model image for a plurality of viewpoints. The plurality of viewpoints are determined according to the use of shape information estimation. The plurality of viewpoints are, for example, viewpoints around the entire periphery of the subject. When the positions (viewpoints) of a plurality of cameras that capture the subject are fixed, the orientations of the plurality of cameras may be fixed. The silhouette image rendering unit 121 outputs the multi-viewpoint model silhouette image to the shape information restoring unit 123.

  The camera parameter output unit 122 (camera parameter unit) acquires camera parameters for each viewpoint by extracting camera parameters from the three-dimensional model image to which the joint information is added. That is, the camera parameter output unit 122 acquires camera parameters for each multi-viewpoint model silhouette image. The camera parameter output unit 122 outputs the camera parameters of the multi-viewpoint model silhouette image to the shape information restoration unit 123 for each viewpoint.

  The shape information restoration unit 123 restores the three-dimensional shape information of the three-dimensional model from the multi-view model silhouette image based on the camera parameters of each viewpoint. That is, the shape information restoration unit 123 restores the three-dimensional shape information of the three-dimensional model based on the camera parameters of each viewpoint and the multi-viewpoint model silhouette image. The method of shape information restoration processing is not limited to a specific method, but is, for example, a visual volume intersection method. Various parameters such as the range of the voxel space of the shape information to be restored, the volume of the voxel in the voxel space of the shape information to be restored, and the resolution of the voxel in the voxel space of the shape information to be restored are arbitrarily determined. The shape information restoration unit 123 outputs the three-dimensional shape information of the three-dimensional model to the learning unit 13.

  The joint information output unit 124 acquires a three-dimensional model image to which joint information is added. The joint information output unit 124 extracts three-dimensional joint information from the three-dimensional model image to which the joint information is added. The coordinate system representing the joint position information constituting the joint information is, for example, a world coordinate system using the xyz axis. The joint information output unit 124 outputs the three-dimensional joint information to the joint information voxelization unit 125.

  The joint information voxelization unit 125 voxels joint information in a voxel space similar to the 3D shape information of the 3D model. That is, the joint information voxelization unit 125 generates the three-dimensional joint information of the three-dimensional model in the same voxel space as that when the three-dimensional shape information of the three-dimensional model is restored. The number which is the joint part information given to each joint position information may be given to each joint position information in an arbitrary order or rule. For example, the number which is joint part information may be a continuous value of 1 to 16 given to each joint position information. The joint information voxelization unit 125 outputs the three-dimensional joint information of the three-dimensional model (joint information obtained by voxelizing the three-dimensional model) to the learning unit 13.

  FIG. 3 is a diagram illustrating an example of the configuration of the learning unit 13. The learning unit 13 includes a voxel set generation unit 131, a voxel set determination unit 132, a network construction unit 133, and a parameter learning unit 134.

  The voxel set generation unit 131 acquires three-dimensional shape information of a plurality of three-dimensional models from the shape information restoration unit 123. The voxel set generation unit 131 generates a set of three-dimensional shape information of the three-dimensional model (hereinafter referred to as “shape information voxel set”) by collecting the three-dimensional shape information of a plurality of three-dimensional models. That is, the voxel set generation unit 131 converts the three-dimensional shape information of a plurality of three-dimensional models into a voxel set. When the voxel set generation unit 131 converts the 3D shape information of the 3D model into voxel sets, the spatial positions of the shape information voxel sets may overlap, or the spatial positions of the shape information voxel sets do not overlap. In this way, the spatial position of any shape information voxel set may be moved.

  The voxel set generation unit 131 acquires three-dimensional joint information of a plurality of three-dimensional models from the joint information voxelization unit 125. The voxel set generation unit 131 generates a set of three-dimensional joint information of the three-dimensional model (hereinafter referred to as “joint information voxel set”) by collecting the three-dimensional joint information of the plurality of three-dimensional models. That is, the voxel set generation unit 131 converts the three-dimensional joint information of a plurality of three-dimensional models into a voxel set. When the voxel set generation unit 131 converts the three-dimensional joint information of the three-dimensional model into a voxel set, the joint information voxel sets may overlap with each other, or the joint information voxel sets may not overlap with each other. The spatial position of the joint information voxel set may be moved.

  The voxel set generation unit 131 associates the shape information voxel set and the joint information voxel set having the same spatial position. The same parameters are determined as various parameters of the shape information voxel set and the joint information voxel set having the same spatial position. For example, in the following, in order to perform consistent learning processing in the deep neural network, the length of one side of the cube space of the shape information voxel set and the joint information voxel set is set to 8 which is an example of an exponential power of 2 . Note that the length of one side of the voxel set space is not limited to a specific length.

  The voxel set determination unit 132 selects a voxel set used for learning parameters of the deep neural network. The voxel set determination unit 132 determines, for each shape information voxel set, whether or not the shape information voxel set is used for learning the parameters of the deep neural network based on a predetermined condition. For example, for the shape information voxel set, the voxel set determination unit 132 determines whether the space of the shape portion of the three-dimensional model is equal to or greater than a predetermined ratio with respect to the entire space of the shape information voxel set. That is, the voxel set determination unit 132 uses the shape information voxel set in which the number of voxels whose numerical value is 1 is equal to or greater than a predetermined ratio to the number of voxels in the entire shape information voxel set to learn the parameters of the deep neural network. It is determined that this is a voxel set to be used. The predetermined ratio is, for example, one third.

  The voxel set determining unit 132 determines, for each joint information voxel set, whether or not the joint information voxel set is used for learning the parameters of the deep neural network. The voxel set determining unit 132 determines that the joint information voxel set associated with the shape information voxel set used for parameter learning is a joint information voxel set used for deep neural network parameter learning.

  Deep learning requires a large amount of learning data and a huge amount of processing time. For this reason, conventionally, there is a case where high-speed learning processing of deep learning using a graphics processing unit (GPU) is tried. In the present embodiment, the target of the learning process is three-dimensional data. For this reason, the GPU handles much larger amount of learning data than the learning process of two-dimensional data. When a large amount of input learning data is used as it is for deep neural network learning, there is a high possibility that the GPU memory is insufficient. Therefore, the voxel set determination unit 132 converts the target of the learning process into a voxel set, and causes the parameter learning unit 134 to selectively learn only the voxel set necessary for learning. As a result, the learning unit 13 can save memory and increase the processing speed.

  In addition, since the learning unit 13 handles the shape information as three-dimensional data, the ratio of the area in which no shape information exists in the voxel set to the entire space of the voxel set is very large. In other words, the ratio of the area having no meaning as shape information in the voxel set to the entire space of the voxel set is very large. For this reason, regions that have no meaning as shape information in the voxel set have a strong influence on the parameter learning results when they are used for deep neural network parameter learning as well as regions that have meaning as shape information. It will end up. Therefore, the use of the region having no meaning as shape information by the learning unit 13 for learning the parameters of the deep neural network has a great adverse effect on the accuracy of the parameter learning results.

  Since the learning unit 13 converts the three-dimensional shape information of the three-dimensional model into a voxel set, and excludes a region that has no meaning as the three-dimensional shape information of the three-dimensional model from the parameter learning target, the learning result of the parameter Accuracy can be improved. Note that when the loss calculation is performed, the learning unit 13 may set the weight of the joint part information (label) of the region having no meaning as the shape information to a very small value.

  FIG. 4 is a diagram illustrating an example of the configuration of a deep neural network. The network construction unit 133 constructs a deep neural network. The input of the deep neural network is a shape information voxel set having information of the kernel size “8 × 8 × 8” and the number of channels of 1ch. The input shape information voxel set of the deep neural network is a shape information voxel set determined to be used for learning by the voxel set determination unit 132.

  The deep neural network uses a "convolution" process (convolution process) using a filter with a kernel size of "3x3x3" and a stride with a kernel size of "2x2x2" for the input shape information voxel set A pooling process with a width of 2 is executed with the number of channels of 10 ch. The activation function is, for example, ReLU (Rectified Linear Unit function) (“Conv1” in the upper part of FIG. 4).

  The deep neural network changes the number of channels to 20 ch, and similarly executes “convolution” processing (“Conv2”). In addition, the network construction unit 133 changes the number of channels to 16ch, and similarly executes “convolution” processing (“Conv3”). Furthermore, the network construction unit 133 executes “devolution” processing using a filter having a kernel size “3 × 3 × 3” three times with the number of channels of 16 channels (“DeConv1” to “DeConv3”).

  The deep neural network restores the three-dimensional shape information with the kernel size reduced to the original three-dimensional shape information with the kernel size. The network construction unit 133 executes “convolution” processing using a filter having a kernel size “1 × 1 × 1” with the number of channels of 16 channels (“Conv1” in the lower part of FIG. 4).

  The deep neural network outputs a numerical value that is a result of performing the “softmax” process in the channel direction for each channel. The output of the deep neural network is a joint information voxel set corresponding to the input shape information voxel set. The deep neural network outputs joint position information corresponding to joint part information from a channel corresponding to joint part information.

The kernel size of the output joint information voxel set is “8 × 8 × 8” similarly to the kernel size of the input shape information voxel set. The number of channels in the output layer of the deep neural network is a number representing the result of adding the estimated number of joint sites and the number of joint site information not assigned to any of the estimated joint sites. Here, the number of channels in the output layer of the deep neural network is 17 ch obtained by adding 1 ch to 16 ch representing the estimated number of joint parts (number of joint part information). The value included in each channel of one voxel represents the probability that the joint position information associated with that channel matches that voxel. The optimization method is, for example, “Adam”. The learning rate is, for example, 10 ⁻⁴ .

  The type of optimization method may be arbitrarily determined as various parameters. Various parameters such as kernel size, stride width, type of activation function, learning rate, and the like may be set to arbitrary values or functions. However, various parameters are determined so that the input shape information voxel set and the output joint information voxel set have the same voxel resolution.

  Returning to FIG. 3, the description of the configuration example of the learning unit 13 is continued. The parameter learning unit 134 acquires the shape information voxel set determined to be used for learning from the voxel set determination unit 132. The parameter learning unit 134 acquires the joint information voxel set determined to be used for learning from the voxel set determination unit 132. The parameter learning unit 134 acquires information representing the configuration of the constructed deep neural network from the network construction unit 133. The parameter learning unit 134 learns the parameters of the deep neural network using the constructed deep neural network.

  The parameter learning unit 134 uses the initial parameters of the deep neural network as parameters defined in the deep learning library “chainer”, for example. The parameter learning unit 134 executes, for example, 10,000 iterations of learning using the deep neural network constructed by the network construction unit 133. The initial parameters and the number of iterations are arbitrarily determined in advance. The parameter learning unit 134 outputs information representing the configuration and parameters of the learned deep neural network to the analysis unit 15.

  In this way, the parameter learning unit 134 can determine the parameters of the deep neural network for estimating the three-dimensional joint position information of the subject from the multi-viewpoint photographed image. Further, the parameter learning unit 134 repeatedly performs the “convolution” process and the pooling process on the three-dimensional shape information, so that the three-dimensional joint position information of the subject can be estimated from the multi-viewpoint photographed image.

  FIG. 5 is a diagram illustrating an example of the configuration of the input data generation unit 14. The input data generation unit 14 includes a silhouette generation unit 141 and a shape information generation unit 142. The silhouette generation unit 141 acquires a multi-view live-action image in which a subject as an object for which joint information is estimated is captured. The multi-view photographed image may be an image obtained by photographing the subject with any type of camera, but is, for example, an image obtained by photographing the subject with a color camera. The input data generation unit 14 may acquire the multi-viewpoint photographed image from a recording medium such as a hard disk drive.

  The silhouette generation unit 141 generates a silhouette image of a multi-view subject (hereinafter referred to as a “multi-view subject silhouette image”) by extracting a subject area from the multi-view live-action image. The silhouette generation unit 141 may generate a multi-view subject silhouette image using any method, but for example, generates a multi-view subject silhouette image using a background difference or graph cut method.

  The shape information generation unit 142 restores the three-dimensional shape information of the subject from the multi-view subject silhouette image based on the camera parameters of each viewpoint. That is, the shape information generation unit 142 restores the three-dimensional shape information of the subject based on the camera parameters of each viewpoint and the multi-view subject silhouette image. The shape information generation unit 142 restores the three-dimensional shape information of the subject from the multi-view subject silhouette image by executing a process similar to the process executed by the shape information restoration unit 123 of the learning data generation unit 12. For example, the shape information generation unit 142 restores the three-dimensional shape information of the subject from the multi-view subject silhouette image using the same parameters as the various parameters used by the shape information restoration unit 123. The shape information generation unit 142 outputs the three-dimensional shape information of the subject to the analysis unit 15.

  FIG. 6 is a diagram illustrating an example of the configuration of the analysis unit 15. The analysis unit 15 acquires the three-dimensional shape information of the subject from the shape information generation unit 142. The analysis unit 15 acquires information representing the configuration and parameters of the learned deep neural network from the parameter learning unit 134. The analysis unit 15 estimates the joint position information of the subject from the three-dimensional shape information of the subject by using the three-dimensional shape information of the subject as an input of the learned deep neural network.

  The analysis unit 15 generates the three-dimensional joint information of the subject by adding the joint part information associated with the channel of the output layer of the deep neural network to the joint position information. Accordingly, the analysis unit 15 can estimate the three-dimensional joint information of the subject from the multi-view subject silhouette image based on the camera parameters of each viewpoint. The analysis unit 15 outputs the three-dimensional joint information of the subject to the external device. When the external device is a recording medium, the analysis unit 15 may record the three-dimensional joint information of the subject on the external device.

Next, an example of the operation of the estimation device 10 will be described.
FIG. 7 is a flowchart illustrating an example of the operation of the estimation apparatus 10. The estimation apparatus 10 generates a multi-viewpoint model silhouette image (step S101). The estimation apparatus 10 restores the 3D shape information of the 3D model from the multi-viewpoint model silhouette image based on the camera parameters of each viewpoint (step S102). The estimation apparatus 10 determines the parameters of the deep neural network based on the 3D shape information of the 3D model and the 3D joint information of the 3D model (step S103).

  The estimation device 10 restores the three-dimensional shape information of the subject from the multi-view subject silhouette image (step S104). The estimation apparatus 10 estimates the three-dimensional joint information of the subject by using the three-dimensional shape information of the subject as an input of the learned deep neural network (step S105). The estimation device 10 outputs and records the three-dimensional joint information of the subject to the external device (step S106).

  FIG. 8 is a flowchart illustrating an example of the operation of the learning data generation unit 12. The learning data generation unit 12 acquires a three-dimensional model image to which joint information is assigned for each viewpoint (step S201). The learning data generation unit 12 generates a multi-view model silhouette image from the three-dimensional model image for each viewpoint, and outputs the multi-view model silhouette image and camera parameters (step S202).

  The learning data generation unit 12 restores the three-dimensional shape information of the three-dimensional model from the multi-view model silhouette image based on the camera parameters (step S203). The learning data generation unit 12 outputs the 3D shape information of the 3D model and the 3D joint information of the 3D model to the learning unit 13 (step S204).

  FIG. 9 is a flowchart illustrating an example of the operation of the learning unit 13. The learning unit 13 generates a shape information voxel set and a joint information voxel set (step S301). The learning unit 13 determines for each voxel set whether the voxel set is used for learning (step S302). The learning unit 13 constructs a deep neural network having the shape information voxel set as an input and the joint information voxel set as an output (step S303). The learning unit 13 learns the parameters of the deep neural network (step S304).

  FIG. 10 is a flowchart illustrating an example of the operation of the input data generation unit 14. The input data generation unit 14 acquires a multi-viewpoint live-action image in which the subject is imaged (step S401). The input data generation unit 14 generates a multi-view subject silhouette image from the multi-view live-action image (step S402). The input data generation unit 14 restores the three-dimensional shape information of the subject from the multi-view subject silhouette image based on the camera parameters of each viewpoint (step S403). The input data generation unit 14 outputs the three-dimensional shape information of the subject to the analysis unit 15 (step S404).

  FIG. 11 is a flowchart illustrating an example of the operation of the analysis unit 15. The analysis unit 15 estimates the three-dimensional joint information of the subject from the three-dimensional shape information of the subject using the learned deep neural network (step S501). The analysis unit 15 outputs and records the three-dimensional joint information of the subject to the external device (step S502).

  As described above, the learning data generation unit 12 as the learning data generation device of the first embodiment includes the silhouette image rendering unit 121, the camera parameter output unit 122, the shape information restoration unit 123, and the joint information voxelization unit 125. With. The silhouette image rendering unit 121 acquires a 3D model image for each viewpoint determined around the 3D model. The silhouette image rendering unit 121 generates a multi-view model silhouette image for each viewpoint by rendering the 3D model image. The camera parameter output unit 122 acquires camera parameters for each viewpoint. The shape information restoration unit 123 restores the three-dimensional shape information of the three-dimensional model from the multi-view model silhouette image based on the camera parameters of each viewpoint. The joint information voxelization unit 125 generates the 3D joint information of the 3D model in the same voxel space as the voxel space of the 3D shape information of the 3D model. Thereby, the learning data generation unit 12 as the learning data generation device of the first embodiment can more easily generate a plurality of learning data including three-dimensional joint information.

  The learning data generation unit 12 as the learning data generation device of the first embodiment may further include a learning unit 13. The learning unit 13 learns parameters of a deep neural network that outputs three-dimensional joint information according to three-dimensional shape information. The output layer of the deep neural network has a number of channels corresponding to the number of joints represented by the three-dimensional joint information. The learning unit 13 generates a shape information voxel set by collecting a plurality of three-dimensional shape information. The learning unit 13 determines whether or not the shape information voxel set is used for learning the parameters of the deep neural network based on a predetermined condition.

  The learning device 11 according to the first embodiment generates learning data based on a computer graphics image. Thereby, the learning device 11 according to the first embodiment can easily generate a large amount of learning data. The learning device 11 of the first embodiment can easily expand a large amount of learning data. Since the learning device 11 according to the first embodiment acquires an image to which joint information is given, it is possible to reduce the trouble of acquiring joint information from the image.

(Second Embodiment)
The second embodiment is different from the first embodiment in that the estimation device 10 stores various data related to machine learning. In the second embodiment, only differences from the first embodiment will be described.

  FIG. 12 is a diagram illustrating an example of the configuration of the estimation system 1b. The estimation system 1 b includes an estimation device 10. In the estimation system 1b, when the estimation device 10 executes only the three-dimensional joint information estimation process, the estimation system 1b may not include the learning device 11. That is, when the estimation device 10 executes only the estimation process in the estimation system 1b, the learning device 11 does not have to execute machine learning. When the learning device 11 performs machine learning in the estimation system 1a, the estimation system 1b includes the learning device 11.

  The learning device 11 includes a learning data generation unit 12 and a learning unit 13. The learning data generation unit 12 may be provided in a learning data generation device that is different from the learning device 11. In this case, the learning data generated by the learning data generation device may be given to the learning device 11 via a network or a storage medium. The learning data generation unit 12 as a learning data generation device may further include a learning unit 13.

  The learning unit 13 performs machine learning using the learning data. The learning unit 13 is not limited to the deep neural network, and may execute machine learning such as genetic programming or clustering, for example. The learning unit 13 records various data related to machine learning in the estimation device 10 via a communication line or a recording medium. The various types of data related to machine learning are, for example, data representing an algorithm, data representing a configuration and parameters of a learned deep neural network.

  The estimation device 10 includes an input data generation unit 14, an analysis unit 15, and a storage unit 16. The storage unit 16 stores various data related to machine learning generated by the learning unit 13. The analysis unit 15 acquires the learned deep neural network parameters from the storage unit 16. The learned deep neural network parameter is a parameter based on a learning result using a shape information voxel set that satisfies a predetermined condition among shape information voxel sets. The predetermined condition is, for example, a condition that the space of the shape portion of the three-dimensional model is a predetermined ratio or more with respect to the entire space of the shape information voxel set.

  As described above, the estimation device 10 according to the second embodiment includes the analysis unit 15. The analysis unit 15 uses the deep neural network learned to output the three-dimensional joint information of the three-dimensional model according to the three-dimensional shape information of the three-dimensional model generated by the learning device 11, and uses the deep neural network learned. Estimate the original joint information. Thereby, the estimation apparatus 10 of the second embodiment can accurately estimate the three-dimensional joint position of the subject. In addition, the learning data generation unit 12 as the learning data generation device of the second embodiment can more easily generate a plurality of learning data including three-dimensional joint information.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 1a, 1b ... Estimation system, 10 ... Estimation apparatus, 11 ... Learning apparatus, 12 ... Learning data generation part, 13 ... Learning part, 14 ... Input data generation part, 15 ... Analysis part, 16 ... Memory | storage part, 121 ... Silhouette image Rendering unit 122 ... Camera parameter output unit 123 ... Shape information restoration unit 124 ... Joint information output unit 125 ... Joint information voxelization unit 131 ... Voxel set generation unit 132 ... Voxel set determination unit 133 ... Network construction 134, parameter learning unit, 141 ... silhouette generation unit, 142 ... shape information generation unit

Claims (8)

By obtaining a three-dimensional model image, which is a computer graphics image representing a three-dimensional model of a subject having a joint, for each viewpoint determined around the three-dimensional model, and performing rendering processing on the three-dimensional model image A silhouette image rendering unit for generating a silhouette image of the three-dimensional model for each viewpoint;
A camera parameter section for acquiring camera parameters for each viewpoint;
Based on the camera parameters of each viewpoint, a shape information restoration unit that restores the three-dimensional shape information of the three-dimensional model from the silhouette image of each viewpoint;
A learning data generation device comprising: a joint information voxelization unit that generates three-dimensional joint information of the three-dimensional model in the same voxel space as the three-dimensional shape information voxel space of the three-dimensional model. A learning unit that learns parameters of a deep neural network that outputs the joint information according to the shape information;
The learning data generation apparatus according to claim 1, wherein an output layer of the deep neural network includes a number of channels corresponding to the number of the joints represented by the joint information. A deep neural network trained to output the three-dimensional joint information of the three-dimensional model according to the three-dimensional shape information of the three-dimensional model generated by the learning data generating device according to claim 1 or 2. An estimation apparatus comprising: an analysis unit that estimates the three-dimensional joint information of the subject related to the three-dimensional model by using the three-dimensional shape information of the subject related to the three-dimensional model as an input of the deep neural network. A learning unit that learns parameters of a deep neural network that outputs the joint information according to the shape information;
The learning unit generates the set of shape information by collecting a plurality of the shape information, and whether to use the set of shape information for deep neural network parameter learning based on a predetermined condition The learning data generation device according to claim 1, wherein the determination is made. A storage unit for storing a configuration and parameters of a deep neural network that outputs the joint information according to the shape information;
The parameter of the deep neural network is a parameter based on a learning result using the set of shape information that satisfies a predetermined condition among the set of shape information generated by collecting a plurality of the shape information. The estimation device according to claim 3. An estimation method executed by an estimation device,
A deep neural network trained to output the three-dimensional joint information of the three-dimensional model according to the three-dimensional shape information of the three-dimensional model generated by the learning data generating device according to claim 1 or 2. An estimation method comprising a step of estimating three-dimensional joint information of a subject related to the three-dimensional model using   A computer program for causing a computer to function as the learning data generation device according to claim 1, claim 2 or claim 4.   A computer program for causing a computer to function as the estimation device according to claim 3 or 5.

Images