JP6348623B2 - 3D model creation method, 3D model creation device, 3D model creation system, and 3D model creation program - Google Patents

Description

  The present invention relates to a technique for creating a three-dimensional polygon model that represents a whole human body using a computer, and a tertiary that can superimpose skeleton data and motion data on the three-dimensional polygon model to give a desired motion. The present invention relates to a technique for creating original models.

  In recent years, in games and advertisements, a technique for displaying a person represented by a three-dimensional polygon model created using a computer on a display or displaying a moving image with a motion added to the three-dimensional polygon model has been widely used. It is used (for example, patent document 1).

The three-dimensional polygon model is created by the following procedure.
First, using a distance image sensor arranged at a predetermined position, image data that is two-dimensional image data of a subject and a depth map representing a two-dimensional distribution of the distance from the distance image sensor to the subject are acquired. To do. Then, a large number of points each having a spatial coordinate can be generated from the plane coordinates corresponding to each pixel position of the imaging data and the distance information at each point of the depth map. At this time, points of distance corresponding to the background (wall or the like) in the depth map are excluded, and points constituting the contour of the subject are extracted. Subsequently, a triangle formed by connecting three adjacent points out of these point groups is sequentially generated by a technique such as Delaunay triangulation (see Non-Patent Document 1) to create a polygon.

  In the above method, since the distance image sensor is arranged at only one predetermined position, it is possible to create only a three-dimensional polygon model that can be seen from one specific direction (the direction from which the subject is viewed). When creating a 3D polygon model that can be viewed from an arbitrary viewpoint, the imaging data and depth map of the subject viewed from multiple directions are acquired, and one 3D polygon model formed in each direction is obtained. Need to be integrated into. Hereinafter, in this specification, a 3D polygon model created only from information viewed from one direction is a partial 3D polygon model, and a 3D polygon model obtained by integrating the partial 3D polygon model is an overall 3D polygon model. Call.

JP 2014-67372 A

“MathWorks: Delaunay triangulation”, [online], Fast Facts, [searched February 19, 2016], Internet <URL: http://jp.mathworks.com/help/matlab/math/delaunay-triangulation. html? requestedDomain = jp.mathworks.com>

  Conventionally, the operation of integrating a plurality of partial 3D polygon models into one overall 3D polygon model is performed by rotating or moving each partial 3D polygon model while an operator looks at the screen. This is done manually by deciding the coupling position and combining the data of the end of each partial 3D polygon model. In addition, even if partial 3D polygon models are integrated in this way, their connected parts often do not completely match, and the operator manually corrects the misalignment at the joint of each partial 3D polygon model. I went there.

In addition, the depth information obtained by the distance image sensor includes some errors. The points including the error in the distance information are located away from the contour of the subject, and when a plurality of such points are collected, another small polygon aggregate that is separated from the subject polygon is formed. In this manner, the operator manually deletes the polygon that is configured independently from the polygon of the subject.
As described above, conventionally, there has been a problem that it takes time and labor for the operator to perform complicated work to create the entire three-dimensional polygon model of the subject.

  The problem to be solved by the present invention is to provide a technique for easily creating a three-dimensional model of a subject that can be viewed from an arbitrary direction.

A method for creating a three-dimensional model according to the present invention made to solve the above problems is as follows.
a) Image data that is two-dimensional image data of a subject and depth data representing a two-dimensional distribution of the distance from the distance image sensor to the subject by each of four or more distance image sensors arranged so as to surround the subject. A process of obtaining
b) generating spatial coordinates of a plurality of points from the imaging data and the depth data for each of the four or more range image sensors;
c) A partial three-dimensional combination of a plurality of polygons created by connecting a predetermined number of points located on the same plane among the plurality of points for each of the four or more distance image sensors. Creating a polygon model;
d) Spatial coordinates corresponding to one common spatial coordinate system obtained by converting the spatial coordinates of a plurality of points constituting each partial three-dimensional polygon model based on the positional relationship between the corresponding distance image sensor and the subject. Converting the plurality of partial three-dimensional polygon models into one and creating a whole three-dimensional polygon model;
e) deleting the polygon aggregate formed by connecting a predetermined number or less of polygons in the entire three-dimensional polygon model.

  The polygon is typically a triangle formed by connecting three points, which is the minimum number required to define one surface, but a quadrangle formed by connecting four points or more points. It may be a polygon formed by connecting. As a method for creating a partial three-dimensional polygon model from a plurality of points, for example, the Delaunay triangulation method described in Non-Patent Document 1 can be used.

  In the three-dimensional model creation method according to the present invention, four or more distance image sensors are arranged so as to surround the subject. Then, a set of imaging data and depth data is acquired by each distance image sensor, and a partial three-dimensional polygon model is created by combining a plurality of polygons connecting a plurality of points acquired from each set. Then, the partial 3D polygon model is converted into a converted partial 3D polygon model corresponding to one common space coordinate system by coordinate conversion, and these are combined and integrated into the entire 3D polygon model. That is, it is not necessary for the operator to rotate and move the partial 3D polygon model or to combine the end portion data as in the prior art, and a 3D model composed of the entire 3D polygon model can be created easily.

  Further, in the three-dimensional model creation method according to the present invention, a polygon aggregate composed of a predetermined number or less of polygons is mechanically deleted from the entire three-dimensional polygon model. A polygon aggregate composed of a plurality of points including errors in distance information in the depth map is located away from a large number of polygons constituting the subject, and the number of polygons configured is small. In the 3D model creation method according to the present invention, such polygon aggregates are mechanically deleted based on the number of components, so that the operator individually confirms polygons including points including errors in distance information as in the past. Thus, it is not necessary to delete manually, and a 3D model composed of the entire 3D polygon model can be easily created.

In the method of creating a three-dimensional model according to the present invention,
f) In each of the partial three-dimensional polygon models, the angle formed by the line-of-sight vector from the distance image sensor to the center point of the polygon and the normal vector of the polygon among the plurality of polygons is 90 degrees or less. Preferably, the method includes a step of deleting polygons.

  Some commercially available range image sensors have a function to estimate not only the points that can be directly captured in the field of view of the range image sensor but also the points that are located behind the subject and automatically add the estimated points. There is. However, the accuracy of the estimated points added in this way is not necessarily high. Further, in the method for creating a three-dimensional model according to the present invention, since the point of the subject can be obtained by four or more distance image sensors arranged so as to surround the subject, the estimation automatically added by the distance image sensor A point is not necessary. In the method of the above aspect, the polygon is deleted by utilizing the fact that the normal formed by the polygon including the estimated point and the line-of-sight vector from the distance image sensor toward the center of the polygon is 90 degrees or less. In addition, the accuracy of the partial 3D polygon can be increased.

Furthermore, in the method for creating a three-dimensional model according to the present invention,
g) Static 3D combining the entire 3D polygon model with skeleton data prepared in advance and including a plurality of bones represented by straight lines and a plurality of joints represented by points and each connecting the bones Creating a model;
h) setting a weight value according to the distance between the point and the bone for each point of the stationary three-dimensional model;
i) superimposing motion data expressing a predetermined motion by motion of the bone and the joint on the data of the stationary three-dimensional model, and moving the point based on the motion data and the weight value;
Can be included.

  In the three-dimensional model creation method of the above aspect, the motion data can be applied to the stationary three-dimensional model in which the skeleton data is combined to create a three-dimensional model that can perform any motion.

In addition, a three-dimensional model creation device according to the present invention made to solve the above problems,
a) Imaging data that is two-dimensional image data of a subject acquired by each of four or more distance image sensors arranged so as to surround the subject, and a two-dimensional distribution of distances from the distance image sensor to the subject A data receiving unit for receiving input of depth data representing
b) a point coordinate generation unit that generates spatial coordinates of a plurality of points for each set of the imaging data and the depth data;
c) A partial 3D polygon model creation unit for creating a partial 3D polygon model made up of a combination of a plurality of polygons, each created by connecting a predetermined number of points located on the same plane among the plurality of points When,
d) Spatial coordinates corresponding to one common spatial coordinate system obtained by converting the spatial coordinates of a plurality of points constituting each partial three-dimensional polygon model based on the positional relationship between the corresponding distance image sensor and the subject. An overall 3D polygon model creation unit that creates an overall 3D polygon model in which a plurality of the partial 3D polygon models are integrated into one,
e) In the overall three-dimensional polygon model, a polygon aggregate deletion unit that deletes a polygon aggregate composed of a predetermined number of polygons or less, and
It is characterized by providing.

Furthermore, a system for creating a three-dimensional model according to the present invention, which has been made to solve the above problems,
a) Acquire imaging data that is arranged so as to surround a predetermined subject position, each of which is two-dimensional image data of the subject, and depth data representing a two-dimensional distribution of the distance from the distance image sensor to the subject. 4 or more range image sensors that are possible,
b) a data acquisition unit for acquiring imaging data and depth data of the subject by each of the four or more distance image sensors in accordance with a predetermined input operation by a user;
c) a vertex coordinate generation unit that generates spatial coordinates of a plurality of points for each set of the imaging data and the depth data acquired by each of the four or more distance image sensors;
d) For each of the four or more distance image sensors, a partial three-dimensional combination composed of a combination of a plurality of polygons created by connecting a predetermined number of points located on the same plane among the plurality of points. A partial 3D polygon model creation unit for creating a polygon model;
e) Spatial coordinates corresponding to one common space coordinate system by converting the spatial coordinates of a plurality of points constituting each partial three-dimensional polygon model into coordinates based on the positional relationship between the corresponding distance image sensor and the subject. An overall 3D polygon model creation unit that creates an overall 3D polygon model in which a plurality of the partial 3D polygon models are integrated into one,
f) In the overall three-dimensional polygon model, a polygon aggregate deleting unit that deletes a polygon aggregate composed of a predetermined number of polygons or less, and
It is characterized by providing.

  By using the 3D model creation technique according to the present invention, it is possible to easily create a 3D model of a subject that can be viewed from any direction.

1 is a schematic configuration diagram of an embodiment of a 3D model creation system (including a 3D model creation apparatus) according to the present invention. The flowchart of one Example of the preparation method of the three-dimensional model which concerns on this invention. The schematic block diagram of another Example of the production system (a 3D model production apparatus is included) of the three-dimensional model which concerns on this invention. An example of the AR marker used in Example 2. FIG.

  Embodiments of a 3D model creation method, a creation apparatus, and a creation system according to the present invention will be described below with reference to the drawings. In this embodiment, a three-dimensional polygon model is created by computer processing data obtained by photographing a human (subject) with a distance image sensor, and an arbitrary motion is given to the three-dimensional polygon model by superimposing motion data. This is a technique for creating a three-dimensional model.

  FIG. 1 is a schematic configuration diagram of a three-dimensional model creation system 1 (hereinafter also simply referred to as “creation system”) in the present embodiment. FIG. 2 is a flowchart of a 3D model creation method (hereinafter also simply referred to as “creation method”) in the present embodiment.

  The creation system 1 of the present embodiment is arranged so as to surround the subject position P in the imaging space 10 and operates under the control of the imaging control unit 16, and the three distance image sensors 11 to 14, and the three-dimensional model creation device 20 (corresponding to an embodiment of the three-dimensional model creation apparatus according to the present invention. Hereinafter, it is also simply referred to as “creation apparatus”). Three of the four distance image sensors 11 to 14 (11 to 13) are arranged horizontally at equal intervals so as to capture the subject from the horizontal direction, and one (14) is the subject from above the subject. It is arranged to capture. In the field of view of the three distance image sensors 11 to 13 that capture the subject from the horizontal direction, the subject and a predetermined background 11a to 13a (for example, a single-color cloth or wall) are captured. In addition, the subject and the floor are captured in the field of view of the distance image sensor 14 that captures the subject from vertically above. Each distance image sensor 11-14 is arrange | positioned so that another distance image sensor may not enter into the visual field. Although the distance image sensors 11 to 14 of this embodiment are Kinect (registered trademark of MICROSOFT CORPORATION, product name), other sensors may be used as long as they can acquire image data and depth data. .

  In addition to the storage unit 21, the creation device 20 includes a data reception unit 22, a point coordinate generation unit 23, a partial 3D polygon model creation unit 24, a polygon reduction unit 25, a backside polygon deletion unit 26, a texture setting unit 27, an overall tertiary. An original polygon model creation unit 28, a noise deletion unit 29, an overlapping point arrangement unit 30, an overall three-dimensional polygon model correction unit 31, a bone data generation unit 32, a weight setting unit 33, and a motion data superimposition unit 34 are provided as functional blocks. Yes. The entity of the creation device 20 is a general personal computer, and an input unit 40 and a display unit 50 are connected to it. The functional blocks 22 to 34 can be realized by causing a CPU to execute a 3D model creation program installed in a computer. The operation of each part will be described later. The storage unit 21 stores a coordinate transformation matrix created by the procedure described below and motion data described later.

The coordinate transformation matrix stored in the storage unit 21 is created as follows.
Prior to creating the three-dimensional model of the subject, the positions and fields of view of the four distance image sensors 11 to 14 are fixed. Then, a plurality of balls (at least three balls not located on the same straight line) are arranged as subjects, and the plurality of balls are photographed by the distance image sensors 11 to 14, and the imaging data and depth data of the plurality of balls are captured. To get. Here, the imaging data is two-dimensional color image data (data in which each pixel arranged two-dimensionally has color information), and the depth data is data representing a two-dimensional distribution of distance from the distance image sensor. is there.

  When the imaging data and the depth data of the plurality of balls are acquired for each of the distance image sensors 11 to 14, the spatial coordinates of the plurality of balls viewed from the distance image sensors 11 to 14 are obtained from the data sets. It is done. The three axes defining the spatial coordinates required at this time are different for each of the four distance image sensors 11-14. Subsequently, spatial coordinates obtained from different distance image sensors for the same ball are compared with each other, thereby obtaining a three-axis relationship between the four distance image sensors. A conversion matrix for converting into one common spatial coordinate system is created for each distance image sensor. As one common spatial coordinate system, one spatial coordinate system of the four distance image sensors can be used as it is.

Hereinafter, a method for creating a three-dimensional model in this embodiment will be described.
When the user gives an instruction to create a 3D model by a predetermined input (for example, an input operation of pressing the “start” button on the screen through the input unit), the imaging control unit 16 gives the user a predetermined pose at the subject position. Instruct them to stand. Here, the predetermined pose is a pose in which the area that becomes the blind spot of all the distance image sensors 11 to 14 is reduced as much as possible at the time of photographing. For example, the posture (T Posture).

  The imaging control unit 16 operates all the distance image sensors 11 to 14 at the same time after elapse of a certain time (time required for the user to stand at the subject position in the pose) after giving the instruction to the user. Imaging data and depth data are acquired for (step S1). The imaging data and depth data acquired by each of the distance image sensors 11 to 14 are transmitted to the 3D model creation apparatus 20 as a set of data. The Kinect used as the distance image sensors 11 to 14 in the present embodiment compares the contour of the subject with the bone database owned by Kinect itself and matches the bone data (representing a human by a plurality of bones and joints connecting the bones). In addition to the imaging data and the depth data, the subject bone data can be obtained at the same time (step S2). This bone data is also transmitted to the three-dimensional model creation device 20 at the same time.

  The data receiving unit 22 receives imaging data and depth data and bone data sent from the distance image sensors 11 to 14 and stores them in the storage unit 21. When the imaging data and the depth data are stored in the storage unit 21, the point coordinate generation unit 23, for each of the distance image sensors 11 to 14, is a plane coordinate corresponding to each pixel position of the imaging data and the color information that each pixel has. Then, from the distance information at each point of the depth data, a large number of points (point clouds) each having spatial coordinates and color information are generated (step S3).

  Subsequently, the partial three-dimensional polygon model creation unit 24 calculates three adjacent points according to a predetermined rule (for example, a rule based on the Delaunay triangulation method) from a plurality of points having spatial coordinates generated for each of the distance image sensors 11 to 14. Connected triangles (polygons) are sequentially generated, and a partial three-dimensional polygon model is created from the plurality of points (step S4). Here, the partial three-dimensional polygon model is a three-dimensional polygon model in which the subject is viewed from one specific direction (that is, a corresponding distance image sensor). At this time, the color of each surface of the polygon is an average color of the three colors (color based on the color information) constituting the surface. Although a triangle formed by connecting three points is a polygon here, a polygon may be created by connecting four or more points.

  When a partial 3D polygon model is created for each of the distance image sensors 11 to 14, the polygon reduction unit 25 converts each partial 3D polygon model from a high polygon (polygon created from a large number of points) to a low polygon (high polygon). Polygon reduction converted to polygons created with fewer points). In this conversion, an operation for converting each polygon into one point located at the center of gravity of the polygon (the color of the converted point becomes the color of the polygon) is performed for all the polygons, and a new point created thereby Are connected by the Delaunay triangulation method or the like to generate a new polygon (the color of this polygon is the average color of the points constituting the polygon), and a new partial three-dimensional polygon model (low polygon) is created. (Step S5). Since the number of polygons is reduced by this processing, the processing load in each process described later is reduced and high-speed processing is realized. In this embodiment, since a personal computer is used as the 3D model creation apparatus 20, polygon reduction is performed for the purpose of reducing the processing load. However, a high-end computer capable of high-speed processing can be created as a 3D model. When used as the apparatus 20, the following steps may be performed without performing polygon reduction (that is, with a high polygon).

  Next, for each partial 3D polygon model, the back polygon deletion unit 26 calculates the angle formed by the normals of all the polygons constituting the partial 3D polygon model and the line-of-sight vectors of the corresponding distance image sensors 11 to 14. Then, the polygon whose angle is 90 degrees or less is regarded as an unnecessary polygon and is deleted (step S6). This is because the angle between the normal line of the polygon that forms the back side of the subject and the line-of-sight vector of the distance image sensors 11 to 14 toward the center of the polygon, which cannot be directly captured in the visual field of the distance image sensor, is 90. It is a process that uses the fact that it becomes less than the degree. Kinect used in this embodiment has a function of generating an estimated point on the back side of a subject that cannot be directly captured in the visual field, but the accuracy is not necessarily high. Further, in the three-dimensional model creation method of this embodiment, partial three-dimensional polygon models viewed from different directions are created by a plurality of Kinects, so there is no need to use polygons formed by connecting these estimated points. Therefore, in this embodiment, unnecessary polygons are deleted by the above method. Note that this step is not necessary when using a distance image sensor that does not have a function of generating an estimated point on the back side of the subject.

  At this stage, each of the polygons constituting the partial three-dimensional polygon model is expressed by one color, and the color difference from the adjacent polygon is conspicuous. Therefore, the texture setting unit 27 performs processing (texture pasting processing) for pasting the imaging data acquired by the corresponding distance image sensors 11 to 14 to each partial three-dimensional polygon model (step S7). This process is performed by projecting the imaging data onto each partial three-dimensional polygon model. Each partial three-dimensional polygon model does not match the number of pixels constituting the pixel data because the number of points is reduced by the polygon reduction. In other words, the pixel position and the point position do not match when the pixel data is projected. However, these do not necessarily match in the projection processing. Rather, by not matching, it becomes possible to express a line segment or a polygon connecting the points with a plurality of different colors. The discrepancy between the points is, for example, that the resolution of the pixel data and the resolution of the depth data are not the same (for example, the resolution of the depth data is lower), and a plurality of points are generated in accordance with the lower resolution data in step S3 described above. It also happens if you do.

  When the pasting of the texture to the partial 3D polygon model is finished, the overall 3D polygon model creation unit 28 reads out the coordinate transformation matrices corresponding to the distance image sensors 11 to 14 from the storage unit 21. Then, the spatial coordinates of each point constituting all the partial 3D polygon models are converted into the spatial coordinates in the common spatial coordinate system by the corresponding coordinate conversion matrix. As a result, all the partial 3D polygon models are arranged in one spatial coordinate system, the partial 3D polygon models are automatically integrated, and an overall 3D polygon model is created (step S8). Therefore, unlike the conventional case, it is not necessary for the operator to rotate and move each partial 3D polygon model to determine the coupling position or to connect end data at the coupling position. An overall 3D polygon model can be created.

  Next, the noise deleting unit 29 deletes a polygon aggregate composed of a predetermined number or less of polygons in the created overall three-dimensional polygon model (deletes a polygon aggregate that is a noise) (step S9). . The depth data obtained from the distance image sensors 11 to 14 often includes some errors in distance information. A polygon aggregate composed of a plurality of points including such errors floats in a space apart from a large number of polygons constituting the subject, and the number of polygons constituting the aggregate is small. In the 3D model creation method according to the present embodiment, such a polygon aggregate is mechanically deleted based on the number of polygons that constitute the polygon. There is no need to confirm and manually delete the entire three-dimensional polygon model.

  Furthermore, the overlapping point arrangement unit 30 arranges grids at predetermined intervals in each of the three axial directions of the space where the entire three-dimensional polygon model is located, and divides the space into small cubes. This predetermined interval is appropriately determined according to the number of points constituting the entire three-dimensional polygon model. Then, the number of points existing in each cube is confirmed, and when there are a plurality of points in one cube, they are collected into one point located at their average coordinates (step S10). This is a process of deleting an overlapping point existing in a combined region of adjacent partial 3D polygon models when an entire 3D polygon model is created from the partial 3D polygon model. By deleting such overlapping points, it is possible to eliminate an uncomfortable appearance such as the presence of overlapping polygons, and it is possible to avoid an increase in processing load due to an unnecessarily large number of points.

  Next, the overall 3D polygon model correcting unit 31 checks whether or not there is a hole in the overall 3D polygon model created in the above step, and performs processing to fill in the hole if it exists (step S11). . This process is performed by generating an appropriate estimated point inside the hole from a point located in the vicinity of the hole, and creating a new polygon using the generated point. Such a filling process is conventionally known as a function of, for example, Photoshop (registered trademark of Adobe Systems Incorporated, product name). However, when this hole filling process is performed by software such as Photoshop, a large number of points are newly generated to fill the hole, and the load of the subsequent processing may increase. In such a case, it is preferable to reduce the number of points as appropriate by the polygon reduction described above to reduce the load in the subsequent processing. In software such as Photoshop, texture information may be deleted in the hole filling process. In such a case, it is necessary to perform the above-described texture pasting process again at an appropriate stage thereafter (step S12).

Through the above steps, the entire three-dimensional polygon model is created.
As described above, in the 3D polygon model creation method, creation system, or creation apparatus of this embodiment, a partial 3D polygon model corresponding to each of a plurality of distance image sensors is created, and the partial 3D polygon is converted by coordinate conversion. The model is converted into a converted partial 3D polygon model corresponding to one common space coordinate system, and these are combined and integrated into the entire 3D polygon model. That is, the partial three-dimensional polygon model can be automatically integrated by coordinate conversion to create an entire three-dimensional polygon model. Therefore, there is no need for the operator to rotate and move the partial 3D polygon model or to combine the edge data as in the prior art, and a 3D model consisting of the entire 3D polygon model can be easily created.

  In addition, a polygon aggregate composed of a predetermined number or less of polygons is mechanically deleted from the entire three-dimensional polygon model. For this reason, it is not necessary for the operator to individually check polygons including points including errors in distance information and delete them manually, as in the prior art, and an entire three-dimensional polygon model can be created easily.

Next, a procedure for creating a three-dimensional model by giving a motion to the entire three-dimensional polygon model will be described.
First, the bone data generation unit 32 reads out the bone data generated in step S2 and stored in the storage unit 21, and superimposes it on the entire three-dimensional polygon model. Thereby, a stationary three-dimensional model is obtained. Then, the weight setting unit 33 sets a weight value corresponding to the distance from each point of the overall three-dimensional polygon model to the bone (step S13). This weight value is a value representing how much the movement of the bone affects the movement of each point, and is set to a larger value as the distance to the bone is shorter, and a smaller value as the distance is increased. In this embodiment, since Kinect is used as the distance image sensor, bone data is acquired and stored in the storage unit 21 when the subject is photographed. However, when a distance image sensor that does not have a function of generating bone data is used. The bone data generation unit 32 can generate new bone data by collating the outline of the entire three-dimensional polygon model with an appropriate database in which bone data corresponding to various human poses is stored.

  Next, the motion data superimposing unit 34 reads all the motion data stored in the storage unit 21 in advance, displays the name and operation example on the screen, and prompts the user to select. The motion data is data that defines the motion of bones included in the bone data, and various motions can be expressed thereby. When the user selects one motion data, the motion data superimposing unit 34 superimposes the motion data on the stationary three-dimensional model (step S14). The motion data itself is data that moves the bone, but as the bone is moved, each point of the stationary 3D model moves with a size corresponding to the above-mentioned weight value. As a result, the entire 3D polygon model Can be displayed on the screen as if performing a predetermined operation.

Next, an example of using the three-dimensional model created in this way will be described.
In this example, participants interact with each other through their own 3D model in a virtual reality (VR) space created by 3D computer graphics (3DCG) prepared in advance by the event organizer. Is possible.

  First, event participants create their own three-dimensional models by the method of the above embodiment. Then, the data is transferred to the virtual reality space prepared by the event organizer. Thereby, the three-dimensional model of each participant is taken into the virtual reality space. Each participant wears an input device for inputting his / her body movement and a head mounted display (HMD). An example of an input device for body movement is a treadmill walking device. In this method, motion of each part of the participant's body is detected by vibration sensors attached to various parts of the device, and these are integrated to generate motion data. Or you may produce | generate motion data by detecting a participant's physical motion (skeletal gesture detection) using motion sensors, such as Kinect mentioned above. The head mounted display includes a screen on which a virtual reality space is displayed, headphones through which sound in the virtual reality space flows, and a microphone that recognizes the voice of the participant.

  When a participant moves his / her body in response to visual information or auditory information in the virtual reality space acquired from the head-mounted display, the motion data is reflected as the movement of the three-dimensional model in the virtual reality space. Then, when the three-dimensional models captured in the virtual reality space approach each other to a predetermined distance, a conversation is possible between them, and exchange between participants is realized. These events do not necessarily have to be held in a special event venue, and each participant can enjoy exchanges and simulated experiences in a virtual reality space via a device connected via a network. .

  Next, the 3D model creation system 100 of Example 2 in which the creation system 1 is improved based on the knowledge obtained by creating the 3D model using the 3D model creation system 1 of Example 1 will be described. In the second embodiment, the number and arrangement of the distance image sensors and the processing related to the acquisition of the coordinate transformation matrix are different from those in the first embodiment.

  FIG. 3 is a schematic configuration diagram of a 3D model creation system 100 (hereinafter also simply referred to as “creation system”) according to the second embodiment. Constituent elements common to those in FIG. 1 are denoted by common reference numerals in the last two digits, and description thereof will be omitted as appropriate. Note that the processing in each step from photographing the subject (step S1) to superimposing motion data (step S14) is the same as that in the first embodiment (FIG. 2), and thus the description thereof is omitted.

  In the creation system 1 of the first embodiment, four distance image sensors 11 to 14 are used so as to surround the subject position P, but in the creation system 100 of the second embodiment, twelve distance image sensors 1111 to 1112 are used. The distance image sensors 1111 to 1122 are a set of two, and are arranged at six locations surrounding the subject position P in the imaging space 110. The distance image sensors 1111 and 1112 are arranged in front of the subject position P, and the distance image sensors 1117 and 1118 are arranged in the back of the subject position P. Among a set of two distance image sensors, an odd number in the figure (1111 etc.) catches the subject from above, and an even number in the figure (1112 etc.) catches the subject from below. As shown in FIG. The distance image sensors 1111 to 1122 of the second embodiment are Kinect as in the first embodiment, but other sensors may be used as long as they can acquire imaging data and depth data.

  In the second embodiment, the bone data of the subject is acquired by the distance image sensor 1111 (distance image sensor located above the subject front) and 1118 (distance image sensor located below the subject back). In addition, the imaging data and depth data of the subject are acquired by the distance image sensors 1111 to 1122. In the second embodiment, more distance image sensors 1111 to 1122 are used than in the first embodiment, and they are arranged closer to the subject position P than the distance image sensors 11 to 14 in the first embodiment. get. As a result, the range of the subject captured by each distance image sensor is narrower than that of the first embodiment, and the resolution (positional resolution) of the imaging data is increased. Further, since the distance from each of the distance image sensors 1111 to 1122 to the subject is shorter than that in the first embodiment, the accuracy of the depth data is also improved. Furthermore, the imaging space 110 can be made more compact than in the first embodiment.

  In the first embodiment, the image data and the depth data are acquired in a posture where the subject stands with both hands horizontally spread and the legs widened to the shoulder width (posture called T-pose). In the second embodiment, the subject holds both hands. Imaging data and depth data are acquired in a posture that is inclined downward (a posture called an A pose). By increasing the number of distance image sensors and changing the posture of the subject to the A pose, the blind spot is reduced more than in the first embodiment, and the accuracy of the depth data is increased.

  In addition to the storage unit 121, the creation device 120 includes a data reception unit 122, a point coordinate generation unit 123, a partial 3D polygon model creation unit 124, a polygon reduction unit 125, a backside polygon deletion unit 126, a texture setting unit 127, an overall tertiary. An original polygon model creating unit 128, a noise deleting unit 129, an overlapping point organizing unit 130, an overall 3D polygon model correcting unit 131, a bone data generating unit 132, a weight setting unit 133, and a motion data superimposing unit 134 are provided as functional blocks. Yes. The entity of the creation device 120 is a general personal computer, to which an input unit 140 and a display unit 150 are connected. Similarly to the first embodiment, each of the functional blocks 122 to 134 can be realized by causing a CPU to execute a three-dimensional model creation program installed in a computer. The storage unit 121 stores a first coordinate transformation matrix, a second coordinate transformation matrix, and motion data that are created by the procedure described below.

The first coordinate transformation matrix and the second coordinate transformation matrix are created as follows.
Prior to creating the three-dimensional model of the subject, the positions and fields of view of the twelve distance image sensors 1111 to 1122 are fixed. In the first embodiment, a plurality of balls are used as subjects, but in the second embodiment, a thin plate 200 having different AR markers printed on the front and back surfaces is used. For example, the AR marker shown in FIG. 4A is attached to the front surface, the AR marker shown in FIG. 4B is attached to the back surface, and the four corners A to D are attached so that the positions of the front and back surfaces coincide with each other. .

  The user moves in the imaging space 110 while holding the thin plate 200 in the upright position (position where the AR marker is not attached). In the meantime, the twelve distance image sensors 1111 to 1122 are caused to automatically detect the four corners of the AR marker at predetermined time intervals (for example, every 5 seconds). For example, at a certain time, the distance image sensors 1111 to 1116 capture the surface of the thin plate 200 (AR marker in FIG. 4A), while the distance image sensors 1117 to 1122 are on the back surface of the thin plate 200 (AR in FIG. 4B). Marker). When automatic detection is completed a predetermined number of times (for example, 150 times), the shape (outer shape) of the outer frame of the AR marker is confirmed based on the positions A to D of the four corners of the AR marker acquired at each time point. If the positions of the four corners of the AR marker are acquired correctly, this outline is a rectangle (square), but the infrared rays emitted by Kinect interfere with the infrared rays emitted by another Kinect, resulting in errors in depth data, and image processing. As a result, the outer shape of the AR marker is distorted. Therefore, data in which the angle of the corner of the outer shape of the AR marker deviates from 90 degrees (for example, data in which a distortion of ± 5 degrees or more is generated) is deleted, and only data for which correct position information is obtained is the first. Used to create a coordinate transformation matrix. Then, by collating the spatial coordinates of the four corners A to D of the AR marker acquired by a plurality of the 12 distance image sensors 1111 to 1122 at the same time, the spatial coordinates of the 12 distance image sensors 1111 to 1122 A first coordinate transformation matrix is obtained for transforming the system to a common spatial coordinate system (for example, transforming the spatial coordinate system of the distance image sensors 1112 to 1122 to the spatial coordinate system of the distance image sensor 1111).

  As described in the first embodiment, in this three-dimensional model creation system, a point cloud of a subject is acquired by each distance image sensor, then a partial three-dimensional polygon model is created from the point cloud, and then a coordinate transformation matrix is obtained. By using this, the spatial coordinate system of the partial 3D polygon model is converted into a common spatial coordinate system to create an entire 3D polygon model. Therefore, it is necessary to use the coordinate transformation matrix created in the spatial coordinate system of the polygon model for the transformation of the spatial coordinate system of the partial 3D polygon model (creation of the entire 3D polygon model).

  However, when the present inventor confirmed using the creation system 1 of the first embodiment, when Kinect is used as the distance image sensor, the point cloud as raw data and the polygon data generated from the point cloud are coordinate axes. I found that there was a gap. Therefore, in order to correct this coordinate axis shift, the second coordinate transformation matrix is used in the second embodiment.

  For the distance image sensors 1111 to 1122, the AR marker is photographed one by one to acquire a point cloud, and polygon data is created from the point cloud. Then, by collating the spatial coordinates of the four corners A to D of the AR marker on the point cloud with the spatial coordinates of the four corners A to D of the AR marker in the polygon data, the coordinates of the point cloud are converted into the coordinates of the polygon data. A second coordinate transformation matrix is created. This second coordinate transformation matrix is created for each of the distance image sensors 1111 to 1122 and stored in the storage unit 121. Note that when creating the second coordinate transformation matrix, only one distance image sensor actually acquires data, but it is preferable to operate other distance image sensors. Since all the distance image sensors are operated when acquiring subject data, it is inevitable that a part of infrared rays emitted from the distance image sensors interfere with each other. Therefore, by creating a coordinate transformation matrix from data obtained under the same circumstances as this, a highly accurate second coordinate transformation matrix can be obtained that takes into account the situation where a part of infrared rays interfered.

  In step S8 of the first embodiment, the coordinate transformation matrix corresponding to the distance image sensors 11 to 14 is read from the storage unit 21, and the space of each point constituting the partial 3D polygon model created by each distance image sensor 11 to 14 is used. An overall three-dimensional polygon model was created by converting the coordinates into spatial coordinates in a common spatial coordinate system. In the second embodiment, the second coordinate transformation matrix for transforming the coordinate axis of the point cloud into the coordinate axis of the polygon data is calculated on the first coordinate transformation matrix created in the spatial coordinate system of the point cloud to calculate the spatial coordinate system of the polygon data. Convert to stuff. Thus, even when using a distance image sensor such as Kinect in which there is a deviation between the coordinate axes of the point cloud data and the polygon data, the deviation can be corrected. Accordingly, the spatial coordinate system of the partial three-dimensional polygon model can be more accurately converted to a common spatial coordinate system, and an entire three-dimensional polygon model without deviation can be created.

The above two embodiments are only examples, and can be appropriately changed in accordance with the gist of the present invention.
In the above embodiment, the 3D model creation system 1 captures the subject on the spot, acquires the imaging data and the depth data, and transmits them to the 3D model creation apparatus 20. It is also possible to input the depth data together with the arrangement data of the distance image sensor that has acquired the data to the three-dimensional model creating apparatus 20 and create a three-dimensional model by the same steps as described above.
The configuration described in each independent claim is an essential component in the present invention. Each step and each component described in the above embodiment are examples, and it is not always necessary to include all the steps and components described above in order to embody the present invention.
Furthermore, it is not always necessary to configure the three-dimensional model creation apparatus 20 with a single computer. For example, the functional blocks of the three-dimensional model creation apparatus 20 may be distributed and arranged in a plurality of computers connected via a network so as to cooperate with each other.

  The three-dimensional model creation system 1 of the above embodiment can be used by being placed in a game center, a commercial facility, or the like, for example, like a photo booth (registered trademark of Sega Games Inc.). In this case, for example, when a predetermined usage fee is input from the user, a 3D model creation instruction is given, and the entire 3D polygon model, bone data, and motion data are written through the above-described steps. A medium (for example, a DVD-ROM) can be provided to the user, or the data can be transferred to a portable terminal such as a user's smartphone or personal computer. An application that reads 3D model data into these portable terminals and the user edits their 3D model to create an avatar, and controls the main character consisting of the 3D model to play against other users By installing software for various applications, such as a network communication game application or an application for enjoying online shopping by changing clothes selected by the user to the 3D model, the user's own 3D model Can be enjoyed in various forms.

DESCRIPTION OF SYMBOLS 1,100 ... Three-dimensional model creation system 10, 110 ... Imaging space 11-14, 1111-1122 ... Distance image sensor P ... Subject position 16, 116 ... Shooting control part 20, 120 ... Three-dimensional model creation apparatus 21, 121 ... Storage unit 22, 122 ... Data reception unit 23, 123 ... Point coordinate generation unit 24, 124 ... Partial three-dimensional polygon model creation unit 25, 125 ... Polygon reduction unit 26, 126 ... Back polygon deletion unit 27, 127 ... Texture setting unit 28, 128 ... Overall 3D polygon model creation unit 29, 129 ... Noise deletion unit 30, 130 ... Duplicate point arrangement unit 31, 131 ... Overall 3D polygon model correction unit 32, 132 ... Bone data generation unit 33, 133 ... Weight Setting unit 34, 134 ... Motion data superimposing unit 35, 135 ... Motion data Data superimposing section 40, 140 ... input unit 50, 150 ... display unit 160 ... struts 200 ... sheet

Claims (11)

a) Image data that is two-dimensional image data of a subject and depth data representing a two-dimensional distribution of the distance from the distance image sensor to the subject by each of four or more distance image sensors arranged so as to surround the subject. A process of obtaining
b) generating spatial coordinates of a plurality of points from the imaging data and the depth data for each of the four or more range image sensors;
c) A partial three-dimensional combination of a plurality of polygons created by connecting a predetermined number of points located on the same plane among the plurality of points for each of the four or more distance image sensors. Creating a polygon model;
d) Spatial coordinates corresponding to one common spatial coordinate system obtained by converting the spatial coordinates of a plurality of points constituting each partial three-dimensional polygon model based on the positional relationship between the corresponding distance image sensor and the subject. Converting the plurality of partial three-dimensional polygon models into one and creating a whole three-dimensional polygon model;
e) a method of creating a three-dimensional model, comprising: a step of deleting a polygon aggregate formed by connecting a predetermined number or less of polygons in the whole three-dimensional polygon model.   A part of the four or more distance image sensors is arranged so as to catch the subject from above, and another part of the four or more distance image sensors is arranged so as to catch the subject from below. The method for creating a three-dimensional model according to claim 1. f) In each of the partial three-dimensional polygon models, the angle formed by the line-of-sight vector from the distance image sensor to the center point of the polygon and the normal vector of the polygon among the plurality of polygons is 90 degrees or less. The method for creating a three-dimensional model according to claim 1, further comprising a step of deleting a polygon. g) Static 3D combining the entire 3D polygon model with skeleton data prepared in advance and including a plurality of bones represented by straight lines and a plurality of joints represented by points and each connecting the bones Creating a model;
h) setting a weight value according to the distance between the point and the bone for each point of the stationary three-dimensional model;
i) superimposing motion data expressing a predetermined motion by motion of the bone and the joint on the data of the stationary three-dimensional model, and moving the point based on the motion data and the weight value;
The method for creating a three-dimensional model according to claim 1, wherein:   The method of creating a three-dimensional model according to any one of claims 1 to 4, further comprising a step of performing polygon reduction to reduce the number of polygons of the partial three-dimensional polygon model and / or the whole three-dimensional polygon model. .   6. The three-dimensional model according to claim 5, wherein the polygon reduction is a process of generating a point representing the polygon for all the polygons and generating a new polygon using the generated points. How to make. a) Imaging data that is two-dimensional image data of a subject acquired by each of four or more distance image sensors arranged so as to surround the subject, and a two-dimensional distribution of distances from the distance image sensor to the subject A data receiving unit for receiving input of depth data representing
b) a point coordinate generation unit that generates spatial coordinates of a plurality of points for each set of the imaging data and the depth data;
c) A partial 3D polygon model creation unit for creating a partial 3D polygon model made up of a combination of a plurality of polygons, each created by connecting a predetermined number of points located on the same plane among the plurality of points When,
d) Spatial coordinates corresponding to one common spatial coordinate system obtained by converting the spatial coordinates of a plurality of points constituting each partial three-dimensional polygon model based on the positional relationship between the corresponding distance image sensor and the subject. An overall 3D polygon model creation unit that creates an overall 3D polygon model in which a plurality of the partial 3D polygon models are integrated into one,
e) In the overall three-dimensional polygon model, a polygon aggregate deletion unit that deletes a polygon aggregate composed of a predetermined number of polygons or less, and
A three-dimensional model creation device comprising:   The three-dimensional model creation apparatus according to claim 7, wherein the three-dimensional model creation apparatus is embodied by the same number of computers as the distance image sensors connected to each other via a network.   A three-dimensional model creation program for causing a computer to operate as the three-dimensional model creation device according to claim 7 or 8. a) Acquire imaging data that is arranged so as to surround a predetermined subject position, each of which is two-dimensional image data of the subject, and depth data representing a two-dimensional distribution of the distance from the distance image sensor to the subject. 4 or more range image sensors that are possible,
b) a data acquisition unit for acquiring imaging data and depth data of the subject by each of the four or more distance image sensors in accordance with a predetermined input operation by a user;
c) a vertex coordinate generation unit that generates spatial coordinates of a plurality of points for each set of the imaging data and the depth data acquired by each of the four or more distance image sensors;
d) For each of the four or more distance image sensors, a partial three-dimensional combination composed of a combination of a plurality of polygons created by connecting a predetermined number of points located on the same plane among the plurality of points. A partial 3D polygon model creation unit for creating a polygon model;
e) Spatial coordinates corresponding to one common space coordinate system by converting the spatial coordinates of a plurality of points constituting each partial three-dimensional polygon model into coordinates based on the positional relationship between the corresponding distance image sensor and the subject. An overall 3D polygon model creation unit that creates an overall 3D polygon model in which a plurality of the partial 3D polygon models are integrated into one,
f) In the overall three-dimensional polygon model, a polygon aggregate deleting unit that deletes a polygon aggregate composed of a predetermined number of polygons or less, and
A system for creating a three-dimensional model, comprising:   A part of the four or more distance image sensors is arranged so as to catch the subject from above, and another part of the four or more distance image sensors is arranged so as to catch the subject from below. The three-dimensional model creation system according to claim 10.

Images