JP6814036B2 - Element image group generator and its program - Google Patents

Description

The present invention relates to an element image group generating device and a program thereof that generate a second element image group having color information by applying texture mapping to a first element image group having no color information generated in advance from a three-dimensional model.

In the conventional integral photography (IP) method, a single camera captures a subject through a lens array in which a plurality of minute lenses are arranged. At this time, since the camera captures the focal plane of the lens array, each minute lens constituting the lens array functions in the same manner as the minute camera. As a result, the image of the subject photographed through the lens array becomes an element image group in which minute images (element images) corresponding to the positions of the minute lenses are arranged. The element image group is an image in which light ray information from the subject is stored, and the number of light rays that can be stored depends on the resolution of the camera. Therefore, a high-resolution camera is required for IP-type shooting. It is also necessary to correct the effects of chromatic aberration and distortion of the imaging optical system.

Further, conventionally, a technique for generating an element image group of an IP system by using computer graphics (CG) has been studied. In the conventional technology, a virtual three-dimensional space is generated in a computer, a three-dimensional subject and a background (set) are reproduced in the same manner as in the real world, and light rays from the subject are tracked in the virtual three-dimensional space to obtain elements. Generate a group of images. As a method for generating an element image by CG, the methods described in the following Patent Documents 1 and 2 and Non-Patent Documents 1 and 2 are known. These element image generation methods trace the light rays from the subject through a depth control lens that controls the light rays from the subject. In these methods, it is necessary for the observer to take pictures with the camera as many times as the number of pixels of the element image group. That is, since the element image generation method of Non-Patent Document 1 uses the ray tracing method for each pixel, it is equivalent to a camera that shoots for each pixel, and the number of shots is the same as the number of pixels of the element image group. .. Therefore, in order to further increase the speed, as a prior art, there is also known a method of photographing a CG object with a camera by normal projection and converting these images into an element image group (Patent Document 3 and Non-Patent Document 3). In such an element image conversion method, the number of times the camera takes a picture is only the number of pixels forming the element image.

Japanese Patent No. 4567422 International Publication No. 00/059235 Japanese Unexamined Patent Publication No. 2011-234142

Spyros S. Athineos et al., “Physical modeling of a micro lens array setup for use in computer generated IP,” Proc of SPIE-IS & T Electronic Imaging, Vol.5664 pp.472-479,2005 Satoshi Nakajima et al., “High-speed image creation method for 3D displays using the principle of Integral Photography”, Journal of the Institute of Image Information and Television Engineers Vol. 54, No. 3, pp.420-425 (2000) M. Katayama et al., “A method for converting three-dimensional models into auto-stereoscopic images based on integral photography,” Proc of SPIE-IS & T Vol.6805 68050Z-1 68050Z-8

In the above-mentioned conventional technique, in order to generate a high-definition stereoscopic image, it is indispensable to increase the number of pixels of the element image group. However, in the above-mentioned conventional technique, it takes time to generate the element image group as the number of pixels of the element image group increases, so that further speeding up is required.

Therefore, an object of the present invention is to provide an element image group generating device and a program thereof that can shorten the generation time of the element image group.

In view of the above-mentioned problems, the element image group generation device according to the present invention performs texture mapping on the first element image group without color information generated in advance from the three-dimensional model, thereby performing the texture mapping on the second element image with color information. An element image group generation device for generating a group, which is a subject image input means, a three-dimensional model storage means, a first element image group storage means, a pixel position information storage means, a feature amount calculation means, and three-dimensional. The configuration includes a model selection means, a first element image group selection means, a pixel position information selection means, and a second element image group generation means.

According to such a configuration, the element image group generating device inputs the subject image by the subject image input means. The subject image is a photographed image of the subject, and includes color information (texture) of the subject.
The element image group generating device stores a plurality of three-dimensional models in advance by the three-dimensional model storage means. The three-dimensional model represents the three-dimensional shape of the face or object of any person and does not include color information.

The element image group generating device stores in advance the first element image group according to the position and direction of the three-dimensional model by the first element image group storage means. The first element image group is composed of element images of a three-dimensional model without color information.
The element image group generating device stores in advance pixel position information in which the pixel positions of the three-dimensional model, the subject image, and the element image of the first element image group are associated with each other by the pixel position information storage means.

The element image group generating device calculates a shape feature amount representing the shape of the subject and a situation feature amount representing the position and direction of the subject from the subject image by the feature amount calculating means.
The element image group generating device selects a three-dimensional model most similar to the subject from the three-dimensional model storage means based on the shape feature amount by the three-dimensional model selection means.
The element image group generation device selects the first element image group according to the position and direction of the situation feature amount from the first element image group storage means for the selected three-dimensional model by the first element image group selection means.

The element image group generating device selects the pixel position information according to the position and direction of the situation feature amount from the pixel position information storage means for the selected three-dimensional model by the pixel position information selection means.
The element image group generating device texture-maps the subject image to the selected first element image group based on the pixel position of the selected pixel position information by the second element image group generating means, thereby performing the second element of the subject. Generate a group of images.

In this way, the element image group generating device stores the first element image group without color information in advance, and performs texture mapping on the first element image group. Therefore, since the element image group generating device does not need to perform the ray tracing method or the normal projection which requires a large amount of calculation each time, the amount of calculation can be suppressed and the generation time of the first element image group can be shortened.

According to the present invention, the following excellent effects are obtained.
Since the element image group generating device according to the present invention performs texture mapping on the first element image group stored in advance, it is not necessary to perform a ray tracing method or a normal projection that requires a large amount of calculation each time. As a result, the element image group generating device can shorten the generation time of the first element image group and quickly generate the second element image group.

It is a block diagram which shows the structure of the stereoscopic image display system which concerns on 1st Embodiment and 3rd Embodiment. It is explanatory drawing explaining an example of a subject. It is explanatory drawing explaining an example of a 3D model. It is explanatory drawing explaining an example of the 1st element image group. It is explanatory drawing explaining the generation of the 1st element image group and the texture coordinate conversion table according to the horizontal position and the vertical position of a face, (a) is the face is located in the center, (b) is the face on the right side. When it is located, (c) indicates that the face is located on the upper side. It is explanatory drawing explaining the generation of the 1st element image group and the texture coordinate conversion table according to the depth position of a face, (a) is the face is positioned on the front side, (b) is the face is positioned on the back side. The case to do is shown. It is explanatory drawing explaining the generation of the 1st element image group and the texture coordinate conversion table according to the rotation direction of a face. It is explanatory drawing explaining the generation of the 1st element image group and the texture coordinate conversion table according to the facial expression, (a) shows the case where the eyes are open, and (b) shows the case where eyes are closed. It is explanatory drawing explaining the relationship between the 3D model, the 1st element image group, and a texture coordinate conversion table. It is explanatory drawing explaining the calculation method of the direction of a face as a situation feature quantity. It is explanatory drawing explaining an example of the texture extracted from the subject image. It is explanatory drawing explaining an example of the 1st element image group. It is a flowchart which shows the operation of the element image group generating apparatus which concerns on 1st Embodiment and 3rd Embodiment. It is a block diagram which shows the structure of the stereoscopic image display system which concerns on 2nd Embodiment. It is explanatory drawing explaining the complementation of the texture coordinate conversion table corresponding to a horizontal position and a vertical position. It is a flowchart which shows the operation of the element image group generating apparatus which concerns on 2nd Embodiment. It is a block diagram which shows the structure of the stereoscopic image display system which concerns on 4th Embodiment. It is explanatory drawing explaining the generation method of a multi-viewpoint image. It is a flowchart which shows the operation of the element image group generating apparatus which concerns on 4th Embodiment. It is explanatory drawing explaining an example of the subject image acquired by two subject image acquisition apparatus.

Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, the same means and processes are designated by the same reference numerals, and the description thereof is omitted.

(First Embodiment)
[Outline of stereoscopic image display system]
The outline of the stereoscopic image display system 1 according to the first embodiment of the present invention will be described with reference to FIG.
As shown in FIG. 1, the stereoscopic image display system 1 displays a stereoscopic image (element image group) by an IP method, and includes a subject image acquisition device 10, an element image group generation device 20, and an IP stereoscopic image display. The device 30 is provided.

The subject image acquisition device 10 acquires a subject image obtained by photographing the subject, and includes an imaging means 11 and an image storage means 13.
The photographing means 11 is a general camera for photographing a subject. In the present embodiment, the photographing means 11 is composed of one camera.
The image storage means 13 stores a subject image taken by the photographing means 11, and is, for example, a storage medium such as an SSD (Solid State Drive), an HDD (Hard Disk Drive), or a memory.

The element image group generating device 20 generates a stereoscopic image to be displayed by the IP stereoscopic image display device 30. In the present embodiment, the element image group generating device 20 stores in advance a three-dimensional model without a texture and a first element image group generated from the three-dimensional model in order to shorten the processing time. Then, the element image group generation device 20 selects a three-dimensional model most similar to the subject from the subject image of the subject image acquisition device 10. Further, the element image group generation device 20 uses the subject image of the subject image acquisition device 10 to attach the texture of the subject to the first element image group representing the selected three-dimensional model.
The configuration of the element image group generating device 20 will be described later.

The IP stereoscopic image display device 30 displays a stereoscopic image generated by the element image group generating device 20. The IP stereoscopic image display device 30 is a general IP type display device, and includes, for example, a lens array (not shown) and a display element.

[Configuration of element image group generator]
Hereinafter, the configuration of the element image group generation device 20 will be described.
The element image group generating device 20 includes a subject image input means 21, a database 22, a feature amount calculation means 23, a three-dimensional model selection means 24, a first element image group selection means 25, and a texture coordinate conversion table selection means. (Pixel position information selection means) 26 and a second element image group generation means 27 are provided.

The subject image input means 21 inputs a subject image stored by the subject image acquisition device 10 (image storage means 13). Then, the subject image input means 21 outputs the input subject image to the feature amount calculation means 23 and the second element image group generation means 27.
In the present embodiment, the subject is the face F of a person as shown in FIG. Therefore, the subject image is an image obtained by photographing the face F of a person as the subject.

The database 22 is a database that stores various information necessary for generating an element image group, and is a three-dimensional model database (three-dimensional model storage means) 221 and a first element image group database (first element image group storage means). It includes 223 and a texture coordinate conversion table database (pixel position information storage means) 225.

The three-dimensional model database 221 is a database in which a plurality of three-dimensional models are stored in advance. In the present embodiment, the three-dimensional model database 221 stores the three-dimensional model M of the face of a person, as shown in FIG. The three-dimensional model M represents the three-dimensional shape of a person's face.

Here, it is preferable that the three-dimensional model database 221 stores a large number of three-dimensional models M so that three-dimensional models M similar to various subjects can be selected. For example, the three-dimensional model database 221 stores a three-dimensional model M of the faces of persons having different races, genders, physiques, and ages.

For example, a three-dimensional model of a polygon is a model that expresses a three-dimensional shape by combining polygons such as triangles and quadrangles. Then, by pasting a texture on the three-dimensional model of the polygon (texture mapping), the texture can be improved. That is, as shown in FIG. 3, the three-dimensional model M of the present embodiment represents only the shape of the face, and the texture (color information) of the face is not added.

The first element image group database 223 is a database in which the first element image group is stored in advance. Here, the first element image group database 223 stores a number of first element image groups 90 according to the position, direction, and facial expression of the three-dimensional model M for each three-dimensional model M. In the present embodiment, as shown in FIG. 4, the first element image group 90 is composed of the element image 91 of the three-dimensional model M without color information (texture).

The texture coordinate conversion table database 225 is a database that stores the texture coordinate conversion table (pixel position information) in advance. In the present embodiment, the texture coordinate conversion table database 225 stores the texture coordinate conversion table for each of the first element image groups 90. The texture coordinate conversion table is information in which the pixel positions of all the pixels of the three-dimensional model M, the subject image, and the element image 91 of the first element image group 90 are associated with each other.

<How to generate 3D model, 1st element image group, texture coordinate conversion table>
Hereinafter, a method of generating the three-dimensional model, the first element image group 90, and the texture coordinate conversion table stored in the database 22 will be specifically described.

The three-dimensional model M can be generated by any method, for example, by photogrammetry. Photogrammetry captures the face of the person who is the source of the 3D model M from different viewpoints, obtains depth information from the parallax of each shot image (temporary texture image), and obtains depth information based on the obtained depth information. This is a method for generating M.

The first element image group 90 can be generated by the ray tracing method as in the conventional case. Specifically, in the ray tracing method, a virtual camera including an image sensor, a lens array, and a depth control lens is arranged in a virtual three-dimensional space in which the three-dimensional model M is arranged. Then, in the ray tracing method, information on a ray that passes through the lens array and the depth control lens for each pixel of the image sensor included in the virtual camera and reaches the three-dimensional model M is calculated. The first element image group 90 may be generated by oblique projection as in the prior art.

Here, the correspondence between the pixel positions of the three-dimensional model M and the pixel positions of the captured image that is the source of the three-dimensional model M is obtained when the three-dimensional model M is generated. Further, if the subject image is normalized to the same size as the captured image, the correspondence relationship between the pixel position of the three-dimensional model M and the pixel position of the subject image is also determined.
Further, a correspondence relationship between the pixel position of the three-dimensional model M and the pixel position of each element image 91 constituting the first element image group 90 is also required by the ray tracing method or oblique projection.
As described above, the texture coordinate conversion table can be obtained when the three-dimensional model M and the first element image group 90 are generated.

The texture coordinate conversion table represents the correspondence between the input (x1, y1, R1, G1, B1) and the output (x2, y2, R2, G2, B2) (reference numeral 93 in FIG. 9). x1 and y1 represent the pixel positions (coordinates) of the subject image. R1, G1, B1 represent the RGB values of the coordinates (x1, y1). Further, x2 and y2 represent pixel positions (coordinates) of the element images 91 constituting the first element image group 90. R2, G2 and B2 represent RGB values of coordinates (x2, y2). Here, since there are a plurality of element images 91 for one subject image, the texture coordinate conversion table has a maximum number of element images 91 for one input (x1, y1, R1, G1, B1). It has outputs (x2, y2, R2, G2, B2).

When the face F is located in the center of the subject image 92 as shown in FIG. 5 (a), and when the face F is located on the right side as shown in FIG. 5 (b), as shown in FIG. 5 (c). It is necessary to generate a second element image group according to the horizontal position and the vertical position of the face F, such as when the face F is located on the upper side.
Further, since the size of the face F is different, such as when the face F is located on the front side as shown in FIG. 6 (a) and when the face F is located on the back side as shown in FIG. 6 (b). , It is necessary to generate a second element image group according to the depth position of the face F.

Further, as shown in FIG. 7, when the face F is rotated in the directions of the horizontal axis (X axis), the vertical axis (Y axis), and the depth axis (Z axis), the appearance of the face F is different. , It is necessary to generate a second element image group according to the rotation direction of the face F.
Further, a second element image group corresponding to the facial expression of the face F is generated, such as when the eyes are open as shown in FIG. 8A and when the eyes are closed as shown in FIG. 8B. There is a need.
The facial expression is set in advance, for example, the eyes are closed, the eyes are open, the mouth is closed, the mouth is open, and the like.

From the above, as shown in FIG. 9, the database 22 includes the first element image group 90 and the first element image group 90 according to the position (horizontal position, vertical position and depth position), direction and facial expression of the face F for each three-dimensional model M. The texture coordinate conversion table 93 is stored in advance. That is, there are as many first element image groups 90 and texture coordinate conversion table 93 as the number of combinations of the position, direction, and facial expression type of the face F with respect to one three-dimensional model M.

The database 22 assigns identification information (ID) that can uniquely identify the three-dimensional model M to each three-dimensional model M. That is, the three-dimensional model M stored in the database 22 can be specified by referring to the identification information.

Returning to FIG. 1, the configuration of the element image group generating device 20 will be described.
The feature amount calculation means 23 calculates the shape feature amount and the situation feature amount of the subject from the subject image 92 input from the subject image input means 21.

In the present embodiment, the feature amount calculation means 23 calculates the face feature amount representing the face F as the shape feature amount. Specifically, the feature amount calculation means 23 extracts facial feature points such as eyes, nose, and mouth from the subject image 92. Then, the feature amount calculation means 23 uses the extracted facial feature points to determine the distance between the eyes (distance between the pupils of both eyes), the distance between the eyes and the nose and the mouth, the size of the mouth (distance between the left and right ends of the mouth), and the like. Calculate the amount of facial features.

The feature amount calculation means 23 may calculate the facial feature amount by using the method described in Japanese Patent No. 4954945. In this conventional technique, the orientation of the face F with respect to the subject image acquisition device 10 and the three-dimensional position of the feature points of the face F are alternated by estimating the factor matrix of the matrix showing the feature point data included in the subject image 92. It is estimated to be.

Further, the feature amount calculating means 23 calculates the position and direction of the face F and the facial expression feature amount as the situation feature amount.
The position of the face F represents the horizontal position, the vertical position, and the depth position of the face F included in the subject image 92. For example, the horizontal position and the vertical position of the face F are the positions of the center of gravity of the face region included in the subject image 92. Further, for example, the depth position of the face F can be obtained by TOF (Time Of Flight) or parallax.

The direction of the face F is a rotation direction (rotation angle) of the face F included in the subject image 92 when a preset direction (for example, the front surface) is used as a reference. The direction of the face F is the direction of rotation as seen from the subject, and is flipped horizontally in the subject image 92.
Here, the rotational direction of the left and right of the face F, as shown in FIG. 10, and measuring the area of the area and the left L _L of the right L _R of the face F, obtained from the area ratio of the left and right. For example, if the area ratio of the left and right are the same front direction, the area of the right L _R becomes the left direction if wider than the area of the left L _L. Further, the vertical rotation direction of the face F is obtained by obtaining the area on the upper side and the area on the lower side of the face F from the area ratio of the upper and lower sides (not shown). Further, the direction of rotation of the face F with respect to the horizontal plane (that is, the inclination of the face F) is obtained from the angle formed by the line segment connecting the crown and the chin tip and the vertical axis (not shown).

The facial expression feature amount is a feature amount expressing the facial expression of the face F based on the distance between the two facial feature points and the area of the area surrounded by three or more facial feature points. For example, facial expression features represent eyes closed, eyes open, mouth closed, mouth open, and the like.

As the facial expression feature amount, the feature amount described in the following references can be used. The method described in this reference is to obtain linear features, triangular features, triangular features by luminance average, and triangular features by luminance histogram based on the interrelationship of facial feature points.
References: Nomiya et al., "Facial expression expression scene detection in video based on facial expression recognition by feature selection and feature extraction using facial feature points", DEIM Forum 2011

The feature amount calculating means 23 outputs the calculated shape feature amount and the situation feature amount to the three-dimensional model selection means 24.

The three-dimensional model selection means 24 selects the three-dimensional model M most similar to the face F of the subject image 92 from the three-dimensional model database 221 based on the shape feature amount calculated by the feature amount calculating means 23. For example, in the three-dimensional model selection means 24, among each three-dimensional model M of the three-dimensional model database 221, the inter-eye distance of the facial feature amount, the size of the mouth, and the distance between the eyes, the nose, and the mouth are Select the closest 3D model M. Then, the three-dimensional model selection means 24 outputs the identification information of the selected three-dimensional model M and the situation feature amount input from the feature amount calculation means 23 to the first element image group selection means 25.

The first element image group selection means 25 displays the first element image group 90 according to the situation feature amount calculated by the feature amount calculation means 23 for the three-dimensional model M selected by the three-dimensional model selection means 24. It is selected from the group database 223. That is, the first element image group selection means 25 selects the first element image group 90 that matches the position, direction, and facial expression of the face F from the first element image group database 223. Then, the first element image group selection means 25 converts the selected first element image group 90 and the identification information and the situation feature amount of the three-dimensional model M input from the three-dimensional model selection means 24 into the texture coordinate conversion table selection means. Output to 26.

The texture coordinate conversion table selection means 26 uses the texture coordinate conversion table 93 according to the situation feature amount calculated by the feature amount calculation means 23 for the three-dimensional model M selected by the three-dimensional model selection means 24. Choose from. That is, the texture coordinate conversion table selection means 26 selects the texture coordinate conversion table 93 corresponding to the first element image group 90 selected by the first element image group selection means 25 from the texture coordinate conversion table database 225. Then, the texture coordinate conversion table selection means 26 outputs the selected texture coordinate conversion table 93 and the first element image group 90 input from the first element image group selection means 25 to the second element image group generation means 27. ..

The second element image group generation means 27 selects the first element image group 90 selected by the first element image group selection means 25 based on the pixel positions of the texture coordinate conversion table 93 selected by the texture coordinate conversion table selection means 26. The subject image 92 is texture-mapped.

Specifically, the second element image group generating means 27 extracts the texture of the face F from the subject image 92 input from the subject image input means 21 as shown in FIG. Then, the second element image group generation means 27 uses the texture extracted from the subject image 92 as the first element according to the correspondence between the pixel positions of the first element image group 90 and the subject image 92 represented by the texture coordinate conversion table 93. Paste it on the image group 90. In this way, the second element image group generating means 27 generates the second element image group. As shown in FIG. 12, the second element image group 94 is composed of the element image 95 to which the texture of the face F is attached.

[Operation of element image group generator]
The operation of the element image group generating device 20 will be described with reference to FIG. 13 (see FIG. 1 as appropriate).
It is assumed that the database 22 stores the three-dimensional model M, the first element image group 90, and the texture coordinate conversion table 93 in advance.

As shown in FIG. 13, the subject image input means 21 inputs the subject image 92 from the subject image acquisition device 10 (step S1).
The feature amount calculation means 23 calculates the face feature amount from the subject image 92 input in step S1 (step S2).
The feature amount calculation means 23 calculates the position and direction of the face F and the facial expression feature amount as the situation feature amount from the subject image 92 input in step S1 (step S3).

The 3D model selection means 24 selects the 3D model M most similar to the face F of the subject image 92 from the 3D model database 221 based on the face feature amount calculated in step S3 (step S4).
The first element image group selection means 25 selects the first element image group 90 according to the situational feature amount calculated in step S3 from the first element image group database 223 for the three-dimensional model M selected in step S4 (. Step S5).

The texture coordinate conversion table selection means 26 selects the texture coordinate conversion table 93 according to the situational feature amount calculated in step S3 from the texture coordinate conversion table database 225 for the three-dimensional model M selected in step S4 (step S6). ..
The second element image group generating means 27 texture-maps the subject image 92 to the first element image group 90 selected in step S5 based on the pixel positions of the texture coordinate conversion table 93 selected in step S6 (step S7). ..

[Action / Effect]
As described above, since the element image group generating device 20 according to the first embodiment of the present invention performs texture mapping on the first element image group 90 stored in advance, a ray tracing method or a normal projection with a large amount of calculation is performed. You don't have to do it every time. That is, the element image group generating device 20 only performs texture mapping as compared with the case where the ray tracing method or the normal projection is performed each time as in the prior art, so that the amount of calculation can be suppressed. As a result, the element image group generation device 20 can shorten the generation time of the first element image group 90 and generate the second element image group 94 in real time.

(Second Embodiment)
With reference to FIG. 14, the stereoscopic image display system 1B according to the second embodiment of the present invention will be described as different from the first embodiment.

In the first embodiment, the texture coordinate conversion table database 225 stores as many texture coordinate conversion tables 93 as the number corresponding to the horizontal position, the vertical position, the depth position, the rotation direction, and the expression. Therefore, the texture coordinate conversion table 93 has a huge number of combinations of horizontal position, vertical position, depth position, rotation direction, and facial expression.
Therefore, in the second embodiment, the element image group generating device 20B is different from the first embodiment in that the number of the texture coordinate conversion table 93 to be stored is reduced and the insufficient texture coordinate conversion table 93 is complemented.

[Configuration of element image group generator]
As shown in FIG. 14, the element image group generation device 20B includes a subject image input means 21, a database 22B, a feature amount calculation means 23, a three-dimensional model selection means 24, and a first element image group selection means 25. The texture coordinate conversion table selection means 26B, the second element image group generation means 27B, and the texture coordinate conversion table complement means (pixel position information interpolation means) 28 are provided.

The database 22B includes a three-dimensional model database 221, a first element image group database 223, and a texture coordinate conversion table database (pixel position information storage means) 225B.

The texture coordinate conversion table database 225B is a database that stores the texture coordinate conversion table 93 in advance.
In the present embodiment, the texture coordinate conversion table database 225B stores in advance the texture coordinate conversion table 93 when the face F is positioned at a preset reference position (for example, the center position) in order to correspond to the position change of the face F. To do.

Further, in the present embodiment, the texture coordinate conversion table database 225B has a texture only in a predetermined reference direction for each of the vertical axis (Y axis) and the depth axis (Z axis) in order to correspond to the direction change of the face F. The coordinate conversion table 93 is stored in advance. For example, in the case of the depth axis (Z axis), the left-handed texture coordinate conversion table 93 is stored in advance with the predetermined reference direction as counterclockwise (counterclockwise) (the same applies to the Y-axis). In this case, the texture coordinate conversion table database 225B does not store the right-handed (clockwise) texture coordinate conversion table 93 with respect to the depth axis (Z-axis).

Since the face F is not vertically symmetrical in the texture coordinate conversion table database 225B, the texture coordinate conversion table 93 at all angles is stored in advance for the horizontal axis (X axis).
Further, in the present embodiment, the texture coordinate conversion table database 225B stores the texture coordinate conversion table 93 for each facial expression in advance as in the first embodiment.

The texture coordinate conversion table selection means 26B selects the texture coordinate conversion table 93 as in the first embodiment. Further, when the texture coordinate conversion table database 225B does not store the texture coordinate conversion table 93 to be selected, the texture coordinate conversion table selection means 26B outputs a completion request to the texture coordinate conversion table complement means 28. At this time, the texture coordinate conversion table selection means 26B adds the situation feature amount input from the first element image group selection means 25 to the completion request in order to specify the texture coordinate conversion table 93 to be complemented.

The texture coordinate conversion table selection means 26B uses the texture coordinate conversion table 93 complemented by the texture coordinate conversion table complement means 28 and the first element image group 90 input from the first element image group selection means 25 as the second element image group. Output to the generation means 27B.

When the texture coordinate conversion table complement means 28 complements the texture coordinate conversion table 93 in the rotation direction with respect to the Y axis, the second element image group generating means 27B flips the second element image group 94 left and right after texture mapping. As a result, the second element image group generating means 27B can generate the second element image group 94 oriented in the direction opposite to the predetermined reference direction (for example, clockwise).

The second element image group generation means 27B may invert the horizontal coordinates x2 included in the output of the texture coordinate conversion table 93 left and right, and perform texture mapping based on the texture coordinate conversion table 93. Also by this, the second element image group generation means 27B can generate the second element image group 94 oriented in the direction opposite to the reference direction.
In other respects, the second element image group generating means 27B is the same as the first embodiment, and thus the description thereof will be omitted.

The texture coordinate conversion table complement means 28 complements the texture coordinate conversion table 93 according to the situational features (position and direction) of the completion request input from the texture coordinate conversion table selection means 26B.

<Complementation of texture coordinate conversion table: Correspondence between horizontal position and vertical position>
Hereinafter, a method for complementing the texture coordinate conversion table 93 will be specifically described.
First, a method of complementing the horizontal position and the vertical position will be described.

Texture coordinate conversion table complementing unit 28 reads from the texture coordinate conversion table database 225B, the face F is the reference position _(C X, C _Y) texture coordinate transformation table 93 in. Then, texture coordinates conversion table complementing unit 28 extracts the horizontal position G _X and vertical position G _Y of the face F contained in the status feature amount. Then, texture coordinates conversion table complementing unit 28, as shown in FIG. 15, and calculates the horizontal position G _X of the face F, the positional deviation amount ΔX in the horizontal direction of the reference position C _X. Then, texture coordinates conversion table complementing unit 28 calculates a vertical position G _Y of the face F, the positional shift amount ΔY of the vertical direction of the reference position C _Y. Then, the texture coordinate conversion table complementing means 28 moves the pixel position indicated by the texture coordinate conversion table 93 when the face F is positioned in front in accordance with the horizontal displacement amount ΔX and the vertical displacement amount ΔY. Let me. Note that ΔX and ΔY are integer element image intervals, and are rounded off to the nearest whole number.

<Complementation of texture coordinate conversion table: Correspondence of depth position>
The texture coordinate conversion table complement means 28 extracts the depth position of the face F included in the situation feature amount. Next, the texture coordinate conversion table complementing means 28 selects one texture coordinate conversion table 93 closest to the depth position of the extracted face F from the front side and the back side in the depth direction (Z-axis direction), respectively. Then, the texture coordinate conversion table complementing means 28 complements (for example, interpolates) the texture coordinate conversion table 93 corresponding to the depth position of the face F from the two selected texture coordinate conversion tables 93.

Here, the texture coordinate conversion table complementing means 28 preferably selects two texture coordinate conversion tables 93 within the displayable depth range of the IP stereoscopic image display device 30 set in advance. In this way, the element image group generating device 20B suppresses the number of texture coordinate conversion tables 93 in the depth direction by considering the resolution characteristic of the IP stereoscopic image display device 30, and makes the resolution of the stereoscopic image above a certain level. Can be kept.

<Complementation of texture coordinate conversion table: Correspondence of rotation direction>
The texture coordinate conversion table complement means 28 extracts the rotation direction of the face F included in the situation feature amount. Next, the texture coordinate conversion table complementing means 28 reads out the texture coordinate conversion table 93 that matches the rotation direction of the extracted face F and the rotation direction is opposite.

Next, the texture coordinate conversion table complement means 28 converts the read texture coordinate conversion table 93 in the reverse direction. For example, it is assumed that the rotation direction of the face F is clockwise, and the texture coordinate conversion table database 225B stores only the counterclockwise texture coordinate conversion table 93 and does not store the clockwise texture coordinate conversion table 93. In this case, the texture coordinate conversion table complement means 28 calculates a value obtained by multiplying the rotation direction of the face F by a negative coefficient (that is, ‘-1’). As a result, the element image group generating device 20B can select the corresponding left-handed texture coordinate conversion table 93 and convert the selected texture coordinate conversion table 93 clockwise. In this way, the element image group generating device 20B complements the texture coordinate conversion table 93 corresponding to the rotation direction of the face F.

Although the completion of the horizontal position, the vertical position, the depth position, and the rotation direction has been described individually, the texture coordinate conversion table complement means 28 may complement the texture coordinate conversion table 93 by combining two or more of these.

[Operation of element image group generator]
The operation of the element image group generating device 20B will be described with reference to FIG. 16 (see FIG. 14 as appropriate).
It is assumed that the database 22B stores the three-dimensional model M, the first element image group 90, and the texture coordinate conversion table 93 in advance.
Since the processes of steps S1 to S5 are the same as those of the first embodiment, the description thereof will be omitted.

As shown in FIG. 16, the texture coordinate conversion table selection means 26B determines whether or not the texture coordinate conversion table database 225B stores the texture coordinate conversion table 93 according to the situation feature amount (step S10).

When the texture coordinate conversion table 93 is stored (Yes in step S10), the element image group generating device 20B performs the process of step S6.
Since the process of step S6 is the same as that of the first embodiment, the description thereof will be omitted.

When the texture coordinate conversion table 93 is not stored (No in step S10), the texture coordinate conversion table selection means 26B outputs a completion request to which the situation feature amount is added to the texture coordinate conversion table complement means 28 (step S11). ..

The texture coordinate conversion table complement means 28 complements the texture coordinate conversion table 93 according to the completion request in step S11. Here, the texture coordinate conversion table complement means 28 complements the texture coordinate conversion table 93 in correspondence with the horizontal position, the vertical position, the depth position, and the rotation direction (step S12).

The second element image group generating means 27 is a subject in the first element image group 90 selected in step S5 based on the texture coordinate conversion table 93 selected in step S6 or the texture coordinate conversion table 93 supplemented in step S12. Texture mapping the image 92 (step S13).

[Action / Effect]
As described above, the element image group generating device 20B according to the second embodiment of the present invention can complement the texture coordinate conversion table 93 corresponding to the position and rotation direction of the face F, so that each position and rotation direction of the face F can be complemented. It is not necessary to store the texture coordinate conversion table 93 in the database, and the capacity of the database 22B can be reduced.

(Third Embodiment)
With reference to FIG. 1, the stereoscopic image display system 1C according to the third embodiment of the present invention will be described as different from the first embodiment.

In the first embodiment, the texture coordinate conversion table 93 has been described as associating the pixel positions of all the pixels with respect to the three-dimensional model M, the subject image 92, and the element image 91 of the first element image group 90.
In the third embodiment, the texture coordinate conversion table 93 is different from the first embodiment in that the pixel positions are associated only with the vertex pixels of the three-dimensional model M.

[Configuration of element image group generator]
As shown in FIG. 1, the element image group generation device 20C includes a subject image input means 21, a database 22C, a feature amount calculation means 23, a three-dimensional model selection means 24, and a first element image group selection means 25. A texture coordinate conversion table selection means 26 and a second element image group generation means 27C are provided.

The database 22C includes a three-dimensional model database 221, a first element image group database 223, and a texture coordinate conversion table database (pixel position information storage means) 225C.

The texture coordinate conversion table database 225C is a database that stores the texture coordinate conversion table 93 in advance.
In the present embodiment, the texture coordinate conversion table database 225C stores the texture coordinate conversion table 93 in the vertex pixels of the three-dimensional model M in advance. Here, the texture coordinate conversion table 93 is information in which the pixel positions (vertex pixel positions) of the vertex pixels of the three-dimensional model M, the subject image 92, and the element image 91 of the first element image group 90 are associated with each other. .. That is, the texture coordinate conversion table 93 does not have a correspondence relationship of pixel positions with respect to pixels other than the vertex pixels of the three-dimensional model M.
The vertex pixels are pixels corresponding to the vertices of the polygons constituting the three-dimensional model M.

The second element image group generating means 27C texture-maps the subject image 92 to the first element image group 90 with the vertex pixels of the three-dimensional model M, and complements the first element image group 90 with pixels other than the vertex pixels. Is. Specifically, the second element image group generating means 27C refers to the texture coordinate conversion table 93, and attaches the texture extracted from the subject image 92 to the first element image group 90 for the vertex pixels of the three-dimensional model M. wear. Further, the second element image group generating means 27C complements (for example, interpolates) the first element image group 90 for pixels other than the apex pixels of the three-dimensional model M.

[Operation of element image group generator]
The operation of the element image group generating device 20C will be described with reference to FIG. 13 (see FIG. 1 as appropriate).
It is assumed that the database 22C stores the three-dimensional model M, the first element image group 90, and the texture coordinate conversion table 93 in advance.
Since the processes of steps S1 to S6 are the same as those of the first embodiment, the description thereof will be omitted.

The second element image group generating means 27C texture-maps the subject image 92 to the first element image group 90 selected in step S5 based on the texture coordinate conversion table 93 selected in step S6. Here, the second element image group generating means 27C texture-maps the subject image 92 with the vertex pixels of the three-dimensional model M, and complements the first element image group 90 with pixels other than the vertex pixels (step S7C).

[Action / Effect]
As described above, the element image group generating device 20C according to the third embodiment of the present invention does not need to store the texture coordinate conversion table 93 for pixels other than the apex pixels of the three-dimensional model M, so that the capacity of the database 22C Can be reduced.

The element image group generating device 20C according to the third embodiment can be applied not only to the first embodiment but also to the second embodiment. In this case, the element image group generation device 20C complements (for example, interpolates) the texture coordinate conversion table 93 for pixels other than the vertex pixels of the three-dimensional model M, and the first element according to the complemented texture coordinate conversion table 93. The image group 90 is complemented.

(Fourth Embodiment)
With reference to FIG. 17, the stereoscopic image display system 1D according to the fourth embodiment of the present invention will be described as different from the third embodiment.

In the third embodiment, the texture coordinate conversion table 93 for the vertex pixels of the three-dimensional model M has been described as being stored in advance. When the number of pixels of the element image 91 is small (for example, 10 pixels in the vertical direction and 10 pixels in the horizontal direction), the vertex pixels may not be sufficiently included in the element image 91. On the other hand, since the multi-viewpoint image is an image of the subject taken from different viewpoints, the vertices hidden in one viewpoint may be visible in another viewpoint. That is, the multi-viewpoint image may include more vertex pixels than the element image 91.
The fourth embodiment is different from the third embodiment in that a multi-viewpoint image is generated in advance from the element image 91 and the information of the pixels corresponding to the vertex pixels included in the multi-viewpoint image is used.

[Configuration of element image group generator]
As shown in FIG. 17, the element image group generating device 20D includes a subject image input means 21, a database 22D, a feature amount calculation means 23, a three-dimensional model selection means 24, and a first element image group selection means 25. The texture coordinate conversion table selection means 26, the second element image group generation means 27D, and the multi-viewpoint image selection means 29 are provided.

The database 22D includes a three-dimensional model database 221, a first element image group database 223, a texture coordinate conversion table database (pixel position information storage means) 225D, and a multi-viewpoint image database (multi-viewpoint image storage means) 227. Be prepared.

The texture coordinate conversion table database 225D is a database that stores the texture coordinate conversion table 93 in advance.
In the present embodiment, the texture coordinate conversion table 93 is information in which the vertex pixel positions of the three-dimensional model M, the multi-viewpoint image, the subject image 92, and the element image 91 of the first element image group 90 are associated with each other. ..

The multi-viewpoint image database 227 is a database that stores the multi-viewpoint images described later in advance. In the present embodiment, the multi-viewpoint image database 227 stores the multi-viewpoint images in advance for each of the first element image groups 90.

<Multi-viewpoint image>
A method of generating a multi-viewpoint image and a vertex pixel will be described with reference to FIG.
As shown in FIG. 18A, the first element image group 90 is composed of element images 91 obtained by photographing a textureless three-dimensional model M from different viewpoints. For example, the element image 91 of FIG. 18A corresponds to the three element images 91 in the upper left portion of the first element image group 90 of FIG.

The multi-viewpoint image is composed of pixels extracted from the same position in each element image 91. As shown in FIG. 18 (a), 1 single eye viewpoint image 95 ₁ of the is composed of pixels extracted from the upper left pixel of each element image 91. Further, as shown in FIG. 18 (b), the multi-viewpoint image 95 ₂ of the second is constituted by pixels extracted from the second pixel from the upper left of the element image 91.

For example, it is assumed that the first element image group 90 has 1920 pixels in the horizontal direction and 1080 pixels in the vertical direction, and the element image 91 has 10 pixels in the vertical and horizontal directions. In this case, a multi-viewpoint image 95 for 100 viewpoints (100 images) is generated from one first element image group 90. Further, each multi-viewpoint image 95 has a size of 192 pixels in the horizontal direction and 108 pixels in the vertical direction.

At this time, it is preferable that the subject image acquisition device 10 acquires the subject image 92 at a position corresponding to the viewpoint position in the real space. As a result, the viewpoints of the multi-viewpoint image 95 and the subject image 92 match, so that high-quality texture mapping becomes possible.

Returning to FIG. 17, the configuration of the element image group generating device 20D will be continued.
The multi-viewpoint image selection means 29 selects the multi-viewpoint image 95 corresponding to the first element image group 90 selected by the first element image group selection means 25 from the multi-viewpoint image database 227. Then, the multi-viewpoint image selection means 29 outputs the selected multi-viewpoint image 95 and the first element image group 90 input from the first element image group selection means 25 to the texture coordinate conversion table selection means 26.

The second element image group generation means 27D texture-maps the subject image 92 to the first element image group 90 with the vertex pixels of the three-dimensional model M, and complements the first element image group 90 with pixels other than the vertex pixels. Is.

First, the second element image group generation means 27D obtains the apex pixel position to be the target of texture mapping on the multi-view image 95 in the texture coordinate conversion table 93, and corresponds to the apex pixel position of the multi-view image 95. The apex pixel positions of the dimensional model M and the first element image group 90 are obtained.

Next, the second element image group generating means 27D performs texture mapping via the multi-viewpoint image 95.
Specifically, the second element image group generating means 27D texture-maps the subject image 92 to the multi-viewpoint image 95 according to the correspondence of the obtained vertex pixel positions. Then, the second element image group generating means 27D generates the first element image group 90 from the multi-viewpoint image 95 to which the texture of the subject is pasted in the reverse procedure of FIG. That is, the second element image group generating means 27D extracts pixels at the same position (for example, upper left) from each multi-viewpoint image 95, and the element image 91 (for example, the first element image group) corresponding to the extracted pixel positions. At 90, the upper left element image 91) is generated.
After that, the second element image group generating means 27D complements the first element image group 90 with pixels other than the vertex pixels.
As a result, the second element image group generating means 27D can reduce the number of memory accesses for reading the subject image 92, and can generate the second element image group more quickly.

Since the second element image group generating means 27D can obtain the pixel position of the first element image group 90 from the texture coordinate conversion table 93, the texture may be directly mapped to the first element image group 90.

[Operation of element image group generator]
The operation of the element image group generating device 20D will be described with reference to FIG. 19 (see FIG. 1 as appropriate).
It is assumed that the database 22D stores the three-dimensional model M, the first element image group 90, the multi-viewpoint image 95, and the texture coordinate conversion table 93 in advance.
Since the processes of steps S1 to S5 are the same as those of the third embodiment, the description thereof will be omitted.

The multi-viewpoint image selection means 29 selects the multi-viewpoint image 95 corresponding to the first element image group 90 selected in step S5 from the multi-viewpoint image database 227 (step S14).
Since the processes of steps S6 and S10 to S12 are the same as those of the third embodiment, the description thereof will be omitted.

The second element image group generation means 27D texture-maps the subject image 92 to the first element image group 90 with the vertex pixels of the three-dimensional model M, and complements the first element image group 90 with pixels other than the vertex pixels. Here, the second element image group generating means 27D refers to the texture coordinate conversion table 93 and obtains the apex pixel positions of the three-dimensional model M and the first element image group 90 corresponding to the apex pixel positions of the multi-viewpoint image 95. , Texture mapping is performed according to the obtained vertex pixel positions (step S15).

[Action / Effect]
As described above, the element image group generation device 20D according to the fourth embodiment of the present invention has pixel information corresponding to the vertex pixels included in the multi-viewpoint image 95 even when the number of pixels corresponding to the vertex pixels is not sufficient. Because you can use, you can perform higher quality texture mapping.
The element image group generating device 20D according to the fourth embodiment can be applied not only to the first embodiment but also to the second embodiment and the third embodiment.

Although each embodiment of the present invention has been described in detail above, the present invention is not limited to each of the above-described embodiments, and includes design changes and the like within a range not deviating from the gist of the present invention.

(Modification example 1)
In each of the above-described embodiments, one pixel (x1, y1) of the subject image corresponds to one pixel (x2, y2) of the element image, but the present invention is not limited thereto.
That is, the second element image group generating means may correspond a plurality of pixels of the subject image to one pixel of the element image. For example, the second element image group generating means complements the pixel EI1 of the element image from the pixels P1, P2, P3, P4 of the subject image (for example, linear interpolation) by the following conversion formula (1).

EI1 = αP1 + βP2 + γP3 + ζP4 ... Equation (1)
In the conversion formula (1), α, β, γ, and ζ are integers satisfying α + β + γ + ζ = 1.

(Modification 2)
In each of the above-described embodiments, the texture coordinate conversion table is stored for each facial expression, but the present invention is not limited to this.

Here, the facial expression changes according to the shape of the eyes and eyebrows. In addition, the depth positions of the eyes and eyebrows are constant even if the facial expression changes. Therefore, by switching to a subject image having a different facial expression, it is possible to express a change in facial expression.

Specifically, the texture coordinate conversion table may be shared regardless of the facial expression, and the texture coordinate conversion table may be stored according to the position and direction of the face. As a result, the element image group generating device does not need to store the texture coordinate conversion table for each facial expression, and the capacity of the database can be further reduced.

In this case, the feature amount calculating means preferably extracts feature points whose pixel positions do not change even if the facial expression changes. For example, the feature amount calculation means extracts the outer corners of the eyes, the inner corners of the eyes, the nose, the ears, and the contours of the face as feature points.

(Modification 3)
In each of the above-described embodiments, the subject is described as the face of a person, but the subject is not particularly limited.

For example, the subject is the upper body of a person. In this case, the database stores in advance a three-dimensional model of the upper body including the face, a first element image group corresponding to these three-dimensional models, and a texture coordinate conversion table.

Further, the subject may be any object. In this case, the database stores in advance three-dimensional models of various objects, a first element image group corresponding to these three-dimensional models, and a texture coordinate conversion table.

Further, the number of subjects may be plural. In this case, the element image group generating device extracts each object from the subject image and calculates the feature amount for each object. Then, the element image group generating device selects the three-dimensional model corresponding to each object, the first element image group, and the texture coordinate conversion table based on the feature amount of each object.
For example, consider the case where the subject image contains a plurality of objects such as apples, oranges, and bananas. In this case, the element image group generating device extracts apples, oranges, and bananas from the subject image, and includes a three-dimensional model corresponding to each of the apples, oranges, and bananas, the first element image group, and a texture coordinate conversion table. Select.

(Modification example 4)
In each of the above-described embodiments, the subject image is input from one subject image acquisition device, but the present invention is not limited thereto.

That is, the element image group generating device may input the subject image from a plurality of subject image acquisition devices. As shown in FIG. 20, consider a case where subject images 92 _R and 92 _L are input from two subject image acquisition devices 10 _R and 10 _L arranged in the horizontal direction. In this case, the subject image acquisition device 10 _R on the right side shoots from the left direction of the face F when viewed from the subject. Therefore, in the subject image (right subject image 92 _R ) acquired by the subject image acquisition device 10 _R , the left side of the face F is projected larger than the right side. On the other hand, the subject image acquisition device 10 _L on the left side shoots from the right direction of the face F when viewed from the subject. Therefore, in the subject image (left subject image 92 _L ) acquired by the subject image acquisition device 10 _L , the right side of the face F is projected larger than the left side. As described above, the right subject image contains a lot of information on the left side of the face, and the left photographed image contains a lot of information on the right side of the face.

Therefore, the element image group generating device considers the light ray reproduction direction in the element image group, and performs texture mapping with the subject images taken in different directions. Specifically, the element image group generating device texture-maps the pixels of the element image to be reproduced in the left direction using the right subject image. Further, the element image group generating device texture-maps the pixels of the element image to be reproduced in the right direction using the left subject image. As a result, the element image group generating device can more accurately reproduce the light reflection characteristics, and can perform texture mapping using a plurality of subject images in the vertical direction as well as in the horizontal direction.

(Modification 5)
Although each of the above-described embodiments has been described as having one subject image, the present invention is not limited to this.

Here, the subject image acquisition device stores the subject image captured by the photographing means in the image storage means. Further, the image storage means can store a plurality of captured subject images. Therefore, in the element image group generating device, the number of subject images stored by the image storage means increases with the passage of time, so that the number of subject images used for texture mapping can be increased. That is, the second element image group generated first is texture-mapped from one subject image. However, since the subject image increases with the passage of time, the second element image group generated later can be texture-mapped from a plurality of subject images.

Hereinafter, a method of texture mapping from a plurality of subject images will be described.
The feature amount calculation means calculates the feature amount from the first subject image. Next, the three-dimensional model selection means matches each vertex pixel of the three-dimensional model with the feature points of the first subject image. Next, the element image group generating device generates a texture coordinate conversion table corresponding to the three-dimensional model for the first subject image. Here, the element image group generating device repeats these processes for each subject image.

Then, the element image group generating device determines the most probable subject image in each vertex pixel of the three-dimensional model, and generates the final texture coordinate conversion table. Further, the element image group generating device can improve the texture of the stereoscopic image by performing texture mapping using the final texture coordinate conversion table.

(Other variants)
In each of the above-described embodiments, the element image group generator has been described as independent hardware, but the present invention is not limited thereto. For example, hardware resources such as a CPU, a memory, and a hard disk provided in a computer can be realized by an element image group generation program that cooperates as an element image group generation device. This program may be distributed via a communication line, or may be written and distributed on a storage medium such as a CD-ROM or a flash memory.

1,1B, 1C, 1D stereoscopic image display system 10 Subject image acquisition device 11 Imaging means 13 Image storage means 20, 20B, 20C, 20D Element image group generation device 21 Subject image input means 22, 22B, 22C, 22D Database 23 Features Quantity calculation means 24 Three-dimensional model selection means 25 First element image group selection means 26, 26B Texture coordinate conversion table selection means (pixel position information selection means)
27, 27B, 27C, 27D Second element image group generating means 28 Texture coordinate conversion table complementing means 29 Multi-viewpoint image selecting means 30 IP stereoscopic image display device 221 3D model database (3D model storage means)
223 First element image group database (first element image group storage means)
225, 225B, 225C, 225D texture coordinate conversion table database (pixel position information storage means)
227 Multi-viewpoint image database (multi-viewpoint image storage means)

Claims (8)

It is an element image group generator that generates a second element image group with color information by applying texture mapping to a first element image group without color information generated in advance from a three-dimensional model.
Subject image input means for inputting the subject image of the subject, and
A three-dimensional model storage means for storing a plurality of the three-dimensional models in advance,
A first element image group storage means for storing the first element image group according to the position and direction of the three-dimensional model in advance, and
Pixel position information storage means for storing in advance pixel position information in which the pixel positions of the three-dimensional model, the subject image, and the element image of the first element image group are associated with each other.
A feature amount calculating means for calculating a shape feature amount representing the shape of the subject and a situation feature amount representing the position and direction of the subject from the subject image.
A three-dimensional model selection means for selecting a three-dimensional model most similar to the subject from the three-dimensional model storage means based on the shape feature amount.
With respect to the selected three-dimensional model, a first element image group selection means for selecting a first element image group according to the position and direction of the situation feature amount from the first element image group storage means, and
With respect to the selected three-dimensional model, pixel position information selection means for selecting pixel position information according to the position and direction of the situation feature amount from the pixel position information storage means, and
A second element image group generating means that generates a second element image group of the subject by texture mapping the subject image to the selected first element image group based on the pixel position of the selected pixel position information. When,
An element image group generating device characterized by comprising. The three-dimensional model storage means stores the three-dimensional model representing the faces of different persons, and stores the three-dimensional model.
The feature amount calculating means calculates a face feature amount representing the face of the subject as the shape feature amount, and calculates the face feature amount.
The element image according to claim 1, wherein the three-dimensional model selection means selects a three-dimensional model most similar to the face of the subject from the three-dimensional model storage means based on the face feature amount. Group generator. The first element image group storage means stores the first element image group for each predetermined facial expression, and stores the first element image group.
The pixel position information storage means stores the pixel position information for each facial expression.
The element image group generating device according to claim 2, wherein the feature amount calculating means further calculates a facial expression feature amount representing the facial expression of the subject as the situation feature amount. The pixel position information storage means stores pixel position information in the vertex pixels of the three-dimensional model in advance, and stores the pixel position information.
The second element image group generating means is characterized in that the subject image is texture-mapped to the first element image group by the apex pixels and the first element image group is complemented by other than the apex pixels. The element image group generating device according to any one of claims 1 to 3. The subject image input means inputs a subject image obtained by capturing the subject in different directions, and inputs the subject image.
The second element image group generating means is characterized in that texture mapping is performed using the subject image corresponding to the ray reproduction direction of each pixel of the element image among the subject images in different directions. The element image group generating device according to any one of claims 4. The pixel position information storage means stores the pixel position information in a preset reference position and reference direction, and stores the pixel position information.
Claims 1 to 5 further include a pixel position information interpolation means that complements the pixel position information according to the position and direction of the situation feature amount from the pixel position information of the reference position and the reference direction. The element image group generation device according to any one of the above items. It is an element image group generator that generates a second element image group with color information by applying texture mapping to a first element image group without color information generated in advance from a three-dimensional model.
Subject image input means for inputting the subject image of the subject, and
A three-dimensional model storage means for storing a plurality of the three-dimensional models in advance,
A first element image group storage means for storing the first element image group according to the position and direction of the three-dimensional model in advance, and
A multi-viewpoint image storage means for preliminarily storing a multi-viewpoint image composed of pixels at the same position included in each element image of the first element image group.
Pixel position information storage means that stores in advance pixel position information in which the pixel positions of the apex pixels of the three-dimensional model, the subject image, the element image of the first element image group, and the multi-viewpoint image are associated with each other.
A feature amount calculating means for calculating a shape feature amount representing the shape of the subject and a situation feature amount representing the position and direction of the subject from the subject image.
A three-dimensional model selection means for selecting a three-dimensional model most similar to the subject from the three-dimensional model storage means based on the shape feature amount.
With respect to the selected three-dimensional model, a first element image group selection means for selecting a first element image group according to the position and direction of the situation feature amount from the first element image group storage means, and
With respect to the selected three-dimensional model, a multi-viewpoint image selection means for selecting a multi-viewpoint image according to the position and direction of the situation feature amount from the multi-viewpoint image storage means, and
With respect to the selected three-dimensional model, pixel position information selection means for selecting pixel position information according to the position and direction of the situation feature amount from the pixel position information storage means, and
Based on the selected pixel position information, the pixel positions of the three-dimensional model and the first element image group corresponding to the pixel positions of the apex pixels of the multi-viewpoint image are obtained, and the subject image is obtained at the obtained pixel positions. A second element image group generation means that generates a second element image group of the subject by texture mapping and complementing the first element image group other than the obtained pixel position.
An element image group generating device characterized by comprising. An element image group generation program for causing a computer to function as the element image group generation device according to any one of claims 1 to 7.

Images