JP2015184689A - Moving image generation device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To real-timely provide a moving image in which the face components and the body parts of a character move in response to the movement of the face components of a person and the body parts other than the person's face.SOLUTION: The moving image generation device is provided with: a first face-movement detection unit that continuously detects the movement of the face components of a first person from a moving image of the first person, the moving image being continuously input; a first body-movement detection unit that continuously detects the movement of the body parts thereof other than the first person's face; a storage unit that stores the three dimensional information of a first virtual character; and a moving image generation output unit that generates a character moving image, in which the face components of the first character and the body parts thereof other than the first character's face move in response to the movement of the face components and the body parts of the first person, on the basis of the movement of the face components, which has been continuously detected by the first face-movement detection unit, the movement of the body parts, which has been continuously detected by the first body-movement detection unit, and the three dimensional information of the first character, which is stored in the storage unit, the moving image generation output units outputting the generated character moving image after synchronizing same with the input of the moving image of the first person.

Description

  The present invention relates to a moving image generating apparatus and a program.

Means for generating phoneme data from voice data; means for generating an animation of a mouth portion of a two-dimensional or three-dimensional model based on phoneme data; and the voice data and the animation based on the phoneme data. 2. Description of the Related Art An animation generation apparatus having a means for reproducing in synchronization is known (see, for example, Patent Document 1).
Japanese Patent Application Laid-Open No. 2002-108382

  It is not possible to provide a moving image in which a character's face part and body part move in real time in accordance with the movement of the body part other than the person's face part and person's face.

  In the first aspect, the moving image generating apparatus includes a first face motion detecting unit that continuously detects the movement of the facial part of the first person from the continuously input moving images of the first person, A first body motion detection unit that continuously detects the motion of a body part other than the face of one person, a storage unit that stores three-dimensional information of a virtual first character, and a first face motion detection Based on the movement of the face part continuously detected by the body part, the movement of the body part continuously detected by the first body motion detection part, and the three-dimensional information of the first character stored in the storage part , Generating a character video in which the body part other than the face part of the first character and the face of the first character moves according to the movement of the face part and body of the first person, A moving image generation / output unit that outputs in synchronization with the input of the moving image of one person.

  A voice acquisition unit that continuously acquires the voice uttered by the first person in synchronization with the input of the first person's video is further provided, and the video generation output unit is a voice that the voice acquisition unit continuously acquires. May be combined with a character video and output.

  The storage unit further stores the three-dimensional information of the virtual second character, and the moving image generation device starts from the moving image of the second person that is continuously input in synchronization with the moving image of the first person. A second face motion detection unit that detects the movement of the facial part of the face of the second person, and a second body motion detection unit that continuously detects the movement of the body part other than the face of the second person The moving image generation output unit stores the movement of the facial part continuously detected by the first face movement detection unit, the movement of the body part continuously detected by the first body movement detection unit, and the storage unit. The three-dimensional information of the first character, the movement of the facial part continuously detected by the second face movement detection unit, the movement of the body part continuously detected by the second body movement detection unit, and the storage unit Based on the stored three-dimensional information of the second character, the first character according to the movement of the facial part and body part of the first person. Body parts other than the face part of the first character and the face of the first character move, and other than the face parts of the second character and the face of the second character according to the movement of the face part of the second person and the movement of the body part A character moving image in which the body part moves may be generated, and the generated character moving image may be output in synchronization with the input of the first person moving image and the second person moving image.

  A camera unit that continuously captures the first person; and a camera information acquisition unit that continuously acquires information indicating the imaging direction of the camera unit. Based on the information indicating the imaging direction of the camera unit, by generating a moving image of the first character in a direction that matches the orientation of the first person viewed from the camera unit, according to a change in the imaging direction of the camera unit You may generate the character animation from which an angle changes.

  In the second aspect, the program causes a computer to function as the moving image generating apparatus.

  The summary of the invention does not enumerate all the features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

An example of the use environment of interactive system 5 in one embodiment is shown roughly. An example of the block configuration of the moving image generation output system 10 in the first embodiment is schematically shown. An example of the block configuration of the moving image generation output system 12 in the second embodiment is schematically shown.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 schematically illustrates an example of an interactive system 5 in one embodiment. The interactive system 5 includes a moving image generation / output system 10, a user terminal 80, and a user terminal 81.

  The moving image generation / output system 10 communicates with the user terminal 80 and the user terminal 81 through the network 9. The network 9 includes the Internet, a mobile phone network, and the like. The user terminal 80 and the user terminal 81 are, for example, a mobile terminal such as a smartphone, a personal computer, or the like. A user 90 is a user of the user terminal 80. A user 91 is a user of the user terminal 81.

  The moving image generation / output system 10 detects the actions of the voice actor 60 and the voice actor 61 in real time. The movement includes facial expressions, facial movements, limb movements, and the like. In addition, the moving image generation / output system 10 acquires the voices produced by the voice actor 60 and the voice actor 61 in real time. Then, the moving image generating / outputting system 10 generates a character moving image that is a moving image of the character that moves in accordance with the detected actions of the voice actor 60 and the voice actor 61 in real time, and synthesizes the voice of the voice actor 60 and the voice actor 61 into the character video in real time. And distributed through the network 9.

  The user terminal 80 and the user terminal 81 receive distribution of moving images from the moving image generation / output system 10 through the network 9. Further, the user terminal 80 transmits data such as a text message input from the user 90 to the moving image generation / output system 10 through the network 9. The user terminal 81 transmits data such as a text message input from the user 91 to the moving image generation / output system 10 through the network 9.

  When receiving the text message acquired from at least one of the user terminal 80 and the user terminal 81, the moving image generation / output system 10 displays the received text message. The voice actor 60 and the voice actor 61 see the message displayed on the moving image generating / outputting system 10 and perform a reaction on the message with an action and voice. When the voice actor 60 and the voice actor 61 react with motion and voice, the video generation / output system 10 moves in accordance with the motion performed by the voice actor 60 and voice actor 61, and the character video including the voice generated by the voice actor 60 and voice actor 61 is displayed. Generate and distribute through the network 9.

  According to the interactive system 5, the moving image generating / outputting system 10 can deliver a moving image with a live feeling in which a character reacts in real time with motion or voice in response to a message issued by the user 90 or the user 91. Therefore, the user 90 and the user 91 can enjoy chatting as if they were talking to a character.

  FIG. 2 schematically illustrates an example of a block configuration of the moving image generation / output system 10. The moving image generation / output system 10 includes a camera 20, a camera 21, a motion sensor 30, a motion sensor 31, a camera 40, a display device 50, a display device 51, a moving image processing device 100, and a moving image processing device 200. And a server 300.

  First, taking the case of generating a character moving image that moves according to the movement of the voice actor 60, the operation of each part of the moving image generating / outputting system 10 will be described.

  The camera 20 is provided at a position where the face of the voice actor 60 can be imaged. The camera 20 continuously captures the face of the voice actor 60. The camera 20 outputs a moving image generated by continuously capturing the voice actor 60 to the moving image processing apparatus 100. The camera 20 may be a fixed camera that is fixedly provided at a position where the face of the voice actor 60 is imaged.

  The motion sensor 30 outputs information indicating the body movement of the voice actor 60. The motion sensor 30 is provided on the body part of the voice actor 60. Examples of the body part of the voice actor 60 provided with the motion sensor 30 include the head, hands, arms, and feet of the voice actor 60. The motion sensor 30 continuously detects the position and orientation of the motion sensor 30 and outputs information indicating the detected position and orientation of the motion sensor 30 to the moving image processing apparatus 100.

  The microphone 70 is attached to the voice actor 60. The microphone 70 outputs an audio signal generated by the sound produced by the voice actor 60 to the moving image processing apparatus 100.

  The moving image processing apparatus 100 includes a face movement detection unit 110, a body movement detection unit 120, a storage unit 140, an audio acquisition unit 150, a camera information acquisition unit 160, and a moving image generation output unit 180. The moving image generation / output unit 180 includes a moving image generation unit 130 and a moving image output unit 170. The face motion detection unit 110, the body motion detection unit 120, the moving image generation unit 130, the sound acquisition unit 150, the camera information acquisition unit 160, and the moving image output unit 170 are realized by a processor such as an MPU, for example. The storage unit 140 is realized by a non-volatile recording medium such as a hard disk.

  The face motion detection unit 110 continuously detects the motion of the facial parts of the voice actor 60 from the continuously input moving image of the voice actor 60. Specifically, the face motion detection unit 110 continuously detects the motion of the facial part of the voice actor 60 from the moving image output from the camera 20. The face part is a moving element. The facial parts are, for example, the mouth and eyes. The movement of the facial part includes, for example, a time change of the mouth size, a time change of the mouth shape, a time change of the eye size, a time change of the eye shape, and a time change of the eye position.

  The body movement detection unit 120 continuously detects the movements of body parts other than the face of the voice actor 60. Specifically, the body motion detection unit 120 is based on the information output from the motion sensor 30 and the part of the body part where the motion sensor 30 is provided, the head of the voice actor 60 provided with the motion sensor 30, Detect the position and orientation of hands, arms, legs, etc.

  The storage unit 140 stores the virtual three-dimensional information of the first character. The character is, for example, a human character. The character may be an animal character. The character may be an inorganic character or a virtual character. The character handled by the moving image generation / output system 10 is preferably a personified character, but may be a character that is not personified. Note that the characters stored in the storage unit 140 may be collectively referred to simply as “characters”.

  The storage unit 140 further stores three-dimensional information of the character background. The three-dimensional information is information for rendering an image of a character or a background, for example, information describing the shape, color, texture of the object surface, etc.

  The moving image generation / output unit 180 generates a character moving image in which the character moves according to the movement of the voice actor 60, and outputs the generated character moving image in synchronization with the input of the moving image of the voice actor 60. Specifically, the moving image generation output unit 180 outputs a character moving image including the voice of the voice actor 60 in synchronization with the input of the voice actor 60's moving image. The operations of the moving image generation unit 130 and the moving image output unit 170 included in the moving image generation / output unit 180 will be specifically described.

  The moving image generation unit 130 detects the movement of the facial part continuously detected by the face movement detection unit 110, the movement of the body part continuously detected by the body movement detection unit 120, and the character stored in the storage unit 140. Based on the three-dimensional information, a character animation in which the body part other than the face part of the first character and the face of the first character moves according to the movement of the face part and body part of the voice actor 60 is generated and generated. The character video is output in synchronization with the input of the voice actor 60 video. Based on the background three-dimensional information stored in the storage unit 140, the moving image generation unit 130 uses the first character face in the background foreground represented by the three-dimensional information stored in the storage unit 140. A character moving image in which a body part other than the part and the face of the first character moves according to the movement of the facial part and body part of the voice actor 60 is generated. The moving image generation unit 130 generates a character moving image including a plurality of moving image forming images by sequentially generating a plurality of moving image forming images constituting the character moving image by rendering. Note that the moving image constituent image may be a frame image forming a moving image, for example. The moving image generating unit 130 outputs the generated character moving image to the moving image output unit 170. As described above, the moving image generation unit 130 is a functional block that generates a character moving image in which a character moves in accordance with the movement of the voice actor 60 and outputs the generated character moving image in synchronization with the input of the moving image of the voice actor 60.

  The voice acquisition unit 150 continuously acquires the voice generated by the voice actor 60 in synchronization with the input of the moving picture of the voice actor 60. Specifically, the voice acquisition unit 150 acquires a voice signal of the microphone 70 attached to the voice actor 60. The sound acquisition unit 150 sequentially outputs the continuously acquired sound to the moving image output unit 170. The video output unit 170 synthesizes and outputs the voice continuously acquired by the voice acquisition unit 150 into a character video. The video output unit 170 synthesizes the voice continuously acquired by the voice acquisition unit 150 in synchronization with the character video. The moving image output unit 170 outputs a character moving image synthesized with sound to the server 300.

  The camera 40 continuously captures the voice actor 60. The camera 40 can change an imaging direction, a position, and an angle of view by an operation of an operator 42 who operates the camera 40, for example. Information indicating the imaging direction, position, and angle of view of the camera 40 is continuously output to the moving image processing apparatus 100. Note that the camera 40 may present a moving image generated by capturing an image to the operator 42. The camera 40 does not have to output a moving image obtained by imaging to the moving image processing apparatus 100.

  The camera information acquisition unit 160 continuously acquires information indicating the imaging direction, position, and angle of view of the camera 40. Specifically, the camera information acquisition unit 160 acquires information continuously output from the camera 40. Based on the information indicating the imaging direction of the camera 40 acquired by the camera information acquisition unit 160, the video generation unit 130 generates a video of a character whose direction matches the direction of the voice actor 60 from the direction of the imaging direction of the camera 40. . Thereby, the moving image generation unit 130 generates a character moving image in which the angle changes in accordance with the change in the imaging direction of the camera 40. Further, the moving image generation unit 130 matches the direction of the voice actor 60 when viewed from the position of the camera 40 in the direction of the imaging direction of the camera 40 based on the information indicating the position of the camera 40 acquired by the camera information acquisition unit 160. A moving image of the first character in the orientation is generated. Further, based on the information indicating the angle of view of the camera 40 acquired by the camera information acquisition unit 160, the moving image generation unit 130 generates a moving image of the first character having a magnification corresponding to the angle of view. As described above, the moving image generation unit 130 generates a character moving image in which the imaging direction, position, and angle of view change according to the operation of the operator 42 of the camera 40.

  In the moving image generating / outputting system 10, the character moving image is generated in a so-called real time. For example, in the video generation / output system 10, each process in the face motion detection unit 110, body motion detection unit 120, audio acquisition unit 150, camera information acquisition unit 160, video generation unit 130, and video output unit 170 is a real-time process. Done. For example, the moving image composition image corresponding to the character image is output from the moving image generation unit 130 within a predetermined time from the time when the camera 20 repeatedly performs the exposure operation. Thereby, the character animation according to the movement of the voice actor 60 is output in real time.

  The server 300 distributes the character moving image output from the moving image output unit 170 to the user terminal 80 and the user terminal 81 via the network 9. In addition, the server 300 acquires a text message from the user terminal 80 and the user terminal 81 via the network 9. The display device 50 is provided at a position where the voice actor 60 can see. The server 300 displays the message acquired from the user terminal 80 and the user terminal 81 on the display device 50.

  According to the moving image processing apparatus 100 described above, when the voice actor 60 performs a reaction by looking at a message input at the user terminal 80, the reaction is reflected in the character video in real time and distributed to the user terminal 80. . Therefore, the user 90 and the user 91 can enjoy a chat with a live feeling.

  Next, the operation of each part of the moving image generation / output system 10 will be described by taking up a case where a character moving image in which two characters corresponding to the voice actor 60 and the voice actor 61 move is taken up. Note that the processing related to the first character corresponding to the voice actor 60 is as described above, and thus the description thereof is omitted. The difference between the process related to the movement of the character corresponding to the voice actor 61 and the process related to the movement of the character corresponding to the voice actor 60 will be mainly described.

  The camera 21 is provided at a position where the face of the voice actor 61 can be imaged. The camera 21 continuously captures the face of the voice actor 61. The camera 21 outputs a moving image generated by continuously capturing the voice actor 61 to the moving image processing apparatus 200. The camera 21 may be a fixed camera that is fixedly provided at a position where the face of the voice actor 61 is imaged.

  The motion sensor 31 outputs information indicating the body movement of the voice actor 61. The motion sensor 31 is provided on the body part of the voice actor 61. Examples of the body part of the voice actor 61 provided with the motion sensor 31 include the head, hands, arms, and feet of the voice actor 61. The motion sensor 31 continuously detects the position and orientation of the motion sensor 31 and outputs information indicating the detected position and orientation of the motion sensor 31 to the moving image processing apparatus 200.

  The microphone 71 is attached to the voice actor 61. The microphone 71 outputs an audio signal generated by the sound produced by the voice actor 61 to the moving image processing apparatus 100. The display device 51 is provided at a position where the voice actor 61 can see. The server 300 displays the message acquired from the user terminal 80 and the user terminal 81 on the display device 51.

  The function of the face motion detection unit 210 is substantially the same as the function of the face motion detection unit 110. Therefore, regarding the function of the face motion detection unit 210, differences from the function of the face motion detection unit 110 will be mainly described, and description of functions that are substantially the same as the functions of the face motion detection unit 110 will be omitted. The face motion detection unit 210 continuously detects the movement of the facial parts of the voice actor 61 from the video of the voice actor 61 continuously input in synchronization with the video of the voice actor 60. Specifically, the face motion detection unit 210 continuously detects the motion of the facial part of the voice actor 61 from the moving image output from the camera 21. The face movement detection unit 210 outputs information indicating the detected movement of the face part to the moving image processing apparatus 100.

  The body movement detection unit 220 continuously detects the movements of the body parts other than the voice actor 61 face. Note that the body movement detection unit 220 has substantially the same function as the body movement detection unit 120. Specifically, the body motion detection unit 220 is based on the information output from the motion sensor 31 and the part of the body part on which the motion sensor 31 is provided, the head of the voice actor 60 on which the motion sensor 31 is provided, Detect the position and orientation of hands, arms, legs, etc. The body movement detection unit 220 outputs information indicating the detected movement of the body part to the moving image processing apparatus 100.

  The storage unit 140 further stores three-dimensional information of the virtual second character. The second character is a character different from the first character.

  The moving image generation unit 130 stores the movement of the facial part continuously detected by the face movement detection unit 110, the movement of the body part continuously detected by the body movement detection unit 120, and the first stored in the storage unit 140. The storage unit 140 stores the three-dimensional information of the character, the movement of the facial part continuously detected by the face movement detection unit 210, the movement of the body part continuously detected by the body movement detection unit 220, and the like. Based on the movement of the face part and body part of the voice actor 60, the body part other than the face part of the first character and the face of the first character moves based on the three-dimensional information of the second character, A character video in which the body part other than the face part of the second character and the face of the second character moves according to the movement of the human face part and the movement of the body part is generated. Synchronized with the input of the video and the video of the second person Forces. Note that the moving image generation unit 130 further uses the first character in the foreground of the background represented by the three-dimensional information stored in the storage unit 140 based on the three-dimensional information of the background stored in the storage unit 140. Body parts other than the face part of the first character and the face part of the first character move according to the movement of the face part and body part of the voice actor 60, and the body part other than the face part of the second character and the face of the second character Generates a character animation that moves according to the movement of the facial parts and body part of the voice actor 61.

  The voice acquisition unit 150 continuously acquires the voice uttered by the voice actor 61 in synchronization with the input of the moving picture of the voice actor 61. Specifically, the voice acquisition unit 150 acquires a voice signal of the microphone 71 attached to the voice actor 61. The voice acquisition unit 150 sequentially outputs the voices obtained by the voice actor 60 and the voice actor 61, which are continuously acquired, to the video output unit 170. The moving image output unit 170 synthesizes and outputs the sound generated by the voice actor 60 and the voice actor 61 into a character moving image. The moving image generating unit 130 synthesizes the sound produced by the voice actor 60 and the voice actor 61 in synchronization with the character moving image.

  Note that the camera 40 continuously captures the voice actor 60 and the voice actor 61. Based on the information indicating the imaging direction of the camera 40 acquired by the camera information acquisition unit 160, the video generation unit 130 is a video of the second character in a direction that matches the direction of the voice actor 61 from the direction of the imaging direction of the camera 40. Is generated. Thereby, the moving image generation unit 130 generates a character moving image in which the angle changes in accordance with the change in the imaging direction of the camera 40. Further, the moving image generation unit 130 matches the direction of the voice actor 61 when viewed from the position of the camera 40 in the direction of the imaging direction of the camera 40 based on the information indicating the position of the camera 40 acquired by the camera information acquisition unit 160. A moving image of the second character in the orientation is generated. Further, based on the information indicating the angle of view of the camera 40 acquired by the camera information acquisition unit 160, the moving image generation unit 130 generates a moving image of the second character having a magnification corresponding to the angle of view.

  According to the moving image generating / outputting system 10, the moving image processing apparatus 200 is responsible for detecting the movement of the facial part of the voice actor 61 and detecting the movement of the body part of the voice actor 61, and the moving picture processing apparatus 100 performs the movement of the facial part of the voice actor 61 and The detection result of the movement of the body part of the voice actor 61 is acquired from the moving image processing apparatus 200. Therefore, the calculation load in the moving image processing apparatus 100 can be reduced. This makes it easy to generate a character animation in real time by rendering based on the three-dimensional information of a plurality of characters corresponding to a plurality of voice actors. In addition, since the moving image generation unit 130 of one moving image processing apparatus 100 is responsible for rendering, it becomes easy to perform integrated rendering of a plurality of characters and backgrounds corresponding to a plurality of voice actors.

  FIG. 3 schematically illustrates an example of a block configuration of the moving image generation / output system 12 according to the second embodiment. The moving image generation / output system 12 includes a camera 20, a camera 21, a motion sensor 30, a motion sensor 31, a camera 40, a display device 50, a display device 51, a moving image processing device 102, and a moving image processing device 202. And a server 300.

  The moving image processing apparatus 102 has a function similar to that of the moving image processing apparatus 100. The moving image processing apparatus 102 includes functional blocks having functions similar to the functional blocks included in the moving image processing apparatus 100. In the video processing device 102, the face motion detection unit 110, the body motion detection unit 120, the video generation unit 130, the storage unit 140, the sound acquisition unit 150, and the camera information acquisition unit 160 are functions described in relation to the video processing device 100. And has similar functions. Therefore, here, differences from the functions described in relation to the moving image processing apparatus 100 will be mainly described, and description of functions similar to those described in relation to the moving image processing apparatus 100 may be omitted. .

  The moving image processing apparatus 202 includes a moving image generation unit 230, a storage unit 240, and a camera information acquisition unit 260 in addition to the functional blocks provided in the moving image processing apparatus 200. In the moving image processing apparatus 102, the face movement detecting unit 210 and the body movement detecting unit 220 have functions similar to the functions described in relation to the moving image processing apparatus 200. Therefore, here, differences from the functions described in relation to the moving image processing apparatus 200 will be mainly described, and description of functions similar to those described in relation to the moving image processing apparatus 200 may be omitted. .

  In the moving image processing apparatus 102, the storage unit 140 stores the three-dimensional information of the first character and the three-dimensional information of the background. The storage unit 140 does not store the 3D information of the second character.

  The moving image generation unit 130 acquires the movement information of the facial part of the voice actor 60 continuously detected by the face movement detection unit 110 and the movement information of the body part of the voice actor 60 continuously detected by the body movement detection unit 120. The moving image generation unit 130 does not acquire motion information regarding the voice actor 61. The movement of the facial part of the voice actor 60 continuously detected by the face movement detection unit 110, the movement of the body part of the voice actor 60 continuously detected by the body movement detection unit 120, and the first stored in the storage unit 140. The first character moving image in which the body part other than the face part of the first character and the face of the first character moves in accordance with the movement of the face part and body part of the voice actor 60 based on the three-dimensional information of the character. Generate. As described above, the moving image generation unit 130 is based on the movement of the facial part of the voice actor 60 continuously detected by the face movement detection unit 110 and the movement of the body part of the voice actor 60 continuously detected by the body movement detection unit 120. The first character is rendered. Note that, similarly to the processing of the moving image generating unit 130 in the moving image processing apparatus 100, the moving image generating unit 130 is based on the information indicating the imaging direction, position, and angle of view of the camera 40 acquired by the camera information acquiring unit 160. The character animation is generated.

  In the moving image processing apparatus 202, the storage unit 240 stores the three-dimensional information of the second character. The moving image generation unit 230 acquires the movement information of the facial part of the voice actor 61 continuously detected by the face movement detection unit 210 and the movement information of the body part of the voice actor 61 continuously detected by the body movement detection unit 220. The moving image generation unit 230 includes the movement of the facial part of the voice actor 61 continuously detected by the face movement detection unit 210, the movement of the body part of the voice actor 61 continuously detected by the body movement detection unit 220, and the storage unit 240. Based on the stored 3D information of the second character, the body part other than the face part of the second character and the face of the second character moves according to the movement of the face part and body part of the voice actor 61. A second character moving image is generated. As described above, the moving image generation unit 230 is based on the movement of the facial part of the voice actor 61 continuously detected by the face movement detection unit 210 and the movement of the body part of the voice actor 61 continuously detected by the body movement detection unit 220. The second character is rendered. The moving image generation unit 230 outputs the generated second character moving image to the moving image processing apparatus 102.

  Similar to the camera information acquisition unit 160, the camera information acquisition unit 260 continuously acquires information indicating the imaging direction, position, and angle of view of the camera 40. Similar to the moving image generating unit 130 in the moving image processing apparatus 100, the moving image generating unit 230 generates the second character moving image based on the information indicating the imaging direction, position, and angle of view of the camera 40 acquired by the camera information acquiring unit 260. Generate.

  In the video processing device 102, the video output unit 170 generates a single character video by combining the first character video generated by the video generation unit 130 and the second character video generated by the video generation unit 230. To do. In addition, the moving image output unit 170 uses the background three-dimensional information stored in the storage unit 140 and uses the three-dimensional information stored in the storage unit 140 for the background of the first character. The body parts other than the face parts and the face of the first character move according to the movement of the face parts and body parts of the voice actor 60, and the body parts other than the face parts of the second character and the second character face are voice actors 61. Character animation that moves according to the movement of the face part and body part is generated. Also, the moving image output unit 170 synthesizes the voice of the voice actor 60 and the voice actor 61 acquired by the voice acquisition unit 150 into a character moving image and outputs it.

  The moving image generating / outputting system 12 in the second embodiment can also generate character moving images in real time by performing rendering based on the three-dimensional information of a plurality of characters corresponding to a plurality of voice actors.

  In the above description, the processing described as the operation of the moving image processing apparatus 100 is realized by the processor controlling each hardware (for example, hard disk, memory, etc.) included in the moving image processing apparatus 100 according to the program. As described above, at least a part of the processing of the moving image processing apparatus 100 described in relation to the moving image processing apparatus 100 according to the present embodiment is performed by the processor operating the program according to the program to control each hardware, so that the processor, the hard disk It can be realized by the hardware and the program including the memory and the like operating together. That is, the process can be realized by a so-called computer. The computer may load a program for controlling execution of the above-described processing, operate according to the read program, and execute the processing. The computer can load the program from a computer-readable recording medium storing the program.

  Similarly, the processing described as the operation of the moving image processing apparatus 200 can be realized by the processor controlling each hardware of the computer according to the program. The processing described as the operation of the moving image processing apparatus 102 can be realized by the processor controlling each hardware of the computer according to the program. The processing described as the operation of the moving image processing apparatus 202 is realized by the processor controlling each hardware of the computer according to the program.

  The functions of the face motion detection unit 110, the moving image generation unit 130, the face movement detection unit 210, and the moving image generation unit 230 can be implemented on a computer as a whole by a game engine “Unity” of Unity Technologies. The processing of the face motion detection unit 110 and the face motion detection unit 210 can be implemented in a computer by incorporating a plug-in for detecting the motion of the face into Unity.

  In the above description, the interactive system 5 includes two user terminals 80 and user terminals 80. However, the number of user terminals is not limited to two. The number of user terminals may be 1 or 3 or more. Similarly, the number of characters and voice actors is not limited to 2, and may be 1 or 3 or more.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

5 Interactive system 9 Network 10, 12 Video generation / output system 20 Camera, 21 Camera 30 Motion sensor, 31 Motion sensor 40 Camera 42 Operator 50 Display device, 51 Display device 60 Voice actor, 61 Voice actor 70 Microphone, 71 Microphone 80 User terminal, 81 User terminal 90 User, 91 User 100, 102 Movie processing device 110 Face motion detection unit 120 Body motion detection unit 130 Movie generation unit 140 Storage unit 150 Audio acquisition unit 160 Camera information acquisition unit 170 Movie output unit 180 Movie generation output unit 200 202 Moving image processing apparatus 210 Face motion detecting unit 220 Body motion detecting unit 230 Moving image generating unit 240 Storage unit 260 Camera information acquisition unit 300 Server

Claims (5)

A first face motion detection unit that continuously detects the motion of the facial parts of the first person from a continuously input video of the first person;
A first body movement detection unit that continuously detects movement of a body part other than the face of the first person;
A storage unit storing the three-dimensional information of the virtual first character;
The movement of the face part continuously detected by the first face movement detection unit, the movement of the body part continuously detected by the first body movement detection unit, and the storage unit storing the first movement Body parts other than the face parts of the first character and the face of the first character according to the movement of the face parts and the body part of the first person based on the three-dimensional information of the one character A moving image generating apparatus comprising: a moving image generating / outputting unit that generates a moving character moving image and outputs the generated character moving image in synchronization with an input of the moving image of the first person. A voice acquisition unit for continuously acquiring the voice uttered by the first person in synchronization with the input of the first person's video;
The moving image generation apparatus according to claim 1, wherein the moving image generation / output unit synthesizes and outputs the sound continuously acquired by the sound acquisition unit to the character moving image. The storage unit further stores three-dimensional information of the virtual second character,
The moving image generating device
A second face motion detection unit for continuously detecting the motion of facial parts of the face of the second person from the second person's video continuously input in synchronization with the first person's video; ,
A second body motion detector that continuously detects the motion of the body other than the face of the second person,
The moving image generation output unit includes: the movement of the facial part continuously detected by the first face movement detection unit; the movement of the body part continuously detected by the first body movement detection unit; and the storage unit. Stored in the first character, the movement of the facial part detected by the second facial motion detection unit, and the continuous detection of the second body motion detection unit. Based on the movement of the body part and the three-dimensional information of the second character stored in the storage part, the first part is moved according to the movement of the facial part and the body part of the first person. Body parts other than the character's face part and the face of the first character move, and the face part of the second character and the first part according to the movement of the face part and the body part of the second person. Generating the character animation in which the body part other than the face of the character 2 moves, The character videos form, the first person video and moving image generating apparatus according to claim 1 or 2, output in synchronization with video input of the second person. A camera unit that continuously images the first person;
A camera information acquisition unit that continuously acquires information indicating the imaging direction of the camera unit;
The moving image generation / output unit is configured to match the first person in a direction that matches the direction of the first person viewed from the camera unit based on information indicating an imaging direction of the camera unit acquired by the camera information acquisition unit. The moving image generating apparatus according to any one of claims 1 to 3, wherein the moving image of the character is generated to generate the character moving image in which an angle changes according to a change in an imaging direction of the camera unit.   The program for functioning a computer as a moving image production | generation apparatus as described in any one of Claim 1 to 4.

Images