JP2008135923A - Production method of videos interacting in real time, video production device, and video production system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an inexpensive and simple production method and a system of videos interacting in real time. <P>SOLUTION: A production method, a production device, and a production system of videos interacting in real time are disclosed. The production system of videos includes: a display device provided with a screen 500; a computer provided with at least one processor, a memory, and a program; and a photographing device. The program provides media contents and special effect command scripts. When the photographing device photographs an on-site staff video 401, makes it match with special effect command scripts and reproduces it, media contents are displayed on the screen 500 in real time and made to interact with the on-site staff video 401 on the screen 500. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a dynamic video production method, production apparatus, and production system, and more particularly, to a video production method, production apparatus, and production system that interact in real time.

  As imaging devices such as digital cameras, internet cameras, and camera-equipped mobile phones have become inexpensive and widely spread, the fusion of home computers and electronic products has become an undeniable trend. However, many of the current video multimedia applications are mainly still images, and usually focus on photography storage and file management, and basic image processing and simple image composition functions are provided. . In dynamic video equipment, applications related to simple recording, file conversion and playback are the mainstream, sometimes transmitting real-time video in combination with a network, but giving creative value to multimedia content There are many parts that are lacking for doing or remodeling. In addition, some game software attempts to match the user's limb movements to interactive games. Sex is also limited.

  In addition, special effects often seen in television content are not only high hardware and software costs required for production, but also require specialized knowledge, which is a highly specialized area. In addition, the actor needs to perform by imagining a partner that does not exist on the spot, which is heavy and inconvenient in production.

In view of the above complexity of general digital content production, the inventors of the present invention have devised a video production method, a video production device, and a video production system that interact in real time. A machine interface can be provided, and creators can create rich digital contents at low cost.
JP-A-10-13738

  An object of the present invention is to provide an inexpensive and simple real-time video production method, video production apparatus, and video production system.

  In order to solve the above problems, the present invention provides a simple and natural man-machine interface, and a creator can create rich digital contents at a low cost.

  The present invention provides a video production method, a video production apparatus, and a video production system that interact in real time, and includes a display device including a screen, a field worker, at least one processor, a memory, and a program. Computation equipment and photographing device. The program provides media content and special effects command scripts. The imaging device captures field personnel video and matches the special effect command script. Finally, the media content and the matched field personnel video are combined with the special effect command script and displayed on the screen in real time.

  That is, the invention of claim 1 provides a step of preparing a screen, a step of capturing a video in real time and displaying it on the screen, a step of generating an image of a virtual object and displaying it on the screen, A method for producing an image that interacts in real time, comprising the step of interacting an image with the image.

According to a second aspect of the present invention, the method of photographing the video is a method of photographing a video in front of the internet camera using an internet camera. Is the method.
The invention according to claim 3 is characterized in that the step of generating the image of the object further includes the step of generating an image of the corresponding object in a preselected mode. It is a production method of the image to be performed.
The invention of claim 4 is the real-time interactive video production method according to claim 1, wherein the interacting step includes a step of recognizing the video position.
The invention of claim 5 is the real-time interactive image production method according to claim 1, wherein the interacting step includes a step of tracking the variation of the image.
According to a sixth aspect of the present invention, in the method for producing a video that interacts in real time according to the first aspect, the step of interacting includes a step of generating a special effect script on the video.
The invention of claim 7 is the real-time interactive video production method according to claim 6, wherein the special effect script is selected from a special effect command collection described in a script language.
The invention of claim 8 is the real-time interactive video production method according to claim 1, wherein the object is selected from media content.
The invention of claim 9 is characterized in that the step of preparing the screen includes a step in which a video provider exists in front of the photographing apparatus, and the photographing apparatus and the screen are electrically connected. 1 is a method of producing a video that interacts in real time according to 1.
The invention of claim 10 is the real-time interactive video production method according to claim 1, wherein the step of generating an image of the object includes the step of the video following feature tracking of the video. .
The invention according to claim 11 is the method for producing a video that interacts in real time according to claim 1, wherein the step of generating the image of the object includes a step in which the video follows posture analysis and recognition.

  The invention of claim 12 stores a plurality of programs read by a media processing device, the media processing device inputs data including background data and real-time video based on the plurality of programs, and the data Recognizing, a step of tracking a changed portion of the data, a step of preparing media content, a step of combining the media content and the data, and a combination of the media content and the data A video production device that interacts in real time, characterized by executing a display step.

According to a thirteenth aspect of the present invention, the steps of storing a plurality of programs to be read by the media processing apparatus and preparing the media content further include a step of reading the media content and a step of decoding the media content 13. The apparatus for producing a video that interacts in real time according to claim 12, characterized in that:
According to a fourteenth aspect of the present invention, the step of storing a plurality of programs read by the media processing device and combining the media content and the data further includes the step of retracking the changed portion of the data. 13. The video production apparatus that interacts in real time according to claim 12.
The invention of claim 15 stores a plurality of programs read by the media processing device, further reads a special effect, and reprocesses the composite of the media content, the data, and the special effect. 13. The real-time interactive video production apparatus according to claim 12, comprising a step and a step of displaying a combination of the media content, the data and special effects.
According to a sixteenth aspect of the present invention, the step of storing a plurality of programs read by the media processing apparatus and reading the special effect further includes inserting a special effect into the background data. 16. An apparatus for producing an image that interacts in real time according to claim 15.

  The invention of claim 17 includes: a display device including a screen; a computing device including at least one processor, a memory, and a plurality of readable programs including media content and special effect command scripts; and a photographing device that receives video. A real-time interactive video production system, wherein the media content and the video are displayed on the screen while interacting in real time by combining the special effect command script processing and the media content. It is.

The invention according to claim 18 is the real-time interactive video production system according to claim 17, wherein the display device is a liquid crystal monitor.
A nineteenth aspect of the present invention is the real-time interactive video production system according to the seventeenth aspect, wherein the computing device is a computer.

  According to the video production method, video production apparatus, and video production system of the present invention that interact in real time, the concept of interactive special effect tracks is used, and other than video tracks and audio tracks in raw video. In addition, special effects can be added in real time. The difference from general video special effects is that the special effects specified in the present invention can be generated in real time, and it is not necessary to select the target to be applied in advance, and interactive changes can be made. is there.

  A detailed description of the invention is given below with reference to the figures. When describing embodiments of the present invention, it is not a general ratio but is locally expanded for convenience of explanation, but is not intended to limit the present invention.

  The present invention relates to a video production method, a production device (video storage device that interacts in real time), and a production system that interact in real time, a display device including a screen, at least one processor, a memory, and a program. A computing device and a photographing device are provided. The program in the computing device provides media content and special effect command scripts. Media content is displayed on the screen in real time as the photographic device captures the scene video and plays it in alignment with the special effect command script. Media content includes virtual people and can interact with on-site video in real time.

  In one embodiment of the present invention shown in FIG. 1, a device having a processor and memory such as a personal computer, a set top box, a game console or a mobile phone, a display device such as a cathode ray tube display, a liquid crystal display or a plasma display, and a photographing device are provided. In this embodiment, the computer main frame 100, the liquid crystal display 101, and the Internet camera 102 capable of transferring the captured image via a communication line such as the Internet are employed. In this embodiment, the computer main frame 100, the liquid crystal display 101, and the internet camera 102 are connected to each other in a wired or wireless manner. However, there is no limitation on this configuration. The photographing device can be combined with a main frame and a display joined together such as a notebook personal computer or a tablet computer.

  Next, as for live recording, as shown in FIG. The internet camera 102 captures the video of the on-site personnel 104 and displays it on the screen 103 of the liquid crystal display 101. On-site personnel image 105 is displayed on screen 103. The on-site personnel video 105 displays the on-site personnel 104 in front of the Internet camera 102 at that time in real time. In one selection mode in one embodiment, the virtual person 106 and the on-site personnel image 105 interact. What has to be explained here is that the on-site personnel 104 is displayed on the screen 103 in real time and becomes the on-site personnel image 105. The real time here means that the operation of the on-site personnel 104 and the on-site personnel video 105 are synchronized. In addition, the method of interaction between the scene with the on-site personnel 104 and the virtual person 106 and the on-site personnel video 105 is not set in advance, and the user can select the menu or a similar interface. The selection mode can be an application program in which the program is produced, and is stored in a memory or the like in the computer main frame 100.

  FIG. 2 is a diagram illustrating a file architecture according to an embodiment of the media processing apparatus of the present invention. The preselected mode is composed of main content and special effect script file. As an example, first, media content 201 and scenario are set, and multimedia video content such as popular music, old songs or classical music is set. Generate. Next, a special effect command script 202 provided with interaction effects corresponding to each other is designed, which includes basic data such as a time parameter, a relative space parameter, a special effect type, a special effect application target, etc., in a specific language. Is saved as a command file. Users can design different themes according to factors such as gender, age, etc., and can be combined with different special effects. That is, when describing the same main content, a plurality of special effect commands can be installed. For example, when playing popular music, a virtual person is read into a special effect script corresponding to each other, and data matching is performed during playback. The method to do can be adopted. First, the user downloads media content 201 and special effect command script 202. Next, on-site personnel image capturing 203 is performed, combined with real-time image capturing of the photographing device, and after capturing on-site personnel image 105 as shown in FIG. Finally, the video and special effect command script 202 captured in real time after serialization by the dynamic video synthesis 204 and the media content 201 are synthesized, and the on-site personnel are displayed in the virtual world.

  3 and 4 are schematic views showing a state in which the on-site personnel captured by the media processing apparatus and the virtual world are synthesized and reproduced. The display device displays one screen, which is a screen shot of a field personnel photographed by a photographing device (not shown) and displayed in real time on the screen 400 of the display device. When the readable program of the present embodiment is executed, a preselected mode can generate a virtual person 402 such as a person, a god, an anime character, or a monster.

  At this time, the virtual person 402 can interact with the on-site personnel image 401 and can be displayed on the screen 400 in real time. As shown in FIG. 4, the virtual person 402 can have many operations and special effects, and the on-site personnel image 401 can also perform small exercises such as left and right movement. For example, the virtual person 402 can be placed on the shoulder of the on-site personnel image 401 or the cheek of the on-site personnel image 401 can be kissed. At this time, the blush effect 501 and the rejoicing effect 502 can be generated in the on-site personnel image 401 in response to the operation of the virtual person 402. As another example, the virtual person 402 can apply magic to the on-site personnel image 401. At this time, the ear 503 is attached to the head of the on-site personnel image 401 in response to the action of the virtual person 402, When the head of the image 401 is slightly swung, the ear 503 is swung accordingly. That is, the virtual person 402, the on-site personnel image 401, and various effects interact in real time. That is, no matter where the site personnel image 401 moves on the screen 500, the ear 503 is forever on the head of the site personnel image 401. It should be explained here that, from a technical point of view, the recognition technique can be used first, first the hair position of the on-site personnel image 401 is confirmed, and then the moving position of the head using the tracking technique. Then, the ear 503, which is a special effect, is added on the hair, and by repeating such recognition and tracking, a real-time interaction effect between a person and a virtual object is generated on the screen.

  Regarding site personnel, site personnel images can be divided into half-body mode and whole-body mode. In the half-body mode, the head and shoulders of the site personnel are displayed on the screen. Occupies seven. What must be explained here is that real-time performance and accuracy are difficult to achieve at the same time in the construction of interactive digital content, but in the present invention, suitable processing and Adjustments can be made. For example, when applied to dynamic facial magnified images, facial feature detection and accurate localization are primarily considered. In the operation mode, the estimation of the simple parameters for the entire operation is mainly considered. When in whole body mode, tracking of area motion and recognition of configuration is the focus of the interaction module.

  Regarding the method of using the interaction between the virtual image and the field personnel, feature detection, feature tracking, posture analysis, posture recognition, and the like analyze the virtual image and the motion of the field personnel. Feature detection is capturing in which a low level (feature point) and a high level (face features such as eyes and mouth) are considered based on the properties of the application target. The feature matching method can be divided into an implicit rule and an explicit rule. An explicit feature matching rule finds a one-to-one correspondence between features. Implicit feature matching rules represent relationships between features in the previous and next frames by a method such as parameter or transformation. For example, explicit rules and low-level features can be feature point matching (limb tracking), explicit rules and high-level features can be facial expression analysis, and implicit rules and low-level features are density optical flows. Implicit rules and high-level features can be facial organ detection and localization.

  In the feature detection, the following method and apparatus are used, and highly efficient and accurate face detection and organ localization are performed. FIG. 5 is a continuous diagram of the initial selection of the horizontal edge density calculation. In the initial detection, the positions of the eyes and the mouth are initially estimated by the density of the horizontal edge of the gray scale image. Candidate area 601 is the position of the selected eye and mouth. Next, in many candidate areas 601, further sieving is performed using the relative position of the organ and the ratio relationship. Finally, the position is confirmed by eyeball detection. As an example, the skin color may be used as an auxiliary material for the judgment basis. As an example, the localization of organs such as the nose, eyebrows, and ears is estimated by a ratio relationship. The contour of the face is displayed by an elliptic equation. As an example, under full body mode, skin color and hair feature site detectors can provide rapid detection and relatively low level but grouped features for the rest of the human body Depicted by dots.

  With respect to feature tracking, one embodiment of feature tracking in the half-body operation mode focuses on sustained localization of facial organs and motion parameter estimation of the entire area. As an example of the whole-body operation mode, features are compared and tracked by a grouped graph matching method, and the number of feature points is adjusted according to a change state of a calculation resource. It should be explained that in this embodiment, the video capturing device does not track the face of the field personnel, but the face of the site personnel must adapt to the shooting position of the video capturing device. In this way, it is possible to eliminate the need to consider posture estimation.

  As an example of determining the shape of an object in a stationary state with respect to posture analysis and recognition, shape matching can be used, and its related technology, such as Shape Context, can be used. The calculation method can also be an Elastic Matching calculation method, and in combination with the concept of multiple analysis degree, a small degree of deformation and shielding effect is allowed. As an example of continuous motion analysis and recognition, using a hierarchical optical flow tracking method (Pyramidal Optical Flow), first calculate the moving direction and speed of the human body, as an example of the ordering method of usage time, It can be a hidden Markov model or a recurrent neural network, etc., and the semantic content represented by the operation is analyzed.

  FIG. 6 is a flowchart showing an embodiment of the media processing apparatus, its system, and software operation of the present invention. Application program trigger 701, hardware detection 751, warning message 731, application program termination 704 and problem message 732 are used by the program to authenticate hardware demand. A warning message 731 is generated when the hard detection 751 finds a problem, and a problem message 732 is generated otherwise. The warning message 731 informs the user that the hardware equipment necessary for hardware detection is not installed or cannot be operated. For example, a message that the photographic lens is not installed and the photographic lens is not installed. Inform the message that it is complete. The problem message 732 informs the user that the subject has left the lens and is convenient for the next shooting step. The next is preprocessing, where background data collection 706 is performed and stored in the internally stored background data 707, and then a problem message 733 is generated, the purpose of which is to bring the user into the shootable range. . For example, the user is allowed to enter the photographing possible range by the welcome screen, and the video is displayed on the display screen.

  Here, the recognition 709 can recognize the entire face and limbs. The motion tracking 710 can now detect the motion of the face and the entire limb. Further, the media data 761 can include an AVI or MPEG format which is a type of extended file. As an example, the media data 761 may be a compressed file such as a DLL file. Next, the media data 711 and the decryption 713 of the media data are read. Recognition 709, motion tracking 710 and internally stored background data 707 are combined with the following steps to generate a dynamic composite video.

  The media processing device displays composite media data display 716 after combining 714 and re-tracking operation 715 of the camera image and media data (virtual object image). The motion retrack 715 once again detects background and video changes. Next, it is determined in step 752 whether or not to read a special effect, and if so, the process proceeds to a step 718 for reading a special effect insertion. The special effect insertion reading 718 may be, as an example, a special effect grade of Ceffect. Next, the process proceeds to step 753 as to whether or not to save the composite media data. Next, proceed to step 754 whether the time expires, and if so, proceed to the step of reprocessing stored composite media data 722, which can be a JPEG file format or CStyle grade, as an example. Finally, display 723 of the stored composite media data reprocessed and end 724 of the application program are performed (a video storage device or production device that interacts in real time).

  What has to be explained here is that the media processing device can display the composite media data 716 on the screen after the re-tracking 715 of the camera image and the media data by the media processing device. Proceed to read effect insertion 718, save composite media data 720, and then proceed to composition 714 of the camera image and media data, thus producing a real-time effect. As shown in FIGS. 3 and 4, the virtual person 402 can know the positions of the shoulders and cheeks of the on-site personnel image 401 after re-tracking the movement 715. The special effect blush effect 501 can add the blush effect 501 shown in FIG. 3 in real time after the composite media data storage 720 and the operation re-tracking 715. Further, the blush effect 501 can be generated at an accurate position regardless of where the cheek moves by the retracking operation 715.

  The above description shows one embodiment of the production method, production apparatus, and production system software operation process that interacts in real time according to the present invention. In the present invention, in a personal computer, a set top box, a game console, a mobile phone, or the like, the media processing apparatus has means for executing the above-described steps, and each step can be executed by the respective means. In addition, two users can play with each other. Two users are connected by a network such as the Internet or an intranet, select a virtual person to the other party or themselves, issue a command at one end, manipulate the virtual person at the other end, generate various visual special effects, It can be displayed on the other party and his monitor.

  In the above-described embodiment of the present invention, the interactivity of the application software and the reality of the composite effect are considered together, the design of the special effect module and the interaction module are considered together, and integrated into one package. In this way, when the media content is laid out, the processing is first completed, and the system resources can be fully utilized for expressing the reality at the time of interaction.

  The above description shows an embodiment of the present invention and does not limit the scope of the claims of the present invention. All changes or modifications within the scope of the present invention are included in the scope of the claims.

FIG. 2 is a schematic diagram illustrating an architecture according to an embodiment of the present invention. FIG. 3 is a block diagram illustrating a file architecture according to one embodiment of the present invention. It is a schematic diagram which shows the state which synthesize | combined and reproduced the captured field personnel and the virtual world. It is another schematic diagram which shows the state which synthesize | combined and reproduced the captured field personnel and the virtual world. It is a continuation figure of the initial selection of the density calculation of a horizontal edge. It is a flowchart which shows one Example of the driving | operation of the software of this invention.

Explanation of symbols

100 computer main frame 101 liquid crystal display 102 network camera 103, 400, 500 screen 104 field personnel 105, 401 field personnel images 106, 402 virtual person 201 media content 202 special effect command script 203 field personnel image capturing 204 dynamic image Compositing 501 Blush effect 502 Joy effect 503 Ear 601 Candidate area 701 Application program trigger 751 Hardware detection 731 Warning message 704 End of application program 732 and 733 Problem message 706 Collection of background data 707 Internally stored background data 709 Recognition 710 Motion tracking 711 Read media data 761 Media data 713 Decode media data 714 Camera image and media Compositing with data 715 Re-tracking operation 716 Displaying composite media data 752 Whether to read special effect 718 Reading special effect insertion 753 Whether to save composite media data 720 Save composite media data 754 Time expires Whether or not 722 Reprocessing of stored composite media data 723 Display of reprocessed stored composite media data 724 End of application program

Claims (19)

Steps to prepare the screen,
Taking a picture in real time and displaying it on the screen;
Generating an image of a virtual object and displaying it on the screen;
A method for producing a video that interacts in real time, comprising the step of interacting the image of the object and the video.   2. The method for producing an image that interacts in real time according to claim 1, wherein the method of photographing the image is a method of photographing an image in front of the internet camera using an internet camera.   2. The method of producing a real-time interactive video according to claim 1, wherein the step of generating an image of the object further includes a step of generating an image of the corresponding object in a preselected mode.   2. The method for producing an image that interacts in real time according to claim 1, wherein the step of interacting includes a step of recognizing the position of the image.   2. The method of producing a video that interacts in real time according to claim 1, wherein the step of interacting includes a step of tracking fluctuations of the video.   2. The method of producing a video that interacts in real time according to claim 1, wherein the step of interacting includes a step of generating a special effect script on the video.   7. The method of producing a video that interacts in real time according to claim 6, wherein the special effect script is selected from a special effect command collection described in a script language.   The method according to claim 1, wherein the object is selected from media content.   The real-time interaction according to claim 1, wherein the step of preparing the screen includes a step in which a video provider exists in front of the photographing device, and the photographing device and the screen are electrically connected. How to make a video to play.   2. The method of producing a real-time interactive video according to claim 1, wherein the step of generating an image of the object includes the step of following the feature tracking of the video.   2. The method of producing a video that interacts in real time according to claim 1, wherein the step of generating an image of the object includes a step in which the video is subjected to posture analysis and recognition. Save multiple programs that are read by the media processing device,
The media processing device is based on the plurality of programs,
Inputting data including background data and real-time video;
Recognizing the data;
Tracking changes in the data;
Preparing media content;
Combining the media content and the data;
A real-time interactive video production apparatus, comprising: a step of displaying a combination of the media content and the data Storing a plurality of programs read by the media processing device;
The step of preparing media content further includes
Reading the media content;
The real-time interactive video production apparatus according to claim 12, further comprising: decrypting the media content. Storing a plurality of programs read by the media processing device;
The step of combining the media content and the data further includes:
The real-time interactive video production apparatus according to claim 12, further comprising the step of re-tracing the changed portion of the data. Storing a plurality of programs read by the media processing device; and
A step to load special effects,
Reprocessing the composite of the media content, the data and special effects;
13. The real-time interactive video production apparatus according to claim 12, further comprising a step of displaying a combination of the media content, the data, and a special effect. Storing a plurality of programs read by the media processing device;
The step of reading the special effect further includes:
The real-time interactive video production apparatus according to claim 15, further comprising: inserting a special effect into the background data. A display device comprising a screen;
A computing device comprising at least one processor, memory and a plurality of readable programs comprising media content and special effects command scripts;
An imaging device for receiving video,
A real-time interactive video production system characterized in that the media content and the video are displayed on the screen while interacting in real time by the special effect command script processing and the media content.   18. The real-time interactive video production system according to claim 17, wherein the display device is a liquid crystal monitor.   18. The real-time interactive video production system according to claim 17, wherein the computing device is a computer.

Images