JP5026692B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program that display a screen including a moving image of a user.

Patent Document 1 below discloses an image processing apparatus that synthesizes a moving image of a user with an image created by a computer and controls screen display based on the content of the moving image of the user. According to this image processing apparatus, a user can be produced like a character in an image created by a computer, and the fun of game software and the like can be doubled.
Japanese Patent No. 3298870

  However, the above-mentioned background art is to synthesize a moving image of one user photographed by one camera at a fixed position of an image created by a computer, and images taken by two or more cameras. Cannot be used to control the screen display. For this reason, there was a problem that it was difficult to apply to a game and communication by a plurality of users.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of controlling screen display using images captured by each of two or more cameras. Is to provide.

  In order to solve the above problems, an image processing apparatus according to the present invention includes an image acquisition unit that acquires an image captured by each of two or more cameras every predetermined time, and the image acquisition unit that performs the predetermined time interval. The display content of the screen is controlled based on the relationship between the image display means for sequentially displaying the acquired images on the screen and the images acquired by the two cameras acquired by the image acquisition means. Display content control means.

  In addition, the image processing method according to the present invention includes an image acquisition step of acquiring an image captured by each of two or more cameras every predetermined time, and each of the images acquired every predetermined time in the image acquisition step. An image display step for sequentially displaying the image on the screen, and a display content control step for controlling the display content on the screen based on the relationship between the images captured by the two cameras acquired in the image acquisition step. , Including.

  In addition, the program according to the present invention provides an image acquisition unit that acquires an image captured by each of two or more cameras every predetermined time, and the images acquired every predetermined time by the image acquisition unit are displayed on a screen. Image display means for sequentially displaying, and display content control means for controlling the display content of the screen based on the relationship between the images acquired by the two cameras acquired by the image acquisition means. Game machine, portable game machine, arcade game machine, personal computer, server computer, mobile phone, portable information terminal, etc. This program may be stored in a computer-readable information storage medium such as a DVD-ROM, a CD-ROM, or a ROM cartridge.

  According to the present invention, images taken by each of two or more cameras are acquired every predetermined time and displayed on the screen. The display content is controlled based on the relationship between images taken by two of the cameras. As a result, the display content of the screen can be changed according to the content photographed by each camera, and can be suitably applied to games and communication by a plurality of users.

  The display content control means is based on the relationship between the contents of the images taken by the two cameras or the movement of the subject imaged in the images, which is acquired by the image acquisition means. You may make it control the display content of a screen. According to this configuration, for example, the user can pose the same or similar pose in front of each camera, perform a specific operation, adjust the timing, or copy a specific object with each camera. The display content of can be changed arbitrarily.

  In this case, the display content control means generates an image showing the difference in content of each image taken by the two cameras, and controls the display content of the screen based on the image. Also good. The image indicating the difference may indicate a difference in contents of the images captured by the two cameras that are acquired at different times by the image acquisition unit and displayed on the screen. Alternatively, it may indicate the difference between the contents of the images captured by the two cameras that are acquired at the same time by the image acquisition means and displayed on the screen.

  In addition, the display content control unit is acquired by the image acquisition unit, depending on whether or not the direction of movement of the subject captured in each of the images captured by the two cameras is in a predetermined relationship, You may make it control the display content of the said screen. In this way, the display content of the screen can be changed by moving the subject in a specific direction in front of each camera.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a configuration diagram of a network system configured using an entertainment system (image processing apparatus) according to the present embodiment. As shown in the figure, this system comprises a plurality of entertainment systems 10 connected to a network 50 such as the Internet or a LAN. Each entertainment system 10 includes a computer to which a camera unit 46 that captures a user's moving image is connected. Each entertainment system 10 receives and transmits data indicating the user's moving image via the network 50. 10, a common screen including a large number of users' moving images can be displayed.

  FIG. 2 is a diagram illustrating a hardware configuration of the entertainment system (image processing apparatus) according to the present embodiment. As shown in the figure, the entertainment system 10 includes an MPU (Micro Processing Unit) 11, a main memory 20, an image processing unit 24, a monitor 26, an input / output processing unit 28, an audio processing unit 30, and a speaker. 32, an optical disk reading unit 34, an optical disk 36, a hard disk 38, interfaces (I / F) 40, 44, a controller 42, a camera unit 46, and a network interface 48. It is.

  FIG. 3 is a diagram showing the configuration of the MPU 11. As shown in the figure, the MPU 11 includes a main processor 12, sub processors 14a to 14h, a bus 16, a memory controller 18, and an interface (I / F) 22.

  The main processor 12 is an operating system stored in a ROM (Read Only Memory) (not shown), for example, a program and data read from an optical disc 36 such as a DVD (Digital Versatile Disk) -ROM, and a program supplied via a communication network. Based on the data and the like, various information processing is performed, and the sub processors 14a to 14h are controlled.

  The sub-processors 14 a to 14 h perform various types of information processing according to instructions from the main processor 12, and each unit of the entertainment system 10 is read from the optical disk 36 such as a DVD-ROM or the like via the network 50. Control based on supplied programs and data.

  The bus 16 is used for exchanging addresses and data among the various parts of the entertainment system 10. The main processor 12, the sub processors 14a to 14h, the memory controller 18, and the interface 22 are connected to each other via the bus 16 so as to be able to exchange data.

  The memory controller 18 accesses the main memory 20 in accordance with instructions from the main processor 12 and the sub processors 14a to 14h.

  Programs and data read from the optical disk 36 and the hard disk 38 and programs and data supplied via the communication network are written into the main memory 20 as necessary. The main memory 20 is also used for work of the main processor 12 and the sub processors 14a to 14h.

  An image processing unit 24 and an input / output processing unit 28 are connected to the interface 22. Data exchange between the main processor 12 and the sub processors 14 a to 14 h and the image processing unit 24 or the input / output processing unit 28 is performed via the interface 22.

  The image processing unit 24 includes a GPU (Graphical Processing Unit) and a frame buffer. The GPU renders various screens in the frame buffer based on the image data supplied from the main processor 12 and the sub processors 14a to 14h. The screen formed in the frame buffer is converted into a video signal at a predetermined timing and output to the monitor 26. As the monitor 26, for example, a home television receiver is used.

  An audio processing unit 30, an optical disk reading unit 34, a hard disk 38, and interfaces 40 and 44 are connected to the input / output processing unit 28. The input / output processing unit 28 controls data exchange between the main processor 12 and the sub processors 14 a to 14 h, the audio processing unit 30, the optical disk reading unit 34, the hard disk 38, the interfaces 40 and 44, and the network interface 48.

  The sound processing unit 30 includes an SPU (Sound Processing Unit) and a sound buffer. The sound buffer stores various audio data such as game music, game sound effects and messages read from the optical disc 40 and the hard disk 38. The SPU reproduces these various audio data and outputs them from the speaker 32. As the speaker 32, for example, a built-in speaker of a home television receiver is used.

  The optical disk reading unit 34 reads programs and data stored on the optical disk 36 in accordance with instructions from the main processor 12 and the sub processors 14a to 14h. Note that the entertainment system 10 may be configured to be able to read programs and data stored in information storage media other than the optical disk 36.

  The optical disk 36 is a general optical disk (computer-readable information storage medium) such as a DVD-ROM. The hard disk 38 is a general hard disk device. Various programs and data are stored in the optical disk 36 and the hard disk 38 so as to be readable by a computer.

  The interfaces (I / F) 40 and 44 are interfaces for connecting various peripheral devices such as the controller 42 and the camera unit 46. For example, a USB (Universal Serial Bus) interface is used as such an interface.

  The controller 42 is general-purpose operation input means, and is used by the user to input various operations (for example, game operations). The input / output processing unit 28 scans the state of each unit of the controller 42 every predetermined time (for example, 1/60 seconds), and supplies an operation signal representing the result to the main processor 12 and the sub processors 14a to 14h. The main processor 12 and the sub processors 14a to 14h determine the content of the operation performed by the user based on the operation signal. The entertainment system 10 is configured to be able to connect a plurality of controllers 42, and the main processor 12 and the sub processors 14a to 14h execute various processes based on operation signals input from the controllers 42. ing.

  The camera unit 46 includes, for example, a known digital camera, and inputs black and white, grayscale, or color photographed images every predetermined time (for example, 1/60 seconds). The camera unit 46 in the present embodiment is configured to input a captured image as image data in JPEG (Joint Photographic Experts Group) format. The camera unit 46 is installed on the monitor 26 with the lens facing the player, for example, and is connected to the interface 44 via a cable. The network interface 48 is connected to the input / output processing unit 28 and the network 50, and relays data communication between the entertainment system 10 and other entertainment systems 10 via the network 50.

  FIG. 4 shows a screen displayed on the monitor 26 in one entertainment system 10 according to the present embodiment. As shown in the figure, the screen displayed on the monitor 26 is captured by another entertainment system 10 connected via a network 50 and transmitted from the other user at predetermined time intervals ( Moving images), and the user's images (moving images) taken every predetermined time and captured by the camera unit 46 connected to the entertainment system 10 together with the monitor 26 (total of 16 moving images in the figure). ing. The images of each user are aligned vertically and horizontally so that it can be seen at a glance which user is in which pose (posture). In the present embodiment, it is determined whether or not there is a pair having a predetermined relationship among the plurality of images, and an effect is added to the screen displayed on the monitor 26 according to the determination result. Yes. In other words, the entertainment system 10 determines whether there are images of the same or similar pose among a plurality of user images taken by the camera unit 46 of the own device and the camera unit 46 of the other device. If there is something like that, I try to add effects to the screen.

  FIG. 5 shows an example of a screen to which an effect is added. As shown in the figure, when an effect is added, the position of each user's image is moved from the position when the effect is not added (see FIG. 4). In addition, images and characters for effects are additionally displayed. In the case of FIG. 5, the two user images displayed relatively large in the center are reflected in the same or similar pose, or have the same or similar movement, and thus an effect is added. It is an example. Here, even if they do not completely match, the position from the center of the screen may be changed depending on the similarity of images. For example, a pose that completely matches a pose that raises one hand is displayed at the center of the screen, a pose that raises both hands is slightly separated from the center of the screen, and when both hands are lowered, the pose is further separated from the center of the screen. That is, the similarity between images is converted into a two-dimensional or three-dimensional spatial distance, and the arrangement pattern is controlled accordingly. By converting the similarity between images into a distance and displaying it in space (2D, 3D), it becomes possible to visually recognize how much they are different. Further, if the meaning is given not only to the distance but also to the place where the image is arranged (direction from the center of the screen, etc.), it becomes easier to recognize the similarity between the images. For example, in the case of a pose with both hands only, the image is displayed on the left side from the center of the screen in the case of a pose with only the left hand, and in the case of a pose with only the right hand, the image is displayed from the center of the screen. Display it on the right side.

  FIG. 6 is a functional block diagram of the entertainment system 10. As shown in the figure, the entertainment system 10 functionally includes an image acquisition unit 60, an image buffer 62, an image display unit 64, and a display content control unit 66. These functions are realized by executing a program stored on the optical disc 36 in the entertainment system 10. First, the image acquisition unit 60 acquires images captured at predetermined time intervals by the camera unit 46 of the own device at predetermined time intervals, and sequentially stores them in the image buffer 62. Further, images (images of other users) taken by the camera unit 46 of the other device and taken every predetermined time are acquired every predetermined time, and these are also stored in the image buffer 62 sequentially. To do. For example, the image buffer 62 is configured with the main memory 20 as the center, and as shown in FIG. 7, a plurality (16 in this case) of individual image buffers associated with the camera units 46 of each entertainment system 10. 62a-62p are included. In each individual image buffer 62x, from the (oldest) image 62x-1 acquired at the earliest time by the corresponding camera unit 46, the latest (latest) image 62x- acquired by the camera unit 46 is stored. A total of 5 images up to 5 are recorded. When a new image is acquired, the image 62x-1 acquired at the earliest time is discarded, and instead, the newly acquired image is stored in the individual image buffer 62x (x = ap). . In this way, each of the individual image buffers 62a to 62p sequentially stores user images acquired at five times close to the current time.

  The image display unit 64 reads the images 62 acquired at the oldest (early) time from the individual image buffers 62 a to 62 p of the image buffer 62, generates a screen on which these images are arranged, and displays them on the monitor 62. At this time, the display content control unit 66 compares the contents of the images 62 stored in the individual image buffers 62a to 62p and acquired at the oldest (early) time with each other, and takes a pose approximated by the user. Determine whether there is something. If there is a pair of images having such a relationship, the image display unit 64 is instructed to add an effect. In response to this instruction, the image display unit 64 executes effect processing such as changing the position of each image on the screen or adding an effect image such as a character or a picture to the screen.

  FIG. 8 is a flowchart showing image processing by the entertainment system 10. As shown in the figure, in the entertainment system 10, the display content control unit 66 acquires the oldest images 62a-1 to 62p-1 stored in the individual image buffers 62a to 62p of the image buffer 62, and those It is determined whether there is a pair in which the user is captured in the same or similar pose (S101). For example, the images 62a-1 to 62p-1 are acquired, the background is deleted therefrom to generate a background deleted image, and these are binarized to generate a binarized image. In this way, a binarized image is obtained in which the value of “1” is associated with the pixel in the area where the user (subject) is captured, and the value of “0” is associated with the other pixels in the background (background area). And the difference image of the binarized image corresponding to each image 62a-1 to 62p-1 is produced | generated. This difference image is an image showing the difference in the contents of the images 62a-1 to 62p-1 photographed by each camera unit 46, and corresponds to the difference when the difference that appears is a predetermined amount or more. It is determined that the image pair is a copy of a user in the same or similar pose.

  In this way, it is determined whether or not there is a pair in which the user in the same or similar pose is shown in the images 62a-1 to 62p-1, and if there is such a pair, the display content control unit 66 The image display unit 64 is instructed to add an effect (S102). Conversely, if there is no such pair, the display content control unit 66 does not instruct the image display unit 64 to add an effect.

  The image display unit 64 reads the images 62a-1 to 62p-1 stored in the image buffer 62, and generates a screen to which an effect is added in accordance with an instruction from the display content control unit 66 based on the images. Then, it is displayed on the monitor 26 (S103). Thereafter, the next screen update timing is awaited (S104), and the processing after S101 is executed again. In this way, the processing of S101 to S104 is repeated every predetermined time, and a moving image is displayed on the monitor 26.

  According to the above image processing, if the user takes the same or similar pose at the same timing in front of the camera unit 46 of each entertainment system 10, an effect can be generated on the screen displayed on the monitor 26 accordingly. It is possible to realize an interesting system.

  Here, the display content control unit 66 determines whether or not the images 62a to 1 to 62p-1, that is, the images acquired at the same timing include the same or similar pose of the user. However, it may be determined whether there are images taken at the same or similar pose among images acquired at different timings. Specifically, an image 62x-1 captured by each camera unit 46 and an image 62y-n (y is all except x) captured after a predetermined time from the image 62x-1 by other camera units 46 N may be compared with a predetermined value greater than 1 (for example, 2) to determine whether there is an image of the user in the same or similar pose (x, y = a-p). In this way, when another user imitates a pose taken by one user while looking at the screen, an effect can be generated on the screen accordingly.

  In addition, here, the display content control unit 66 determines whether or not any of the images 62a-1 to 62p-1 is captured in the same or similar pose, and the screen is displayed according to the determination result. However, it is determined whether or not an object (subject) having the same shape as shown in FIGS. 9A and 9B is photographed by two or more camera units 46. May be. Alternatively, it may be determined whether or not an object (subject) of the same color as shown in FIGS. 10A and 10B is captured by two or more camera units 46.

  Further, the display content control unit 66 may determine whether or not there is an image of the same movement taken by different camera units 46, and may generate an effect on the screen according to the determination result. That is, as shown in FIG. 11 (a), when the user swings the right hand upward from a state in which the right hand is extended sideways, there is a user who has moved in the same way, and the situation is the other camera unit 46. If an image is captured by the camera unit 46, an effect is generated on the screen. As shown in FIG. 11B, if only a user who has made another movement is captured by another camera unit 46, the effect is not generated on the screen. It may be.

  For example, as shown in FIGS. 12A to 12C, when the user swings up the right hand upward from the state where the right hand is extended sideways and the state is sequentially captured by the camera unit 46, the camera Each time an image is captured by the unit 46, a difference image from the image captured immediately before is generated, and a representative position such as a barycentric position of the difference area that appears in the difference image is calculated. FIG. 13A shows a difference image between the image of FIG. 12B and the image of FIG. 12A acquired immediately before that, and a difference area 72-1 appears there. A representative position 70-1 is calculated for the difference area 72-1. FIG. 13 (b) shows a difference image between the image of FIG. 12 (c) and the image of FIG. 13 (b) acquired immediately before that, and a difference area 72-2 appears there. Yes. A representative position 70-2 is calculated for the difference area 72-2. Then, the vector data connecting the representative positions 70 calculated at each time is set as motion data indicating the user's motion imaged by the camera unit 46.

  Such motion data is generated for images taken by all the camera units 46 and compared to determine whether or not the user photographed by each camera unit 46 has the same or similar motion. be able to. Thus, for example, a multiplayer network game (such as a fighting game, a role-playing game, or a dancing game in which a command is input by shooting a gesture of each player with a camera unit and analyzing it) is played. In some cases, when a plurality of players take the same attack pose, an application that can damage the enemy character several times is conceivable, and a diverse and attractive game application can be realized.

  If the above movement data is directly compared, even if the user moves the hand in the same direction as shown in FIGS. 14A and 14B, it is determined that they are different movements. Therefore, by calculating the direction of movement (for example, the rotation direction centered on the center of the screen) from the movement data and comparing the calculated directions, the user imaged by each camera unit 46 is the same or similar direction. It is possible to determine whether or not the user is moving. If an effect is generated on the screen according to the determination result, the user can generate an effect on the screen by moving in the same direction as that of another person.

1 is a configuration diagram of a network system using an entertainment system (image processing apparatus) according to an embodiment of the present invention. It is a hardware block diagram of the entertainment system which concerns on embodiment of this invention. It is an internal block diagram of MPU. In the entertainment system which concerns on embodiment of this invention, it is a figure which shows an example of the screen (before effect addition) shown on a monitor. In the entertainment system which concerns on embodiment of this invention, it is a figure which shows an example of the screen (after effect addition) projected on a monitor. It is a functional block diagram of the entertainment system which concerns on embodiment of this invention. It is a figure which shows typically the memory content of an image buffer. It is a flowchart which shows operation | movement of the entertainment system which concerns on embodiment of this invention. It is a figure explaining the other operation example of the entertainment system which concerns on embodiment of this invention. It is a figure explaining the other operation example of the entertainment system which concerns on embodiment of this invention. It is a figure explaining the example of further another operation | movement of the entertainment system which concerns on embodiment of this invention. It is a figure which shows the process which produces | generates a user's motion data from the image | photographed image in the entertainment system which concerns on embodiment of this invention. It is a figure which shows the process which produces | generates a user's motion data from the image | photographed image in the entertainment system which concerns on embodiment of this invention. It is a figure explaining the example of further another operation | movement of the entertainment system which concerns on embodiment of this invention.

Explanation of symbols

  10 Entertainment System, 11 MPU, 12 Main Processor, 14a-14h Sub-Processor, 16 Bus, 18 Memory Controller, 20 Main Memory, 22, 40, 44 Interface, 24 Image Processing Unit, 26 Monitor, 28 Input / Output Processing Unit, 30 Audio processing unit, 32 speaker, 34 optical disc reading unit, 36 optical disc, 38 hard disk, 42 controller, 46 camera unit, 48 network interface, 50 network, 60 image acquisition unit, 62 image buffer, 64 image display unit, 66 display content control Part, 70 operating points, 72 operating ranges.

Claims (9)

Image acquisition means for acquiring an image captured by each of the two or more cameras via a network every predetermined time;
An image buffer for storing each image acquired by the image acquisition means as an image group for each of the captured cameras;
Among the image group stored in the image buffer, and an image display means for sequentially displaying each image newly stored in the screen,
Display content control means for controlling the display content of the screen based on the relationship between the images included in each image group between the image groups stored in the image buffer ;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The display content of the screen is controlled based on the relationship between the movement of the subject captured in the image included in the first image group and the image included in the second image group among the image groups .
An image processing apparatus comprising: The image processing apparatus according to claim 2,
The display content control means generates an image indicating a difference between the contents of the images taken by the two cameras, and controls the display content of the screen based on the images.
An image processing apparatus. The image processing apparatus according to claim 3.
The images included in the first image group and the images included in the second image group are acquired at different times by the image acquisition unit and displayed on the screen, and the display content control unit is and generates an image showing the difference in the contents of the image contained in the image and the second image group included in one image group, based on the image, and controls the display contents of the screen,
An image processing apparatus. The image processing apparatus according to claim 3.
The image included in the first image group and the image included in the second image group are acquired at the same time by the image acquisition unit and displayed on the screen, and the display content control unit is and generates an image showing the difference in the contents of the image contained in the image and the included in the image group second image group, based on the image, and controls the display contents of the screen,
An image processing apparatus. The image processing apparatus according to claim 2,
The display content control unit is configured to display the screen according to whether or not the direction of movement of the subject captured in each of the images captured by the two cameras acquired by the image acquisition unit is in a predetermined relationship. Control the display content of
An image processing apparatus. The image acquisition means acquires images taken by each of the two or more cameras via a network every predetermined time,
Each image acquired by the image acquisition means is stored as an image group for each captured camera by the image buffer,
The image display means sequentially displays each newly stored image on the screen among each image group stored in the image buffer,
The display content control means controls the display content of the screen between the image groups stored in the image buffer based on the relationship between the images included in each image group.
An image processing method. Image acquisition means for acquiring images taken by each of the two or more cameras via a network every predetermined time;
An image buffer for storing each image acquired by the image acquisition means as an image group for each of the captured cameras;
Image display means for sequentially displaying each newly stored image on the screen among each image group stored in the image buffer; and
Display content control means for controlling the display content of the screen based on the relationship between images included in each image group between the image groups stored in the image buffer;
As a program to make the computer function as. The display content control means changes the position of each image displayed on the screen based on the similarity between images included in each image group for each image group. The image processing apparatus according to 1.

Images