JP2011092657A - Game system for performing operation by using a plurality of light sources - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a game system for providing a prescribed operation method by a plurality of light sources attached to a player's body, reducing a load on transfer processing in using two cameras, and obtaining the synchronization. <P>SOLUTION: The game system includes: an information processor; a display device; and a sensor unit having two cameras, and is operated by steps, i.e., a step for allowing the respective two cameras of the sensor unit to read the light sources and to obtain camera images; a step for coloring the respective two camera images; a step for synthesizing the respective two camera images and obtaining a synthetic image; and a step for transferring the synthetic image to the information processor. The information processor includes: a step for dissolving the synthetic images by color component, so as to obtain two images; and a step for calculating the position coordinates of the light sources, based on the positional relationship between the light sources of the two images. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

The present invention relates to a game system, and more particularly, to a game system used in a game system in which a user moves and operates by using a sensor unit including two cameras.
The game system may be a dedicated game machine or a personal computer incorporating game software. These are systems that include some input device and are connected to display means such as a display device. There are also high-performance game machines such as those equipped with a network function and a storage device, and these are collectively referred to as a game system.
In particular, game systems for brain training have been developed. There are games that train the brain for the elderly while looking at memory and responsiveness. These play a role in the prevention of dementia. On the other hand, there is a theory that using the body more stimulates the brain and is better for preventing dementia. In this way, the main application of the game system that performs operations by moving the body is often used for medical purposes.

  Conventionally, game systems using the body have been developed. For example, a game equipped with an input device that looks like a musical instrument is used as a rhythm game by looking at the input device as a drum, etc. Can enjoy sports games. In particular, Japanese Patent Application No. 2005-264207 discloses an input device that is held in hand.

  As an input device, it is disclosed by the technique of the touch panel as a representative example. In particular, a technique for detecting a plurality of indicators is disclosed as a touch panel system that can be operated at a plurality of indicated parts in Japanese Patent Application No. 2000-36317 invented by the present inventor. In this method, a plurality of touches touched on the touch panel are counted, a transitional change such as a distance between the touches is read, and a predetermined operation is performed.

  Conventionally, there has been a camera that combines a lens mechanism and a shutter mechanism of two cameras called a stereo camera. This is a camera that can record information in the depth direction by simultaneously photographing an object from a plurality of different directions. By using this, a stereoscopic photograph can be taken. Applying this, there was also an applied technology that took two stereoscopic images by attaching two CCD cameras that could be connected to a computer.

Since the game system of the prior art requires a simpler structure, the body is often moved in a pseudo manner using a characteristic input device.
An instruction such as selection or movement of an object displayed on the display screen while moving the body is performed through a certain reaction. In order to detect this reaction, it is more reliable to mark something on the body and read it with a camera or the like than to use the characteristic device described above. Therefore, it is common to use a mark as a light source or the like, attach a plurality of light sources to the body, and use it as a mark. These are also applied to techniques such as motion capture as techniques for creating 3D video.

If multiple light sources are attached to the head, both hands, and both feet, the camera that captures them needs to have at least two cameras to capture all movements, including front and back, left and right, forward, and backward. is there.
There are two methods for using these two cameras. One is a method for capturing a light source in the manner of triangulation, and the other is a method for capturing a light source using parallax like a human eye. The stereo camera of the present invention is the latter method.

Japanese Patent Application No. 2005-264207 Japanese Patent Application No. 2000-36317

In a game system, when an information processing device that processes information and an input device are connected by transfer means, for example, the connection means is a USB transfer means, and two cameras are connected to the information processing apparatus of the game system. Then, it is possible to simply use two cameras.
However, if two camera images are to be processed, the camera image data from the camera is enormous and high speed is required. Therefore, high performance is required for the transfer processing, and the capacity load of a general game system is required. Will be exceeded. Therefore, it can be said that the method of processing the images of the two cameras is inappropriate. In addition, when two cameras are simply connected to the information processing apparatus, the number of connection cables becomes two and the appearance is not good.

  Further, when two cameras are used, it is necessary to synchronize, which causes a problem. That is, if the image of the camera is taken at exactly the same timing and information processing is not performed at the same timing, the movement cannot be correctly captured if the target is moving. If this is to be controlled by the game system main body, this also requires sophisticated and high-speed processing.

  On the other hand, when an operation method using a plurality of light sources attached to the body is considered, it is necessary to consider a certain processing pattern for detecting a movement with respect to various movements. Even in simple movements such as punching and kicking, waving hands, imitating walking, etc., processing patterns require complicated judgment processing. In particular, a certain device operation on the display screen, for example, an operation such as selection or movement, needs to be sufficiently devised when it corresponds to the movement of the body. The movement of the body is called reaction.

  In addition, with conventional motion capture technology, when multiple light sources are attached to the head, hands, and feet, there is a possibility of losing the positional relationship such as when the hands and feet cross or stand upside down. It was.

  The present invention provides a mechanism for reducing the load of image transfer for a sensor unit using two cameras suitable as a game system that moves and moves the body, and further acquires and processes two camera images. On the other hand, the game system provides various operation means by reaction by having means for synchronizing and attaching a plurality of light sources to the head, hands and feet.

  In order to achieve the above-described object of the present invention, a game system according to the present invention is a game system including an information processing device and a sensor unit having two cameras, and the information processing device and the sensor unit are transfer means. Further, the information processing apparatus is a sensor unit that is connected to a display apparatus and that displays the display apparatus from display means of the information processing apparatus.

The information processing apparatus is a game system including a CPU module, a display processing module, a storage module, and an audio reproduction module.
The sensor unit is connected to the information processing apparatus by a transfer unit and serves as an input device.
The display processing module and the audio reproduction module are connected to a display device.
In the content displayed on the display device connected to the information processing device, an object is displayed, and the object is selected or moved based on information from the sensor unit.

  The camera reads a predetermined light source. The light source may be an LED that emits light using a battery or the like. The light emitting part attached to the sensor unit or the like and the part to be used as the light source are attached with a reflective part using a recurring reflective material. There is a method of causing a camera to read as a light source by reflection of light from a reflection part.

Each of the two cameras of the sensor unit reads a light source to form a camera image, a step of coloring each of the two camera images, a step of combining each of the two camera images into a combined image, and a combination Operates in the step of transferring the image to the information processing device,
The information processing apparatus is an information processing apparatus including a step of separating a composite image with color components to form two images, and a step of calculating the position coordinates of the light source from the positional relationship between the light sources of the two images.

There are a plurality of light sources, and the information processing apparatus includes a light source counting unit that counts the number of light sources and a distance calculation unit that calculates a distance between the light sources.
In addition, when a size calculating unit that calculates the size of each light source is provided, the information processing device detects a combined positional relationship pattern with the distance calculating unit that calculates the distance between the light sources.

  The information processing apparatus may be an information processing apparatus including a plurality of light sources, and a light source counting unit that counts the number of the plurality of light sources and a moving speed calculating unit that calculates a moving speed of each light source. good.

  There are a plurality of light sources, and the information processing apparatus includes a light source count unit that counts the number of light sources, a movement speed calculation unit that calculates a movement speed of each light source, and a light source movement that calculates a movement direction of the light source. An information processing apparatus including a direction calculation unit may be used.

The plurality of light sources may be ring-shaped or band-shaped, and may be attached to a human part. The plurality of light sources may be gloves, shoes, or hats, and may be attached to a human part.
Further, the plurality of light sources may be ring-shaped, band-shaped, glove-shaped, shoe-shaped, or hat-shaped light sources in which the number of light sources and the size of the light sources change when the direction is changed.

  There may be a plurality of the light sources, and the plurality of light sources may be a mat having a plurality of light sources.

  When there are a plurality of light sources, the light source counting means for counting the number of light sources, the distance calculating means for calculating the distance between the light sources, the moving speed calculating means for calculating the moving speed of each light source, The light source movement direction calculation means for calculating the movement direction is configured so that the information processing apparatus performs a predetermined operation with an increase / decrease in the light source count of the light source, a change in the distance, speed, direction, or the like as described above or a transient change. And an information processing apparatus including control means.

  The display device connected to the information processing device has a guidance pose that guides a pose by a human-type display to be taught or a display equivalent thereto, and the game player equipped with the plurality of light sources displays a display pose on the display device. In the case of a game with a guided pose, the game system reads the shape of the pose from the position of the light source when the game player poses, calculates the deviation from the guided pose and the phase difference of the position, It is a game system that displays on a display device in the form of scores and numerical values.

  Further, when the guided pose is a continuous pose movement or motion, a deviation from the continuous pose movement or motion or a position phase difference is calculated and displayed on the display device in the form of a score or a numerical value. It is a game system.

The game system that operates using a plurality of light sources according to the present invention is a game system that operates using a plurality of light sources. A plurality of light sources attached to the body can provide a predetermined operation method. There is an advantage that it is possible to reduce the load of the transfer processing by using and to synchronize.
Further, in the present invention, the type of light source can be determined by acquiring a color image adjacent to the light source from the camera image, and means for determining the positional relationship that is not known when the right hand and the left hand, the right foot and the left foot, etc. intersect. It is a game system that has.

FIG. 1 is a diagram showing an example of use of the present invention. FIG. 2 is a schematic block diagram of the present invention. FIG. 3 is a diagram for explaining the operation of the sensor unit of the present invention. FIG. 4 is a processing step diagram of the present invention. FIG. 5 is a simplified diagram illustrating the labeling process. FIG. 6 is a process step diagram of the sensor unit for transferring information with header information attached by separating spontaneous light emission and reflective light emission according to the present invention. FIG. 7 is a step diagram for discriminating a plurality of color conversion processes as a process for discriminating spontaneous emission and reflective emission according to the present invention. Fig. 8 shows an example of a band colored with a reflex reflector as a light source FIG. 9 is a processing step diagram in the case of using a method for discriminating the color near the light source. FIG. 8 shows an example of a band using a plurality of recurring reflective materials attached to the head as a light source. Figure 9 shows an example of a band with a light source attached to the hand FIG. 10 shows an example of a mat with a plurality of light sources.

Hereinafter, embodiments for carrying out the present invention will be described together with illustrated examples.
FIG. 1 is a diagram showing an example of use of the present invention. As shown in the figure, a game system that operates using a plurality of light sources according to the present invention includes an information processing device 1, a sensor unit 2, and a display device 3. The sensor unit 2 and the display device 3 are each an information processing device 1. Connected. In particular, the information processing apparatus 1 and the sensor unit 2 are connected to each other by a USB cable 4 having a USB transfer means. Two cameras, camera 8 and camera 9, are attached to the sensor unit. An issue site 16 is attached at a position between the two cameras. The sensor unit 2 may be placed horizontally or vertically. Further, the issue part 16 may be arranged anywhere as long as it is close to the camera.

The person on the right has a band with a reflex reflector around his head, hands and feet. The head band 5 has a recurrence reflecting material only in the forehead portion, and the hand band 6 and the foot band 7 are the recurrence reflecting material as a whole. Since the band is removable, the attachment position can be freely determined.
Since the present invention is a game system, depending on the type of game, whether to wear only a hand or only a foot is instructed by the game.

  Next, FIG. 2 is a schematic block diagram. As shown in the figure, a game system that operates using a plurality of light sources according to the present invention includes an information processing device 11, a sensor unit 12, and a display device 13. The sensor unit 12 and the display device 13 are each information processing device 11. Connected. The information processing apparatus 11 includes at least a CPU module, an audio module, a display module, and a storage module. The audio module and the display module are connected to the display device 13 and perform display and audio output.

The detail about the sensor unit 12 used as the principal part of this invention is demonstrated. In the sensor unit 12, two cameras A14 and B15 are arranged side by side at a certain distance. A light emitting portion 16 using infrared light as a light source is disposed at a position between the cameras. The issue part 16 may be arranged anywhere as long as it is close to the camera. The camera of the sensor unit 12 uses a CCD camera. This camera preferably has a shutter function.
The light emission of the issuing part 16 and the shutter of each camera are controlled by the control of the CPU module built in the sensor unit. The image read by the camera is temporarily recorded in a recording buffer in the sensor unit. Each recorded camera image is processed by a CPU module with a built-in sensor unit. The processing contents will be described later. The processed camera image is transferred to the information processing apparatus 11 by transfer processing. As this transfer means, transfer means such as USB communication is used.

Next, FIG. 3 is an operation explanatory diagram of the sensor unit. The camera arrangement method of the sensor unit 21 is such that the orientations of the two cameras A22 and B23 are aligned in parallel with the same orientation. For example, it is arranged like a human eye. This camera arrangement is referred to as a stereo camera system. The camera images 26 and 27 from the two cameras obtained at this time are combined with one target 25, and when the images are combined, the image of the target 25 is shifted like a composite image 28. Position. Based on this shifted distance, the Z coordinate that is the depth can be calculated. In addition, the video of the target that will be the same is corrected for the shift of the video of the target in each camera image, and then the X coordinate and Y coordinate are calculated. In this way, by using the stereo camera system, it is possible to specify a three-dimensional position with respect to the target.
On the other hand, means for specifying a target in the manner of triangulation by turning the two cameras inward by 45 degrees may be considered, but in the present invention, the stereo camera method will be described for convenience.

In the present invention, an infrared light source is used and an infrared filter is attached to the camera. As a result, since only the infrared light source is reflected in the camera image, the target can be identified more easily. The light source emits an infrared LED using a battery or the like (spontaneous light emission), the infrared light emitting part 24 attached to the sensor unit 21 or the like, and the reflection part 25 using a reflex reflecting material at a place where the light source is desired. By attaching, infrared light emission from the light emitting part 24 is strongly emitted in a flash manner, so that the camera reads as a light source by reflection of light from the reflection part 25 (reflective light emission). The re-reflecting material is a member that reflects in the direction in which light shines, that is, in the direction that returns to the direction in which it shines.
The present invention has means for discriminating between the spontaneous emission and the reflection emission, and will be described in detail later.

Next, the processing steps of the present invention will be described with reference to FIG. FIG. 4A is an explanatory diagram of processing steps of the sensor unit 21. According to this processing step, it is possible to explain a method for solving a problem that is to be solved by the invention and that is synchronized with a reduction in the load of the image transfer process.
A processing step of emitting infrared light from the issue part 24 of the sensor unit 21 is performed, and then a shutter of the camera A22 and the camera B23 is turned off to acquire an image. This is performed in order to read the reflected light at the moment when infrared light is emitted strongly in a flash manner. Next, a step of changing the camera image 26 of the camera A22 to red is performed. Since only infrared light is received in the present invention, the camera image is a black and white image. The portion corresponding to the light source appears whiter and the others appear black. In the above step, the black and white color is inverted, and black is a process of converting the color to red. Next, a step of changing the camera image 27 of the camera B23 to blue is performed. This is a color change process of the camera A made blue. Next, a step of synthesizing the image of the camera image 26 and the camera image 27 after the color conversion process into a composite image 28 is performed. This is a process in which two single-color images of red and blue are combined to form one color-colored combined image 28. Next, there is a step of transferring the composite image to the information processing apparatus.
In this way, the composite image 28 can be transferred to the information processing apparatus 11 in synchronism as a single color image.

  Next, the processing steps of the present invention will be described with reference to FIG. FIG. 4B is an explanatory diagram of steps performed by the information processing apparatus 11 based on the composite image 28 acquired from the sensor unit 12. First, a step of acquiring the composite image 28 transferred from the sensor unit 12 is performed. Next, a step of decomposing the composite image 28 with red and blue colors is performed. This is a process of decomposing into two images of only red and blue.

  Next, a step of detecting a light source position from the two separated images and labeling the light source is performed. In general, when image recognition processing is performed, processing is performed for numbering a pattern corresponding to a light source. This numbering process is called labeling process. This labeling process is shown in a simplified diagram for explaining the labeling process of FIG. In this case, since there are two separated images, the labeling process in FIG. 5 is performed twice. When a labeling process is performed on each decomposed image 31 according to a certain direction, a consistent label number is given to the position of the light source. The image diagram 32 is an example in which a label number is given. Accordingly, for example, it can be recognized that the label 1 of the red separated image and the label 1 of the blue separated image are the same light source, although it is a simple determination. If the center position of the pattern subjected to the labeling process is calculated, the center position is shifted from the two separated images. Taking the average of the displaced positions, the approximate center position coordinates of the X-axis coordinates and Y-axis coordinates of the light source are determined. Further, by calculating the distance between the positions of the two separated images, the Z coordinate that is the depth can be calculated. This corresponds to the step of calculating the position coordinates of the labeled light source labeled in FIG.

Next, the number of light sources and the distance between the light sources in FIG. The number of light sources corresponds to the number of labels when labeling is performed. In the present invention, this is also called a light source count number. The distance between the light sources refers to the distance between the light source position coordinates, which are the center position coordinates of the pattern of the light source subjected to the labeling process. The more light sources, the more this information. The number of light sources and the position coordinate values of the light sources are temporarily stored.
Next, a step of calculating a transient change from the previous position coordinates of the light source, the number of light sources, and the distance between the light sources is performed. This is because the number of light sources and the change in the distance between the light sources are calculated based on the information temporarily stored last time, and whether there are transient changes, that is, whether the number of light sources has increased or decreased, It is calculated whether the distance between them has increased or decreased by a certain number, and if a change is recognized, processing is performed as a predetermined operation. The predetermined operation refers to selection or movement of a displayed object on the display device 13 of the information processing apparatus 11. Finally, a series of flows can be explained in the step of updating the display on the display device 13.

  Next, in the processing step of the present invention, a method for detecting two types of light emission, spontaneous emission or reflection emission, is shown. Here, only the processing of the sensor unit will be described.

FIG. 6 shows a method for transferring the difference between the spontaneous emission and the reflection emission to the information processing apparatus 11 as the header information with the information on the emission of the infrared light emitted from the light emitting part 24 and not issued.
First, in the processing group of the group (a) for detecting spontaneous emission, a step of not emitting infrared light from the light emitting part 24 is performed, and then a step of acquiring images by cutting the shutters of the camera A22 and the camera B23 is performed. Next, the step of changing the camera image 26 of the camera A22 to red, the step of changing the camera image 27 of the camera B23 to blue, and then combining the images of the camera image 26 and the camera image 27 after the color conversion process, Step 28 is performed. Next, a step of transferring the information obtained by combining the composite image 28 and the non-light emitting header information to the information processing apparatus 11 is performed.

  If the processing group of the series of detection steps of spontaneous emission (a), the processing group of the detection step of reflective emission (a) starts from the step of emitting infrared light, and the information processing device 11, a step of transferring the combined image 28 and information including the header information with light emission to the information processing apparatus 11 is performed. In this way, the composite image 28 can be transferred to the information processing apparatus 11 in synchronism as one color image and header information.

  While alternately performing the processing group of the spontaneous emission detection step group (a) and the processing group of the reflective emission detection step group (A), the light source is identified as spontaneous emission or reflection emission, It becomes possible to detect.

FIG. 7 is a step diagram showing a method of discriminating by conversion processing of a plurality of colors as processing for discriminating spontaneous emission and reflective emission.
In the color conversion of the camera image, when spontaneous light emission is detected without causing the light emitting part 24 to emit infrared light, the camera image 26 of the camera A22 is converted to green, and the camera image 27 of the camera B23 is converted to yellow. Next, in the case of detecting reflected light emission by emitting infrared light from the light emitting portion 24, the camera image 26 of the camera A22 is converted to red and the camera image 27 of the camera B23 is converted to blue. The respective converted images are combined as a combined image to obtain a combined image 28. In this case, four colors are used in the composite image 28, and the composite image 28 of red, blue, green, and yellow is obtained. In the step of transferring the composite image 28 to the information processing apparatus 11, the composite image 28 can be transferred to the information processing apparatus 11 as a single color image in synchronization.

  In addition, the processing group of the spontaneous emission detection step group (a) and the processing group of the reflection light emission detection group (a) are processed alternately, but even if the frequency ratio of the detection is changed. It doesn't matter. That is, after performing the group (A) of the reflective emission detection step 10 times and then performing the group (A) of the spontaneous emission detection step once, the operation of performing the spontaneous emission is the menu selection operation. For example, it can be used as position detection for applications that do not require special tracking performance.

  Note that any combination of colors used for color conversion and different colors may be used. In general, red, blue, and green, which are said to be the three primary colors, are mainly used. However, the computer can separate 8 colors or 16 colors. In addition, as a detection method when there is no particular need for follow-up and it is not necessary to obtain position information about the depth, for example, in the case of performing spontaneous emission detection, in the description of FIG. Although the color conversion is performed, a simple processing method is possible in which both color conversions are green and the composite image 28 is obtained.

  In the above description, the method of acquiring only the light source using the infrared filter is shown. In the case where the camera of the sensor unit 21 can acquire a color image, a process for discriminating the color in the vicinity of the light source can be used instead of the process for discriminating spontaneous emission.

  FIG. 8 shows an example of a band in which the edge is colored using a re-reflecting material as a light source when a method for discriminating the color near the light source is used. In this manner, a specific color edge 36 is attached around the re-reflecting material 35 attached to the band 37 and serving as a light source. For example, if the color is specified such as black for the head, red for the right hand, and blue for the left hand, the position of the light source, that is, the mounting location can be specified.

FIG. 9 shows processing steps when a method for discriminating colors in the vicinity of a light source is used.
First, the step of emitting infrared light from the light emitting part 24 is performed, then the shutter of the camera A22 and the camera B23 is turned off to acquire an image, and then the camera image 26 of the camera A22 is changed to red. A step of changing the camera image 27 of B23 to blue, and then a step of combining the camera image 26 and the image of the camera image 27 after the color conversion process into a composite image 28 are performed. Next, a step of transferring the composite image 28 to the information processing apparatus 11 is performed.

  Next, the step of not emitting infrared light from the light emitting portion 24 is performed, and the step of acquiring the image by cutting the shutter of the representative camera A22 is performed. At this time, it is acquired as a color image. Next, a step of transferring the color image 28 of the camera A22 to the information processing apparatus 11 is performed.

  As shown in FIG. 9, the processing of the A group and the B group is performed alternately, but the ratio may be that the processing of the A group is performed more and the processing of the B group is performed appropriately. For example, a state in which a person has just paused from the information processing apparatus 11. In a state where there is not much movement, a command may be issued to the sensor unit 12 to perform processing of the B group.

  Next, after acquiring these pieces of information, the information processing apparatus 11 that has received the image records the position of the light source obtained from the composite image of the group A, particularly the portion of the red image acquired by the camera A22, and the group B The color around the portion of the red image is obtained from the color image processed in step (1). As described above, the position of the light source is subjected to a labeling process. If the labeling number is added to be equivalent to the color number, the type of the light source can be determined.

  In the description, only the camera A22 is used, but the camera B23 may be used. It is also possible to use these alternately.

Next, a light source mounting method related to an actual usage method will be described.
The person shown in FIG. 1 is wearing a band-like recurrence reflective material. In particular, in the case of the head band 5, since the area is large, the reflex reflecting material is attached only to the forehead.
Although the example of FIG. 1 is a band shape, it may be in the shape of a hat, gloves, shoes, or socks.

An application is shown in FIG. It can also be described as an application example of a method for calculating the size of a light source. This is a band-like 41 to be worn on the head, and there are several places where a plurality of light sources and re-reflecting materials are attached. According to this, since the pattern of the light source that will be the head part changes depending on the head direction, the head direction can be detected.
For example, if it is forward, the large light source 42 is in the center, and small light sources 43, 44, 45 are detected on the left and right and up, and if it is in landscape orientation, the large light source 42 is on either the left or right side. When one of the small light sources 43, 44, 45 is detected and bowed, the pattern is such that the small light source is in the center and there are three light sources 42, 43, 44 below. Since these distance relationships are always constant with respect to the light source 42, it can be recognized that the light source is a head light source.
In this way, by combining the method for calculating the size of the light source, the direction can be calculated by pattern recognition.

An application is shown in FIG. This is a band to be worn on the hand, and the place where the recurrence reflective material is attached is in a position where it can be seen and hidden. In FIG. 11, the reflex reflector is attached to the palm. According to this, when the hand is held, the light source by the recurrence reflecting material disappears, and when the hand is opened, the light source can be seen. In addition, when the reflex reflector is attached to the surface of the thumb, when the thumb is folded, the light source by the reflex reflector disappears, and when the thumb is protruded, the light source can be seen. Note that other fingers may be used instead of the thumb.
This change in the appearance of the light source is effective for a determination operation for selecting a menu or an object displayed on the display screen of the information processing apparatus.

  When the surroundings of the reflex reflecting material in the band shown in FIG. 11 have a specific color, the left hand and the right hand can be discriminated by using the method described with reference to FIGS.

An application is shown in FIG. This is an example in which a mat-like object in which a plurality of light source patterns are laid is used.
If the mat 64 with the light source in this figure is held, any one of the light sources 65 disappears by stepping on the mat 64 to hide any one of the light sources 65. Therefore, if the disappeared position is specified, for example, It can be specified as a button position for a specific menu operation, muffled game, rhythm game, or the like.
The mat 64 with the light source may be laid on the floor or attached to the wall. In order to remember that the light source is basically non-moving, the information processing apparatus 11 performs a process called calibration. This is a process for recording in advance the position of the light source on the mat.

Next, the processing method when using a plurality of light sources will be described in detail.
The calculation method of the position coordinates of each of the plurality of light sources is as described above. In addition, the following detection processing means is possible for the light sources.
(1) Light source counting means for detecting the number of light sources (2) Light source size means for detecting the size of the light source (3) Distance calculation means for detecting the distance of each light source (4) Moving speed when the light source moves Calculation means (5) Movement direction calculation means when the light source moves

(1) Light source counting means for detecting the number of light sources is means for counting the number of light sources. As described above, with reference to FIG. 5, a brief description of the labeling process has been given. The labeling process will be described in more detail. A light source pattern is processed as a binarized image, and the same label is added to connected pixels to classify a plurality of areas as a group. As an example of a general labeling process, a pixel (x, y) without a label on the image
And add a new label. Next, the same label is added to the pixel connected to the pixel (x, y) in the four directions. Finally, the same label is added to the pixel to which the same label is added and the pixel connected in four directions. This operation is repeated as long as there are pixels to which labels are added in the image. After being classified as a group in this way, by counting the number of groups, the number of light sources, that is, light source counting means is obtained.
In addition, the center coordinate value of the pixel which becomes the center position of the pixel group of the light source classified into the group is the position coordinate value of the light source.

(2) In the light source size means for detecting the size of the light source, the size can be simply calculated by counting the number of pixels for one group in the labeling process.

(3) The distance calculation means for detecting the distance of each light source calculates the distance between the light sources in the position coordinate value indicating the center position of each light source. The calculated distance is determined by the number of light sources. That is, when there are three light sources, there are three distances between the light sources, and when there are four light sources, there are six distances between the light sources. Since each light source is numbered in the labeling process, header information indicating information related to the light source number, such as 1-1, 1-2, is added to the calculated distance. To be recorded.

(4) When the light source moves, the moving speed calculation means temporarily stores the position coordinate value indicating the center position of each light source in the buffer, and the previous position coordinate value, the current position coordinate value, Therefore, the moving speed can be calculated by calculating the detection distance with the detection timing time.

(5) When the light source moves, the moving direction calculation means temporarily stores the position coordinate value indicating the center position of each light source in the buffer, and the previous position coordinate value, the current position coordinate value, The movement direction can be determined by the difference between the plus direction and the minus direction. Since the movement direction can be detected in three dimensions in the X direction and the Y direction in the present invention, the movement direction can be calculated in the Z direction.

The present invention provides a control means for performing a predetermined operation on a menu or an object displayed on the display screen 13 of the information processing apparatus 11 using these detection processing means.
A specific example of control means for performing a predetermined operation using the detection processing means will be given.

  If a light source count means for detecting the number of light sources is used, a predetermined operation can be performed by changing the light source count number. For example, the case where the band attached to the hand of FIG. 11 is used will be described. When the reflex reflector is attached to the palm, the number of light sources decreases when the hand is held, and the number of light sources increases when the hand is open. When using a mouse, there are left-click and double-click operations of the mouse, but in the position coordinates where the hand is held, if you hold the hand once to open it, it will be clicked, and if you hold the hand twice and open it, double A click operation can also be considered.

  In the case of using a band to which the recurrence reflective material attached to both hands is used like the person in FIG. 1, an object displayed on the display screen 13 of the information processing apparatus 11 by a combination of the action of bringing both hands closer and the action of moving away. It is possible to make the operation of sandwiching and separating the operation. The realization means has distance calculation means for detecting each distance of the light source, and makes a determination according to a transient change in the distance. Transient change means that both hands change from a state where both hands are sufficiently close to each other when they are pinched, and from a state where both hands are close to each other when they are released. It refers to changing. By judging the amount of transitional change based on a threshold value, it is possible to apply a filter to blurring and shaking that normally occur in humans.

  Similarly, when a light source moves, a movement speed calculation unit and a movement direction calculation unit can be combined to detect an operation such as a sports game. For example, in the case of a volleyball serve, when raising the ball with the left hand and hitting the ball with the right hand to serve, the band with the reflex reflector attached to both hands as shown in Fig. 1 is used. When used, the operation of raising the left hand is the ball displayed on the display screen 13 of the information processing apparatus 11 by reading the movement of the light source from the band of the left hand, that is, the speed and direction of raising the light with the detecting means. The speed and direction of movement can be reproduced. Similarly, in the operation of hitting the ball with the right hand, the movement of the light source from the band of the right hand, that is, the speed, direction and timing of swinging down the front of the ball to be hit displayed on the display screen 13 of the information processing apparatus 11 Can reproduce speed and direction. Further, since the moving speed is accompanied by a transient change with an acceleration, the transitional change of the moving speed is an element that should be sufficiently considered in a sports game. In the present invention, this transitional change in the moving speed is also indicated.

  Examples of using a light source size means for detecting the size of the light source, a light source size difference, a light source by spontaneous light emission, and the like are effective means for showing some reference point or surface. As described above, in the example of the band using the plurality of reflex reflectors attached to the head in FIG. 8 as a light source, the band is composed of a combination of a light source 42 having a large area and light sources 43, 44, and 45 having a small area. Although this is an example of detecting the head orientation, the pattern detection process can be clarified by patterning the positional relationship of light sources with a small area in the vicinity of the light source 42 with a large area as a reference for the head position. Further, in the case of a mat shape in which a plurality of light source patterns described in FIG. 11 is laid, at least two vertices are assumed to be a large-area re-reflecting material or a spontaneously emitting LED is attached. It is a mark to specify the size and position of the mat.

Here, a case where a predetermined color is applied to the light source, or a case where a color body with a prominent predetermined color is used instead of the light source will be described.
In the case of a simple light source, infrared light is used to attach an infrared filter to the camera to make the light source stand out. However, a similar effect can be obtained by using a color filter instead of an infrared filter. It is done. The color filter may be attached to the lens of the camera, or the processing for extracting the color component may be performed by image processing of the camera image as it is. In this case, the reflective light emission described above is a prominent predetermined color component that is not reflected.

Here, a basic operation method when a game is actually played with the game system of the present case will be described.
The display device connected to the information processing device induces a pose by teaching a humanoid display or a display equivalent thereto. This is called a guidance pose. For example, in the case of a dance game, a character or a 3D human model is displayed and a pose is displayed. For example, in the case of a golf game, the text “Please swing” is displayed. In this case, since anyone knows the golf swing, it is possible to teach by text.

  The game player wearing the plurality of light sources poses the above-described guided pose displayed on the display device. In the dance game of the above example, the dance is performed by imitating the induction pose. At this time, the game system reads the shape of the pose from the positions of the plurality of light sources when the game player poses. If the light source band is attached to at least the head, both hands, and both feet, the pose can be easily read.

The difference between the shape of the pose of the game player read by the game system and the pose shape of the guided pose and the phase difference of the position are calculated and displayed on the display device in the form of scores and numerical values. It is only necessary to calculate the relative deviation between the position where the game player is wearing multiple light sources and the position of the guided pose, so if the light source band is attached to the head, both hands and both feet, Calculation is possible from the position coordinates. There are several methods for calculating these various values, but this is not limited in this case.

  Further, when the guided pose is a continuous pose movement or motion, a deviation from the continuous pose movement or motion or a position phase difference is calculated and displayed on the display device in the form of a score or a numerical value. . For example, in the case of a golf swing of a golf game, since continuous motion is regarded as important, this method can realize a golf game.

  As described above, various operations can be performed by using the detection processing means of the present invention. These operations can also be realized by reducing the camera image transfer process and solving the synchronization means.

1, 11 Information processing device 2, 12 Sensor unit 3, 13 Display device 8, 9 Camera 14, 22 Camera A
15, 23 Camera B
10, 16, 24 Light emitting part 5, 6, 7, 25 Light source
(In the case of spontaneous emission, infrared LED,
In the case of reflective luminescence, a recurring reflective material)

Claims (23)

A game system having an information processing device and a sensor unit connected to the information processing device and incorporating two cameras,
Each of the two cameras reads a light source to form a camera image, the camera image is a single color, a step of coloring each of the two camera images, and a composite image obtained by combining the two camera images. And a step of transferring the composite image to the information processing device,
The information processing apparatus includes a reception step of receiving a composite image, a step of separating the composite image with color components into two images, and a step of calculating the position coordinates of the light source from the positional relationship of the light sources of the two images. A game system for performing operations using a plurality of light sources.   2. The game system according to claim 1, wherein the light source is a reflective material that re-reflects, and the sensor unit includes a light emitting unit that is linked to the sensor unit, and is a reflection of the light source that emits light from the light emitting unit. A game system that operates using a plurality of light sources. 3. The game system according to claim 1, wherein the light source includes a light source (a) that has its own power source and emits light, and a light source (a) that emits light by reflection of light from the light emitting unit by the reflecting material. ,
The step of reading the light source with the two cameras of the sensor unit to form a camera image is the reading step of the light source (a) with the light emitting unit turned off or the reading step of the light source (a) with the emitting unit emitting light. The step of coloring each of the two camera images, the step of combining each of the two camera images into a composite image, and the issue information and composite image of the issuing unit Works in the transfer step to the processing device,
The information processing apparatus includes a reception step of receiving a composite image, a step of separating the composite image with color components into two images, and a step of calculating the position coordinates of the light source from the positional relationship of the light sources of the two images. And a step of processing the issue information of the issue unit as identification information. A game system for performing an operation using a plurality of light sources. 3. The game system according to claim 1, wherein the light source is a light source (a) that emits light by reflection of light from the light-emitting portion by the reflective material, and an edge that has a specific color around the light source (a). (I) and
The step of reading the light source with two cameras of the sensor unit with the issuing unit emitting light to form a camera image, the camera image is a single color, and coloring each of the two camera images, It operates in a step of synthesizing camera images to form a composite image and a step of transferring the composite image (1) to the information processing device,
It operates in the step of acquiring a color image with two cameras of the sensor unit or one of the cameras with the issuing unit emitting light, and the step of transferring the color image (2) to the information processing device,
The information processing apparatus calculates the position coordinates of the light source from the reception step of receiving the composite image (1), the step of separating the composite image into color components to form two images, and the positional relationship between the light sources of the two images. And steps to
Receiving a color image (2), obtaining an identification color of the color image (2) in the vicinity of the position coordinates of the light source, and processing the identification information according to the identification color. A game system that operates using multiple light sources. 5. The game system according to claim 1, wherein there are a plurality of light sources, and the information processing apparatus calculates a distance between each light source and a light source counting unit that counts the number of the plurality of light sources. Means,
In addition to the number of light sources counted by the light source counting means, the information processing apparatus performs a predetermined operation in accordance with a distance between light source positions calculated by the distance calculating means or a transient change in distance. A game system for performing an operation using a plurality of light sources, characterized by comprising control means for performing the above.   The game system according to claim 5, wherein there are a plurality of light sources, and the information processing apparatus is configured such that when the light source count number is reduced or the distance by the distance calculation unit becomes a predetermined value or less, the information processing apparatus A game system for performing an operation using a plurality of light sources, characterized by comprising control means for specifying the position and performing a predetermined operation.   6. The game system according to claim 5, wherein there are a plurality of light sources, and the information processing apparatus is configured such that when the light source count number is increased or the distance by the distance calculation means becomes a predetermined value or more, the information processing apparatus. However, the game system performs operation using a plurality of light sources, characterized by comprising control means for specifying the position and performing a predetermined operation.   6. The game system according to claim 5, wherein there are a plurality of light sources, and the information processing apparatus calculates a size of each light source, and the size of the light source by the size calculation means is greater than or equal to a certain value. A game system for performing an operation using a plurality of light sources, characterized in that the information processing apparatus includes control means for performing a predetermined operation.   9. The game system according to claim 8, wherein there are a plurality of light sources, and the information processing device is different in size of the light sources by the size calculation means for calculating the size, or the light source is a light source that has its own power source and emits light. ) And a light source (a) that emits light by reflection of light from the light emitting unit, and a distance calculation unit that calculates a distance between the light sources, detects a combined positional relationship pattern, and the information processing apparatus A game system for performing an operation using a plurality of light sources, comprising control means for performing a predetermined operation.   5. The game system according to claim 1, wherein there are a plurality of light sources, and the information processing apparatus counts the number of the plurality of light sources, and a movement speed for calculating a movement speed of each light source. In addition to the number of light sources counted by the calculating means and the light source counting means, the information processing apparatus performs a predetermined operation in response to a transient change in the moving speed of the light source calculated by the moving speed means. A game system for performing operations using a plurality of light sources, characterized by comprising control means for performing the operations. The game system according to claim 10, wherein there are a plurality of light sources, and the information processing apparatus includes a light source movement direction calculation unit that calculates a movement direction of the light source in addition to the light source counting unit and the movement speed unit.
In addition to the number of light sources counted by the light source counting means, in accordance with a transient change in the moving direction of the light source calculated by the moving direction calculating means and the moving speed of the light source calculated by the moving speed means, A game system for performing an operation using a plurality of light sources, wherein the information processing apparatus includes control means for performing a predetermined operation.   12. The game system according to claim 1, wherein the plurality of light sources are flat mats or plates having a plurality of light sources, and the information processing device counts the number of the plurality of light sources. And having a control means for performing a predetermined operation in accordance with the reduced position when the light source count number is reduced, a game system that operates using a plurality of light sources .   12. The game system according to claim 1, wherein the light source and the plurality of light sources having an edge with a specific color around the light source are in a ring or a band shape and are attached to a human part. A game system that operates using multiple light sources.   12. The game system according to claim 1, wherein the light source and the plurality of light sources having a specific color around the light source are in a glove shape, a shoe shape, or a hat shape, A game system that operates using multiple light sources. 15. The game system according to claim 13, wherein the light source is a ring or a band or a glove or a shoe, and is attached to a hand.
If the light source is attached to the palm of the hand or finger and the hand is gripped or the finger is folded, the light source is concealed. Like, the light source is mounted,
When the light source count number increases or decreases, and the distance by the distance calculation unit is equal to or greater than a certain value or less than a certain value, the information processing apparatus determines the position of the light source count number according to claim 5, claim 6, or claim 7. A game system for performing an operation using a plurality of light sources, characterized by comprising control means for specifying and performing a predetermined operation.   15. The game system according to claim 13 or 14, wherein a plurality of light sources are attached to both hands or both feet, and the information processing device is configured to include the light source counting means, the moving speed means, and the moving direction according to claim 11. A calculating means having a determining means for determining a punching action by a hand, a kicking action by a foot, a clapping action by a hand, or a stepping action by a foot by the calculating means; The game system which performs operation using the several light source characterized by comprising the control means which performs said operation | movement.   15. The game system according to claim 13 or claim 14, wherein the plurality of light sources are a band, a ring, a glove, a shoe shape, or a hat shape to be worn on a human part, each having a plurality of light sources. The information processing apparatus includes a determination unit that determines a direction based on a positional relationship of the light source, and includes a control unit configured to perform a predetermined operation according to the direction. A game system that operates using a light source.   18. The game system according to claim 1, wherein when there are at least two light sources, the control unit of the information processing apparatus displays an object having a size corresponding to the distance by the distance calculation unit on a display device. A game system that operates using a plurality of light sources.   19. The game system according to claim 1, wherein when there are at least two light sources, the control unit of the information processing apparatus causes the display device to display a distance when the distance calculated by the distance calculation unit becomes a predetermined value or less. Using a plurality of light sources characterized in that an object to be displayed is selected or sandwiched, and when the distance by the distance calculation means exceeds a certain value, the selection of the object is determined or canceled or released. A game system that performs operations.   20. The game system according to claim 1, wherein at least two position coordinates are determined by the predetermined operation by guidance of a display device connected to the information processing apparatus. And a game system for performing an operation using a plurality of light sources.   The game system according to any one of claims 1 to 5, wherein the light source is a light source having a predetermined color or a color body having a prominent color. A game system for performing operation using a plurality of light sources, wherein a predetermined color component is extracted by a filter and the predetermined color component is used in place of the light source.   The game system according to any one of claims 1 to 19, wherein a display device connected to the information processing device includes a guidance pose for guiding a pose with a teaching human-type display or a display equivalent thereto, In the case of a game in which a game player wearing a plurality of light sources poses a guidance pose displayed on the display device, the game system reads the shape of the pose from the positions of the plurality of light sources when the game player poses. A game system that operates using a plurality of light sources, characterized by calculating a deviation from a guiding pose and a phase difference of a position and displaying them on a display device in the form of a score or a numerical value.   23. The game system according to claim 22, wherein the guided pose is a continuous pose movement or motion, and the game system includes a plurality of continuous light sources when the game player performs a continuous pose movement or motion. The movement of the pose is read from the position data, the movement of the continuous pose of the guided pose, the deviation from the motion and the phase difference of the position are calculated and displayed on the display device in the form of scores and numerical values. A game system that operates using multiple light sources.