JP2003084229A - Device and method for displaying pointed position - Google Patents

Abstract

PROBLEM TO BE SOLVED: To clearly point out a position that a user points without operating an instrument emitting a light beam by the user. SOLUTION: A three-dimensional model representing the three-dimensional coordinates of respective parts in a pointing object space (space in front of the user 12) that the user 12 points through pointing operation is stored and video cameras 28A and 28B whose positions and directions are so adjusted that the hand of the user 12 in the pointing operation is put in an image pickup range pick up images; and the three-dimensional coordinate values of a pointed position in the pointing operation of the user 12 are computed and the projection direction of a laser beam is controlled according to the three-dimensional coordinate values of the pointed position so that the pointed position is irradiated with the laser beam which is emitted from a laser light source 32 and deflected on a deflecting device 34.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a pointing position display device and method, and more particularly to a pointing position display method for irradiating a light beam to a position pointed by a user, and a pointing position to which the pointing position display method can be applied. Regarding display device.

[0002]

2. Description of the Related Art Conventionally, a laser pointer has been known as a tool for allowing a third party to recognize a position pointed by an instructor. The laser pointer is a laser diode (L
D) is built in, and the laser beam is emitted when the switch is turned on by the instructor who holds the laser pointer. The instructor adjusts the direction of the laser pointer to emit the laser beam to a desired position. A spot can be formed, and a third party can recognize the position pointed by the pointer by this light spot.

[0003]

By the way, for example, in museums and the like, many exhibits are displayed, and signboards and the like in which explanations for each exhibit are written are also installed. Exhibits that have different items are different for each user, and the user may be interested in the details of a particular exhibit. However, when a large amount of information is presented by a signboard or the like, there is a problem that the display of the exhibit is obstructed, and an efficient method of presenting information to the user is being studied.

Therefore, using the above-mentioned laser pointer,
For example, a laser pointer is installed in the hall so that the user can freely operate it, and a sensor that detects the laser beam emitted from the laser pointer is attached to the exhibit, and when the sensor detects the laser beam, the exhibit with the sensor attached or It is conceivable to present the information related to the specific part of the exhibit to the visitors by image display or audio output. However, if the laser pointer is installed in a public place as described above, the laser pointer may be damaged or stolen, and an accident or the like may occur due to an operation error by the user. It is not preferable to operably equip them.

On the other hand, in order for the user to easily point to a desired location, it is necessary for the user to visually confirm the current designated position and adjust the designated position. It is desirable to irradiate a light beam to the position pointed by the user to form a light spot for confirming the pointed position of, but in order to form the light spot at the position pointed by the user, the user must move the laser pointer. The reality was that there was nothing but operation.

The present invention has been made in consideration of the above facts, and a pointing position display method and an instruction for indicating a position pointed by a user without the user operating an instrument for emitting a light beam. The aim is to obtain a position display device.

[0007]

In order to achieve the above object, the pointing position display device according to the invention of claim 1 is a pointed object pointed by a pointing operation in which a user points a desired position. Alternatively, a storage unit that stores a three-dimensional model that represents the three-dimensional coordinates of each part of the designated space, an image capturing unit that captures an image of the user from a plurality of mutually different directions, and an image capturing the situation in which the user performs an instruction operation Based on a plurality of images obtained by the image capturing means and a three-dimensional model stored in the storage means, a determination means for determining a pointed position by an instruction operation by the user, and a determination means And a light beam emitting means for emitting the light beam so that the light beam is irradiated to the designated position.

According to the first aspect of the present invention, the storage means stores the three-dimensional model representing the three-dimensional coordinates of the pointed object pointed to by the user's pointing operation pointing to a desired position or each part of the pointed space. Remembered Further, in the invention according to claim 1, the user is imaged by the imaging means from a plurality of different directions, and the determination means is obtained by the imaging means imaging the situation in which the user is performing the instruction operation. Based on the plurality of images and the three-dimensional model stored in the storage unit, the position pointed by the user's pointing motion is determined.

The judgment of the designated position by the judging means is as follows.
Specifically, for example, as described in claim 2, the pointing direction according to the pointing operation of the user is determined based on the plurality of images obtained by the imaging, and stored in the storage unit.
It can be realized by recognizing the nearest object existing on the virtual line extending along the determined pointing direction from the user based on the dimensional model and determining the intersection of the recognized object and the virtual line as the pointing position. .

Further, the judgment of the pointing direction by the user's pointing operation is, for example, a characteristic point whose position changes in accordance with a change in the pointing direction by the user's pointing operation (for example, this characteristic point is, for example, For the tip of the user's hand or finger, or the point corresponding to the tip of the indicator held by the user, etc.), the position on each of a plurality of images obtained by imaging can be obtained. , The three-dimensional coordinates of the feature point are obtained from the positions of the feature points on the plurality of images, and the position does not change regardless of the change in the pointing direction due to the user's pointing action (actually within the image pickup range of the image pickup means. Point (for example, a point corresponding to the chest of the user, the base of the arm, the shoulder, or the like, or a virtual point outside the imaging range) is obtained, and the three-dimensional coordinates are obtained as a reference. Extension of an imaginary line passing through points and feature points It can be realized by determining the direction and pointing direction.

Then, the light beam emitting means emits the light beam so that the light beam is irradiated to the designated position judged by the judging means. As described above, according to the first aspect of the present invention, the light spot is formed by determining the pointing position by the pointing operation of the user and irradiating the determined pointing position with the light beam. It is possible to clearly indicate the position pointed to by the user without operating a device for ejecting.

According to the first aspect of the invention, since the pointing position according to the pointing operation of the user is determined using the image obtained by picking up the image of the user, it is necessary to separately detect the pointing position by the user. It is not necessary to provide a sensor or the like.

The three-dimensional model stored in the storage means may be a complete three-dimensional model representing the three-dimensional coordinates of each part on the entire surface of the designated object or the designated space. There is a problem that it takes a lot of time and effort to generate such a perfect three-dimensional model. Therefore, the three-dimensional model stored in the storage means is, as described in claim 3, near the predetermined position at which the user performs the pointing action and on the head of the user who is performing the pointing action. With a plurality of image pickup devices installed at mutually different positions near the height estimated to be equivalent, a light beam is emitted from the light beam emitting means toward the pointed object or the pointed space, and the plurality of image pickup devices perform the light beam emission. Recognizing the three-dimensional coordinates of the position where the light spot of the light beam is formed based on a plurality of images obtained by imaging the pointing object or the pointed space, It is preferable that the three-dimensional model is generated by repeating while changing the injection direction of.

In this case, the generated three-dimensional model has three-dimensional coordinates only for each part of the entire surface of the designated object or the designated space, which is directly visible to the user who performs the pointing operation at a predetermined position. However, since the position that can be pointed by the user's pointing operation at the predetermined position is limited to the surface that is directly visible to the user, there is no practical problem and it is simple. A three-dimensional model can be obtained by various processes.

Further, the position that the user can point differs depending on the physique of the user. Considering this, claim 2
In the described invention, for example, as described in claim 4, the storage means stores a plurality of three-dimensional models respectively generated in a state in which the plurality of imaging devices are installed at different heights from each other. The means determines the height of the user's head on the basis of the plurality of images obtained by imaging, and corresponds to the determined height among the plurality of three-dimensional models stored in the storage means. It is preferable to use a dimensional model to determine the designated position. This makes it possible to accurately determine the position pointed to by the user's pointing action regardless of the user's physique or the like.

The light beam emitting means according to the first aspect of the invention is, for example, as described in the fifth aspect, a light source for emitting a light beam and a light beam incident from the light source in an arbitrary direction. A possible emitting direction changing means, and an emitting direction control means for controlling the emitting direction changing means so that the light beam emitted from the emitting direction changing means is irradiated to the designated position judged by the judging means. Can be configured. As the light source for emitting the light beam, for example, a laser light source or another known light source can be used.

Further, according to the first aspect of the invention, it is difficult to dispose the light beam emitting means on the virtual line corresponding to the pointing direction according to the pointing operation of the user. Therefore, at the pointing position judged by the judging means. When irradiating the light beam, it is necessary for the light beam emitting means to determine the emission direction of the light beam based on the designated position. This is, for example, as described in claim 6, the emission direction control in which the emission direction of the light beam by the light beam emission means and the three-dimensional coordinates of the position where a light spot is formed by the emitted light beam are associated with each other. Information is stored in the storage means, and the light beam emission means stores the emission direction of the light beam for irradiating the light beam at the designated position determined by the determination means with the emission direction control information stored in the storage means. It can be realized by judging based on. This makes it possible to easily realize the irradiation of the light beam to the designated position determined by the determination means.

Further, the emission direction control information can be generated at the same time when the three-dimensional model of the designated object or the designated space is generated, as described in claim 7, for example. For example, in the generation of the three-dimensional model described in claim 3, it is possible to generate a 3D model at a position where the light spot of the light beam is formed.
Since the three-dimensional coordinates are recognized, the three-dimensional coordinates of the recognized light spot formation position are stored in association with the light beam emitting direction from the light beam emitting means, thereby generating the three-dimensional model and at the same time. Direction control information can also be generated.

Also, in the invention described in claim 1, since the user can point a desired position by the pointing operation, the user's pointing operation points a position within the unfavorable safety range. There is a possibility. Also,
When the designated object is an object having a gap, such as a skeleton specimen of a living thing, the user is pointing to the gap part due to a slight deviation of the position pointed by the user, or the user is pointing to the gap part. If there is, there is a possibility that the judging means may make a wrong judgment.

Considering the above, in the invention described in claim 1, for example, as described in claim 8, the light beam emitting means emits the light beam when the irradiation position of the light beam is out of the predetermined range. It is preferable to stop. The above-mentioned predetermined range may be set in consideration of safety.
As a result, it is possible to prevent the light beam emitted from the light beam emitting means from irradiating the outside of the predetermined range regardless of the deviation of the pointing position by the user and the erroneous judgment of the judging means, thereby ensuring safety. can do.

Further, in the invention according to the first aspect, since the pointing position by the pointing operation of the user is judged using the image obtained by picking up the image of the user, the display means capable of displaying an arbitrary image is provided. In this case, for example, as described in claim 9, when the designated position judged by the judging means is within the display surface of the display means, the emission of the light beam from the light beam emitting means is stopped. At the same time, a control is performed so that an image (for example, a cursor) that clearly indicates the designated position is displayed at a position corresponding to the designated position on the display surface.
It is also possible to provide control means. This makes it possible to realize a seamless (seamless) indication position regardless of whether the indication position indicated by the user's indication operation is inside or outside the display surface of the display means.

Further, in the invention described in claim 1, for example, as described in claim 10, at least one of a display means capable of displaying an arbitrary image and a sound output means capable of outputting an arbitrary sound, and a user. A data storage means for storing at least one of image data and audio data representing information on a predetermined object which can be pointed by a pointing operation; and a position where the pointing position judged by the judging means corresponds to the predetermined object. In the case of the above, based on the data stored in the data storage means, causing the display means to display an image representing the information regarding the predetermined object, and producing a voice representing the information regarding the predetermined object. It is preferable to further include a second control unit that performs at least one of outputting by the voice output unit.

According to the tenth aspect of the present invention, when the position pointed by the user's pointing operation is the position corresponding to the predetermined object, information on the predetermined object is obtained based on the data stored in the data storage means. Displaying an image representing (for example, information describing the display if the predetermined object is a museum display) on the display means, and outputting a voice representing information regarding the predetermined object by the voice output means. Since at least one of the above is performed, the user can easily obtain desired information.

Further, in the invention described in claim 1, as described in claim 11, a specific operation (for example, the whole object or one of the objects) is performed at the pointing position judged by the judging means in accordance with an instruction from the outside. There is an object capable of moving parts, emitting light, emitting sound, emitting a smoke screen, etc.), and the user based on a plurality of images obtained by the image capturing means. When it is detected that the object has performed a predetermined motion, the object may be controlled to perform the specific motion.

In the pointed position display method according to the twelfth aspect of the present invention, a three-dimensional coordinate indicating the three-dimensional coordinates of each pointed object or pointed space pointed by the user performing a pointing operation to point at a desired position. The dimensional model is stored in the storage unit, and the situation in which the user is performing the instruction operation is captured by the image capturing unit that captures the user from a plurality of different directions, and a plurality of images obtained by the image capturing are stored. Based on the three-dimensional model stored in the storage means, the pointing position by the pointing operation of the user is determined, and the light beam is emitted from the light beam emitting means so that the light beam is irradiated to the determined pointing position. Since the light is emitted, the position pointed by the user can be specified without operating the device for emitting the light beam by the user, as in the first aspect of the invention.

[0026]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 shows a hand pointing device 10 according to this embodiment. Note that the hand pointing device 10 according to the present embodiment
Is installed in a public place where an exhibit that can be pointed to by a user is exhibited, such as a museum. The hand pointing device 10 has a function as a pointing position display device according to the present invention.

The hand pointing device 10 is a user 1
2 has a stage 14 on which the stage 1 is mounted.
A foot switch 16 for detecting that the user 12 is placed on the stage 14 is embedded in the floor surface of No. 4 (instead of this, a detecting means such as a load cell is laid on the entire surface of the floor. May be). Foot switch 16
Is connected to the control unit 42 (see FIG. 2). The user 12 on the stage 14 performs an instruction operation for pointing a desired exhibit in front of the stage 14 and a click operation for moving the hand forward or backward from the state of performing the instruction operation. The hand pointing device 10 presents information (for example, image display and audio output: details will be described later) related to the exhibit instructed by the instruction operation.
To run.

A console 18 is provided in front of the stage 14, and a display 20 for displaying various information and a speaker 22 for outputting voice are embedded in the console 18. As the display 20, a known display means such as a liquid crystal display (LCD), a plasma display, a CRT, an optical fiber display or the like can be applied. As shown in FIG. 2, the display 20 is connected to the control unit 42,
The speaker 22 is connected to the control unit 42 via the amplifier 24 and the D / A converter 26. Display of images and the like on the display 20 and output of sound from the speaker 22 are performed by the control unit
Controlled by two.

Two video cameras 28A and 28B are installed near the floor between the stage 14 and the console 18, respectively. The video cameras 28A and 28B correspond to the image pickup means of the present invention. Video camera 28A,
28B images the space (hereinafter referred to as the information input space) on the stage 14 where the user 12 performs the instruction operation and the like from the lower side, and arranges the positions and the images so that the imaging ranges are different from each other. The direction is defined (in FIG. 1, the dashed-dotted line of the video camera 28A and the imaging range of the video camera 28B are shown by broken lines).

Therefore, in the hand pointing device 10 according to the present embodiment, the area (recognition area) in which the operation (instruction operation or pointing operation) by the user 12 can be recognized depends on the imaging range of the video camera 28A and the video camera 28B. The area overlaps the imaging range. As is clear from FIG. 1, in the present embodiment, the user 12 who has arrived in the information input space itself is out of the recognition area, and when the user 12 performs an instruction operation or a click operation, the user 12 hands. The image pickup ranges of the video cameras 28A and 28B are set so that only the video camera 28A and 28B fall within the recognition area. The video cameras 28A and 28B are connected to the control unit 42, and the moving image data obtained by the imaging of the video cameras 28A and 28B is output to the control unit 42.

An illumination device 30 is installed near the position where the video cameras 28A and 28B are arranged. The lighting device 30 has an illumination range adjusted so as to illuminate the recognition region from below (the illumination range of the lighting device 30 is shown by an imaginary line in FIG. 1). Illumination light is emitted to the hands of the user 12 located inside. The lighting device 30 is connected to the control unit 42, and turning on / off of the lighting device 30 is controlled by the control unit 42.

As the video cameras 28A and 28B, an area sensor composed of a CCD or the like having sensitivity to near-infrared light is built-in, and it is provided on the light incident side of an imaging lens that forms an image of incident light on the light receiving surface of the area sensor. If a video camera having a configuration in which a filter that transmits only light having a wavelength in the near infrared region is provided is used, the illumination device 30 can be omitted.

A laser light source 32 is provided inside the console 18. The direction of the laser light source 32 is adjusted so as to emit the laser beam substantially directly above, and the deflection device 3 for deflecting the incident laser beam is provided in the console 18 on the laser beam emitting side of the laser light source 32.
4 are arranged. The laser light source 32 is connected to the control unit 42 (see FIG. 2), and the control unit 42 controls turning on and off. As the laser light source 32, any of a gas laser, a solid-state laser, a liquid laser, and a semiconductor laser can be applied.

As shown in FIG. 3, the deflecting device 34 includes a mirror 36 for reflecting the incident laser beam, a rotary stage 38 rotatable about a horizontal axis, and a mirror 36 attached to the rotary stage 38. A bracket 40 that rotatably supports is provided, and the incident light beam can be deflected and emitted in an arbitrary direction by adjusting the rotation angle φ of the rotary stage 38 and the angle θ of the mirror 36. It is possible. The deflection device 34 includes a rotary stage 38.
Is configured to include an actuator for rotating the mirror and an actuator for rotating the mirror 36 (both not shown), and is connected to the controller 42 (see FIG. 2). The controller 42 rotates the rotary stage 38 by the controller 42. Angle φ
Also, the angle θ of the mirror 36 is controlled.

Thus, the laser light source 32 is the light source according to claim 5, the deflecting device 34 is the emission direction changing means according to claim 5, and the control unit 42 is the emission direction control means according to claim 5. It corresponds to each.

Although not shown, the control unit 42 is a CPU,
It is provided with a ROM, a RAM, and an input / output interface, which are connected to each other via a bus and are housed inside the console 18. The various devices connected to the control unit 42 are specifically connected to the input / output interface of the control unit 42. A program for controlling the operations of the above various devices and executing a setup process, an instruction determination process, and the like, which will be described later, is stored in the ROM. Further, the control unit 42 includes a storage unit 4 having a non-volatile storage medium (for example, a magnetic disk) capable of rewriting stored contents.
4 are connected to each other.

In addition, the control unit 42 includes a setup device 4
6 can be connected. This setup device 46
As shown in FIG. 4, an elevating stage 48 that can be attached at a fixed position on the stage 14, and an elevating stage 48
It is configured to include a plurality of video cameras 50A and 50B that are mounted to be movable up and down with respect to each other and that are vertically moved by an actuator (not shown) (the video cameras 50A and 50B are displaced in the Y direction perpendicular to the paper surface of FIG. 4). Has been placed in the position). The setup device 46 is mounted on the stage 14 and is connected to the control unit 42 when performing a setup process to be described later. The control unit 42 controls the vertical movement of the video camera 50 and also captures images of the video cameras 50A and 50B. The moving image data obtained by the above is output to the control unit 42.

Note that the video cameras 50A and 50B have a space in front of the user 12 on the stage 14 (an exhibition object which the user 12 can indicate if the setup device 46 is mounted on the stage 14). The image capturing direction is adjusted so as to capture an image of the displayed instruction target space). It is desirable that the video cameras 50A and 50B be capable of capturing an image of the entire range of the instruction target space or a range close thereto, and it is preferable to use a video camera having a wide image capturing range.

Next, as the operation of this embodiment, first, when the hand pointing device 10 is installed, the setup process executed by the control unit 42 with the setup device 46 mounted on the stage 18 will be described with reference to the flowchart of FIG. Will be described with reference to.

In step 100, the setup device 4
6 video cameras 50A, 50B to a predetermined initial height z ₀
In step 102, initial values φ ₀ and θ ₀ are set as the injection direction control information (φ, θ). In step 104, a laser beam is emitted from the laser light source 32, and the rotary stage 3 of the deflecting device 34 is emitted according to the current emission direction control information (φ, θ) (that is, φ ₀ , θ ₀ ).
By rotating 8 and the mirror 36,
The laser beam emitted from the laser light source 32 is emitted in the emission direction corresponding to the current emission direction control information (φ, θ).

In the next step 106, the video camera 5
The information input space is imaged by 0A and 50B, and in step 108, a single frame image (image a) is captured from the moving image data input by the video camera 50A capturing the information input space. Step 11
In 0, it is searched whether or not an image portion corresponding to a light spot formed by irradiating the object with the laser beam exists in the image a captured in step 108, and in the next step 112, It is determined whether the corresponding image portion has been extracted by the search in step 110. When the determination is affirmative, the process proceeds to step 114, and the position (X _A , Y _A ) of the image portion corresponding to the light spot on the image a.
Is calculated and the calculation result is temporarily stored in the memory.

In step 116, the video camera 5
An image of a single frame (image b) of the moving image data input by 0B capturing the information input space
Take in. In step 118, it is searched whether or not an image part corresponding to a light spot exists in the image b captured in step 116. In the next step 120, whether the corresponding image part is extracted by the search in step 118. Determine whether or not. If the determination is affirmative, the routine proceeds to step 122, where the position (X _B , Y _B ) of the image portion on the image b corresponding to the light spot is calculated, and the calculation result is temporarily stored in the memory.

By the way, in the present embodiment, as shown in the following Table 1 as an example, the three-dimensional coordinate values (specifically, the video camera 50 ( Video camera 5
Coordinate values (X _C , Y _C , Z _C )) in a three-dimensional coordinate system whose origin is the position of 0A or video camera 50B)
And a coordinate conversion lookup in which the position (X _A , Y _A ) on the image a and the position (X _B , Y _B ) on the image b when the images are taken by the video cameras 50A and 50B are associated with each other. A table (coordinate conversion LUT) is stored in advance in the ROM or the like.

[0044]

[Table 1]

The coordinate conversion LUT captures each position in the image capturing range where the three-dimensional coordinate values (X _C , Y _C , Z _C ) are known by the video cameras 50A and 50B, and the position on the image a ( X _A , Y _A ) and the position (X _B , Y _B ) on the image b can be calculated and stored. For example, the mark plate described in Japanese Patent Laid-Open No. 10-326148 can be used. Can be generated using.

In the next step 124, step 114
Based on the position (X _A , Y _A ) obtained in step S1 and the position (X _B , Y _B ) obtained in step 122, the three-dimensional coordinate value (x) of the light spot whose origin is the position of the video camera 50 is converted by the coordinate conversion LUT. _C , y _C , z _C ) is calculated. Step 126
Then, the three-dimensional coordinate values (x _C ,
y _C , z _C ) is corrected according to the current position of the video camera 50 (for example, the coordinate value in the XYZ coordinate system whose origin is a predetermined reference position such as the installation position of the hand pointing device 10), so that XYZ A three-dimensional coordinate value (x, y, z) of the light spot in the coordinate system is obtained. Then, in step 128, the three-dimensional coordinate values (x, y, z) of the light spot obtained by the correction in step 126 are associated with the current emission direction control information (φ, θ), and the current video camera 50 The information corresponding to the height is registered in the designated position / ejection direction determination table (see also FIG. 1) stored in the storage unit 44, and the process proceeds to step 132.

On the other hand, step 112 or step 120
If the determination is negative, the light spot is out of the imaging range of the video cameras 50A and 50B, or the light spot is located behind an obstacle.
Since the light spot cannot be confirmed on the images a and b, the process proceeds to step 130, and the information indicating that the light spot is not detected is associated with the current emission direction control information (φ, θ).
It is registered in the designated position / ejection direction determination table as information corresponding to the current height of the video camera 50, and step 13
Move to 2.

In the next step 132, it is judged whether or not the emission direction of the laser beam has been changed to the final direction. In the setup processing, the change order of the emission directions is predetermined so that the entire imaging range of the video cameras 50A and 50B is scanned in a predetermined order (for example, raster scan). If the determination in step 132 is negative, the process proceeds to step 134, the injection direction control information is updated according to a predetermined order of changing the injection direction, and step 10 is executed.
Return to 4. As a result, steps 104 to 134 are repeated until the determination at step 132 becomes affirmative.

When the determination at step 132 is affirmative, the routine proceeds to step 136, where it is determined whether the video camera 50 has moved to the final height of the setup processing. The height of the user 12 of the hand pointing device 10 is indefinite,
In consideration of the fact that the position of the head of the user 12 (specifically, the position of the eye of the user 12) also differs depending on the height of the individual user 12, the height of the head of the user 12 is set in the setup process. The height range corresponding to the maximum value to the minimum value (all predicted values) is set as the moving range of the video camera 50, and the video camera 50 is moved at a predetermined pitch within the moving range.
The processing described above is performed at each position.

Therefore, if the determination in step 136 is negative, the process proceeds to step 138 and the video camera 5
After changing the height of 0 by a predetermined amount, the process returns to step 102.
As a result, steps 102 to 138 are repeated until the determination at step 136 becomes affirmative. Then, when the determination in step 136 is affirmative, the setup process ends.

By the above-mentioned setup processing, as shown in Table 2 below, in the designated position emission direction table, laser beam emission control information (for each position (height) in the moving range of the video camera 50 ( φ, θ) and the three-dimensional coordinate value (x, y, of the light spot calculated based on the images a, b)
z) will be stored in association with each other. Further, with respect to the emission control information (φ, θ) corresponding to the emission direction in which the light spot cannot be detected on the images a and b, information indicating “light spot undetected” is stored in association.

[0052]

[Table 2]

In the above setup process, the emission direction of the laser beam is sequentially changed so that the laser beam is scanned over the entire imaging range of the video cameras 50A and 50B, and the emission direction of the laser beam is in each direction. Since the three-dimensional coordinate value of the light spot is obtained from the position of the light spot on the images a and b at the time, when the laser beam is applied to the instruction target object such as an exhibit, The three-dimensional coordinate value of the light spot formed on the surface is registered in the designated position / emission direction determination table.

Therefore, the "light spot three-dimensional coordinates" in the above-mentioned designated position / ejection direction judgment table is "three-dimensional model representing the three-dimensional coordinates of the designated object or each part of the designated space" (details). Corresponds to a plurality of three-dimensional models described in claim 4, and the setup process corresponds to generation of the three-dimensional model described in claim 3. In addition, “Light spot 3” in the designated position / ejection direction determination table
The "dimensional coordinate" corresponds to the injection direction control information described in claim 6 and claim 7, as well as the "injection direction control information" in the table, and the storage unit 44 stores the claim 1, claim 3, claim 3, and claim 3. It corresponds to the storage means described in claim 4 and claim 6, respectively.

Next, the instruction determination processing routinely executed by the control unit 42 after the above setup processing is completed will be described with reference to the flowchart of FIG. In step 150, it is determined whether or not the user 12 has arrived based on whether or not the foot switch 16 is turned on, and the process waits until the determination is affirmative. When the user 12 is placed on the stage 14 and the foot switch 16 is turned on, the determination at step 150 is affirmative and step 152 is performed.
Then, various flags and processing modes are initialized.

In the next step 154, it is determined whether or not the user 12 has left based on whether or not the foot switch 16 has been turned off. If user 12 leaves,
If the determination is affirmative, the process returns to step 150 and waits until the user 12 arrives again in the information input space. If the determination is negative, the feature point / reference point coordinate calculation process is performed in step 156. . Hereinafter, this characteristic point / reference point coordinate calculation processing will be described with reference to the flowchart in FIG. 7.

In step 200, the video camera 28A
The image of a single frame (image A) is captured from the moving image data input from, and in the next step 202, the captured image A data is stored in a memory or the like. In step 204, it is determined whether or not the image data of the image A has been acquired and stored in the previous processing cycle. Since the image data of the image A is not stored when the user 12 arrives and first performs the feature point / reference point coordinate calculation process,
When the determination at step 204 is negative, the characteristic point / reference point coordinate calculation processing ends, but at the time of the second and subsequent executions of the processing, the image data of the image A captured in the previous processing cycle is stored in the memory or the like. Therefore, the determination is affirmative and the process proceeds to step 206.

In step 206, the image data of the image A captured in the previous processing cycle and the image data of the image A captured this time are used to calculate the difference between the images represented by the image data for each pixel. By this calculation, among the objects existing within the imaging range of the video camera 28A, only the pixels corresponding to the objects that have moved from the previous processing cycle to the present time have "there is a difference". By extracting only the pixels with "difference", the image area corresponding to the object that has moved is extracted.

When the hand of the user 12 is located in the recognition area, not only when the user 12 performs a click operation such as moving the hand back and forth, but also when the user 12 performs an instruction operation. Since it is difficult to keep the hand completely still in the air even when performing the operation, the region corresponding to the hand of the user 12 moves in most cases between the previous processing cycle and the present. Will be extracted as an image area corresponding to the object with.

In the next step 208, step 206
It is determined whether the image area corresponding to the hand of the user 12 has been extracted by the processing of. Since the object existing in the recognition area is illuminated by the lighting device 30, when the hand of the user 12 is present in the recognition area, the object in the image A and the image B described later is not used. The image region corresponding to the hand of the person 12 appears as a continuous region having an area of a predetermined value or more, which is composed of high-luminance pixels. Therefore, if the hand of the user 12 exists in the recognition area, the image area corresponding to the hand of the user 12 can be easily extracted.

If the image area corresponding to the moving object is not extracted in step 206, and
If it is determined that the image region extracted in 06 is not the image region corresponding to the hand of the user 12, it can be determined that the hand of the user 12 does not exist in the recognition region, and therefore the determination in step 208 is made. When the result is negative, the feature point / reference point coordinate calculation process ends.

On the other hand, if the determination in step 208 is affirmative, the process proceeds to step 210. In this embodiment,
The fingertip of the user's 12 hand is used as the feature point P _X (see also FIG. 9) for recognizing the pointing action and the clicking action by the user 12, and in step 210, on the image A represented by the image data captured this time. The position of the characteristic point P _X (plane coordinate value, see point P _A shown in FIG. 8) is calculated.

In the next step 212, a single frame image (image B) is captured from the moving image data input from the video camera 28B, and the next step 21
At 4, the captured image data is stored in the memory or the like. In step 216, the image B captured in the previous processing cycle
By using the image data of No. 1 and the image data of the image B captured this time, the difference between the images represented by both image data is calculated for each pixel, and only the pixels with “difference” are extracted, so that there is motion. An image area corresponding to the object is extracted. Then, in step 218, the position of the characteristic point P _X on the image B represented by the image data captured this time (plane coordinate value, see point P _B shown in FIG. 8) is calculated.

At the next step 220, the three-dimensional coordinates of the feature point P _X are calculated based on the position of the feature point P _X on the images A and B. The three-dimensional coordinates of the feature point P _X can be calculated as follows, for example.

That is, as shown in FIG. 8, the three-dimensional coordinate C of the video camera 28A is (X, 0, Z) and the three-dimensional coordinate C'of the video camera 28B is (X ', 0, Z).
In addition, the two-dimensional coordinates P _A of the feature point P _X on the light receiving surface of the image pickup device of the video camera 28A are (α ₁ , β ₁ ) two-dimensionally from the image center O _A of the light receiving surface of the image pickup device of the video camera 28A. The distance to the coordinate P _A is r, the coordinate P _B of the feature point P _X on the light receiving surface of the image sensor of the video camera 28B is (α ₁ ', β ₁ '),
Image center O _B of the light receiving surface of the image sensor of the video camera 28B
The distance from to the coordinates P _B r ', the three-dimensional coordinates P _X of the characteristic point P _X and (x, y, z).

Further, the three-dimensional coordinates S of the point S where the angle between the foot of the perpendicular line passing through the three-dimensional coordinate position of the video camera 28A and the foot of the perpendicular line passing through the point P is a right angle (X, 0, z), The three-dimensional coordinate S ′ of the point S ′ at which the angle between the foot of the perpendicular line passing through the three-dimensional coordinate position of the video camera 28B and the foot of the perpendicular line passing through the point P is a right angle (X ′, 0, z) is the point PCS. The angle formed by the angle θ ₁ ,
The angle formed by the point PC ′S ′ is an angle θ ₁ ′, the angle formed by the point PSS ′ is an angle φ, and the angle formed by the point PS ′S is an angle φ ′.

The distance r from the image center O _A to the image on the light receiving surface of the image pickup device is obtained by r = fθ ₁ . Further, α ₁ and β ₁ are α ₁ = fθ ₁ cos (π−φ) = − fθ ₁ cosφ β ₁ = fθ ₁ sin (π−φ) = fθ ₁ sinφ (1), respectively. Here, sin φ = y / {(x−X) ² + y ² } ^1/2 cos φ = (x−X) / {(x−X) ² + y ² } ^1/2 (2), so α ₁ and β ₁ are α ₁ = −fθ ₁ (x−X) / {(x−X) ² + y ² } ^1/2 (3) β ₁ = fθ ₁ y / {(x−X) ² + y ² } ^1/2 ... (4) can be obtained. By dividing equation (10) by equation (3), y = (β ₁ / α ₁ ) (X−x) (5) Similarly, y = (β ₁ ′ / α ₁ ′) (X ′ −x) (6) Eliminating y from the equations (5) and (6), x = (α ₁ β ₁ ′ X′−α ₁ ′ β ₁ X) / (α ₁ β ₁ ′ −α) ₁ 'beta ₁₎ ... it is possible to obtain the X-coordinate of the three-dimensional coordinates P _X of the characteristic point P _X (7).

Next, x is eliminated from the equations (5) and (7), and y = β ₁ β ₁ ′ (X−X ′) / (α ₁ β ₁ ′ −α ₁ ′ β ₁ ) ... By (8), the Y coordinate of the three-dimensional coordinate P _X can be obtained.

Since θ ₁ = tan ^-1 [{(x-X) ² + y ² } ^1/2 / (Z-z)], β ₁ / (from equations (7) and (8) fsin φ) = tan ⁻¹ [{(x−X) ² + y ² } ^1/2
/ (Z−z)] Therefore, z = Z − [{(x−X) ² + y ² } ^1/2 / tan [(β ₁ / f) · (x−X) ² + y ² } ^1/2 / y] (9) Further, from the expression (5), {(x−X) ² + y ² } ^1/2 = (x−X) {1+ (β ₁ / α ₁ ) ² }
^{1/2 From} equations (5) and (8), (x−X) = (X′−X) / {1- (α ₁ ′ / α ₁ ) × (β ₁ /
Since β ₁ ')}, the equation (9) is expressed by z = Z + [(X'-X) {1+ (β ₁ / α ₁ ) ² } ^1/2 / {1- (α ₁ ' / α _{1 ) · (β 1 / β 1} ')}] / tan {(α 1 2 + β 1 2) 1/2 / f} ... ( can be expressed as 10), to obtain the Z coordinate of the three-dimensional coordinates P _X You can

As described above, the video cameras 28A, 28
Since the three-dimensional coordinates of B are already known, equations (7) and (8)
(10) and the feature point P on the image A_XFrom the position
The special feature on the light-receiving surface of the image sensor of the video camera 28A to be obtained
Point P_XPlane coordinates P of_A(α₁, Β ₁) And the special feature on image B
Point P_XOf the video camera 28B obtained from the position
Feature point P on the light-receiving surface of the child_XPlane coordinates P of_B(α₁',
β₁'), The value of_XThree-dimensional coordinates
P_X(X, y, z) can be obtained.

In the next step 222, the characteristic point / reference point coordinate calculation processing is repeatedly executed, resulting in step 2
Among the three-dimensional coordinates of the characteristic point P _X repeatedly calculated in 20, the calculation result of the three-dimensional coordinates of the characteristic point P _X in the latest plurality of cycles and the calculation of the three-dimensional coordinates of the characteristic point P _X in the current cycle. The position (three-dimensional coordinate) of the elbow and shoulder of the user 12 is estimated and calculated based on the movement locus of the characteristic point P _X obtained from the result.

Generally, in the pointing operation for pointing a specific position, when moving the pointing position in the vertical direction, the arm is moved around the shoulder, and when moving the pointing position in the horizontal direction, the elbow is rotated. As the arm is moved. Therefore, when the user 12 performing the pointing action performs the action of moving the pointing position, the feature point P _X moves on a spherical surface centered on the elbow or shoulder. Therefore, by substituting the movement locus of the feature point P _X into a general expression that defines the rotational movement about an arbitrary point as a rotation center and finding a solution for each time segment using the least squares method, The position of the center of rotation (ie the position of the elbow or shoulder) and the radius of gyration can be estimated and calculated. Also, using the above estimation operation,
It is also possible to determine the approximate physique of the user 12 (whether the user 12 is an adult or a child) by obtaining the change width of the position of the rotation center and the change width of the rotation radius.

[0073] Then, in the next step 224, as the reference point P ₀ is located at the estimated computed shoulder position of the user 12 in step 222, the three-dimensional coordinates of the reference point P ₀ (see FIG. 9) To set. The shoulder position of the user 12 estimated and calculated in step 222 is the position of the shoulder on the arm side (the left shoulder for the left arm, the right shoulder for the right arm) used by the user 12 for the pointing operation. If the arm used by the user 12 for the pointing motion is the left arm, the reference point P ₀ is set to the position shown in FIGS. 9A and 9B, and the arm used by the user 12 for the pointing motion is the right arm. If so, the reference point P ₀ will be at the position shown in FIGS. 9A and 9C.

The fact that the heights of the elbows and shoulders of the user 12 are approximately proportional to the height of the user 12 is used, and the next step 2
In step 26, the height of the user 12 is estimated and calculated based on the heights of the elbows and shoulders of the user 12 estimated and calculated in step 222. When the processing of step 226 is performed, the characteristic point / reference point coordinate calculation processing is ended, and step 1 of the instruction determination processing (FIG. 6) is performed.
Move to 58.

Since the feature point / reference point coordinate calculation process described above is repeatedly executed, the three-dimensional coordinate value of the feature point P _X calculated by this process depends on the posture and movement of the user 12. Will be updated sequentially. On the other hand, it is not preferable that the three-dimensional coordinate value of the reference point P ₀ greatly changes each time the feature point / reference point coordinate calculation processing is executed.
For example, the three-dimensional coordinate value of the reference point P ₀ that the user 12 first arrives and obtains first may be fixedly used until the user 12 leaves, or the characteristic point / reference point coordinates may be used. Each time the arithmetic processing is executed, the reference point P ₀ is adjusted so that the deviation from the previously obtained three-dimensional coordinate value of the reference point P ₀ falls within a predetermined range.
You may make it obtain | require the three-dimensional coordinate value of.

In step 158, it is determined whether or not the user 12 is performing an instruction operation based on the determination result of step 208 of the characteristic point / reference point coordinate calculation processing (FIG. 7) described above. In the feature point / reference point coordinate calculation process, the feature point P _X 3 is calculated only when the user 12 is performing the pointing operation (when the determination in step 208 is affirmative).
Since the dimensional coordinate value is calculated, the determination in step 158 is
For example, it can be performed by determining whether or not the three-dimensional coordinate value of the characteristic point P _X has been calculated. If this determination is denied, the process returns to step 154, and steps 154 to 158 are repeated while the user 12 exists in the information input space.

On the other hand, if the determination in step 158 is affirmative, the routine proceeds to step 160, and first, the reference point P ₀ and the characteristic point P _X calculated in the preceding characteristic point / reference point coordinate calculation processing.
Based on the three-dimensional coordinate values of the above, as a line representing the pointing direction by the pointing operation of the user 12, the reference point P ₀ and the characteristic point P _X
A virtual line (see virtual line 54 in FIG. 9) connecting with is set.
Further, the estimation calculation result of the height of the user 12 in step 226 of the characteristic point / reference point coordinate calculation process (FIG. 7) is taken in, and the height (Z coordinate value) corresponding to the eye of the user 12 is estimated and calculated. . Then, as the coordinate value of the pointed position indicated by the pointing operation of the user 12, among the three-dimensional coordinate values registered in the pointed position / ejection direction determination table in association with the estimated eye height of the user 12 From the reference point P ₀
And a three-dimensional coordinate value existing on a virtual line connecting the feature point P _X and the feature point P _X is searched.

The step 160 corresponds to the determination means described in claims 1, 2 and 4 together with the feature point / reference point coordinate calculation processing (FIG. 7) described above. .

In the next step 162, step 160
It is determined whether or not the corresponding three-dimensional coordinate value is extracted from the designated position / injection direction determination table by the search of. As described above, the three-dimensional coordinate values registered in the designated position / ejection direction determination table are the same as those of the video cameras 50A and 5A.
Since the laser beam is scanned over the entire imaging range of 0B to obtain the position from the image portion corresponding to the light spot on the images a and b, if the determination in step 162 is negative, It can be determined that it is difficult to form a light spot by the laser beam emitted from the laser light source 32 on the virtual line along the direction indicated by the instruction operation of the user 12. Therefore, step 16
When the determination of 2 is denied, the process returns to step 154 without performing any processing (that is, without performing the laser beam emission).

When the determination in step 162 is affirmative, the process proceeds to step 164, and the three-dimensional coordinate value extracted by the search in step 160, that is, the user 1
It is determined whether or not the three-dimensional coordinate value representing the designated position by the designation operation 2 is on the display surface of the display 20. If the determination is positive, the process proceeds to step 172, and the display 2
The cursor is displayed at the corresponding position on the display surface of 0, and the process proceeds to step 174. Note that steps 164 and 17
Reference numeral 2 corresponds to the first control means described in claim 9, and the display 20 corresponds to the display means described in claim 9.

If the determination in step 164 is negative, the process proceeds to step 166, and the user 1 determines based on the three-dimensional coordinate value of the position indicated by the user 12's pointing action.
It is determined whether the position indicated by 2 is out of the predetermined range. In the present invention, since the laser beam is irradiated to the position designated by the user 12, the above-mentioned predetermined range can ensure safety when the laser beam is emitted so that the laser beam is irradiated to the predetermined position of the designated space. Whether or not to determine whether or not to do so is defined in advance for each position in the pointing space.

For example, when the laser beam is emitted so that the laser beam is irradiated to a certain position in the pointing space, the emitted laser beam crosses the passage of the user formed in the pointing space. In that case, the position will be excluded from the predetermined range. In the present embodiment, as will be described later, when the designated position is out of the predetermined range, the laser beam emission is stopped, but instead of this, for example, the laser beam emission / emission direction is changed in each laser beam emission direction. It is also possible to judge whether the injection is stopped and register the information indicating the emission / stop of the laser beam in the designated position / injection direction determination table in association with the injection direction control information.

When the determination in step 166 is negative, the process returns to step 154 without performing any processing (that is, without emitting the laser beam). If the determination in step 166 is affirmative, the process proceeds to step 168, and the injection direction control registered in the designated position / injection direction determination table in association with the three-dimensional coordinate value extracted by the search in step 160 is performed. Information, that is, injection direction control information (φ, θ) for irradiating the laser beam to the position designated by the user 12 is fetched from the designated position / emission direction determination table.

Then, in step 170, the laser light source 3
The laser beam is emitted from 2 and step 16
In step 168, the laser beam emitted from the laser light source 32 is rotated by rotating the rotary stage 38 of the deflecting device 34 and rotating the mirror 36 in accordance with the emission direction control information (φ, θ) captured in 8. The injection is performed in the injection direction according to the captured injection direction control information (φ, θ). As a result, a light spot is formed by the emitted laser beam at the position designated by the pointing operation of the user 12.

Incidentally, the above-mentioned steps 162, 166.
Reference numerals 168 and 170, together with the laser light source 32 and the deflection device 34, correspond to the light beam emitting means of the present invention, and particularly steps 162 and 166 correspond to the light beam emitting means according to the eighth aspect.

At the next step 174, it is judged whether or not the click operation is performed by the user 12. Various actions can be adopted as the click action, for example, an action in which the user 12 quickly moves his or her hand forward (see FIG. 10).
(See (A), hereinafter referred to as “forward click”), and an operation in which the user 12 quickly moves his or her hand backward (FIG. 10).
(See (B), hereinafter referred to as "reverse click"). Since this click operation is an extremely natural operation as an operation of pointing and selecting a specific position, the user 12 can perform the click operation without feeling a sense of discomfort.

Whether or not the forward click operation or the backward click operation is performed is determined by, for example, the distance k between the reference point P ₀ and the characteristic point P _X (FIG. 9) while the user 12 is performing the instruction operation.
(Refer to FIG. 4) is repeatedly performed, and in step 174, it is possible to determine whether or not the difference between the distance k calculated this time and the distance k calculated last time is a predetermined value or more. If the determination in step 174 is negative, the process returns to step 154 without performing any processing.

On the other hand, when the determination in step 174 is affirmative, the process proceeds to step 176, and the process according to the click operation performed by the user 12 is performed. That is,
In the present embodiment, an image (which may be a still image or a moving image) representing information related to an exhibit that can be instructed by the user 12 and audio data are instructed by the surface of the exhibit (instructed by the user 12). Corresponding to the three-dimensional coordinate value of the surface) is stored in the storage unit 44 in advance as exhibit information (see also FIG. 2).

Therefore, when the position pointed by the user 12 is outside the display surface of the display 20 and a light spot is formed at the pointed position by emitting a laser beam, the position pointed by the user 12 is changed. Exhibit information is searched using the three-dimensional coordinate values as a key. Then, when the data stored in association with the three-dimensional coordinate value of the designated position is extracted, if the extracted data is image data, the image represented by the image data is displayed on the display 20, If the extracted data is audio data, the audio data is D / A
By outputting to the speaker 22 via the converter 26 and the amplifier 24, the voice represented by the audio data is output from the speaker 22.

Thus, when the user 12 becomes interested in a particular exhibit, an instruction operation is performed to point to the particular exhibit (this causes the laser beam to reach the designated position on the surface of the particular exhibit). By irradiating to form a light spot), and further performing a click operation, information related to a specific exhibit is presented as an image or a sound.
Also, image data and audio data are respectively stored as exhibit information, and it is possible to switch between displaying an image and outputting audio depending on whether the click operation by the user 12 is a forward click or a backward click. Good.

As described above, the storage section 44 corresponds to the data storage means described in claim 10, and the above-described processing in step 176 corresponds to the second control means described in claim 10. Further, the display 20 is defined in claim 10.
The D / A converter 26,
The amplifier 24 and the speaker 22 correspond to the audio output means described in claim 10.

When the user 12 performs an instruction operation to point to a specific exhibit and then performs a click operation,
Instead of the above, move the entire exhibit or a part of the exhibit,
You may make it perform specific operation | movements, such as making an exhibit light-emit, an audio | voice generate | occur | produce from an exhibit, and a smoke screen being emitted from an exhibit. This aspect corresponds to the invention of claim 11.

When the position pointed by the user is within the display surface of the display 20, the processing relating to the pointed position is performed. For example, when the designated position is a position where a button for instructing execution of a specific process (for example, screen scrolling) is displayed, the specific process is executed. When the process of step 176 is performed, the process returns to step 154. As a result, steps 154 to 176 are repeated until the user 12 leaves, and a light spot is formed at the designated position while the user 12 is performing the pointing operation to point within the predetermined range. Each time you click
The information related to the designated position will be presented to the user 12.

In the above, two video cameras 28 are used.
Although an example in which A and 28B are provided has been described, the present invention is not limited to this, and the information input space may be imaged by a larger number of video cameras and the instruction from the user 12 may be determined. Also, in the above, the video camera 28
Although the example in which A and 28B are arranged on the lower side of the information input space has been described, if the instruction target space is an indoor space, it may be arranged on the upper side of the information input space (for example, the ceiling portion of the instruction target space). . Further, it goes without saying that the laser light source 32 and the deflection device 34 may also be arranged on the ceiling of the instruction target space.

Further, in the above, the user 12 who has arrived in the information input space is out of the recognition area, and only the hand of the user 12 is in the recognition area when the user 12 performs an instruction operation or a click operation. Although the imaging range of the video cameras 28A and 28B is defined in the above, the invention is not limited to this, and the imaging range of the video camera may be defined so that the whole body of the user 12 is within the recognition area.

Further, in the above, by performing the setup process, the information is registered in the designated position / injection direction judgment table in which the three-dimensional model according to the present invention and the injection direction control information described in claim 6 are integrated. However, the present invention is not limited to this (though the 3D model and the injection direction control information are generated at the same time), but the 3D model and the injection direction control information are generated separately and stored separately. Good.

Further, in the above description, the laser beam emitted from the laser light source 32 is used as the light beam according to the present invention, but the invention is not limited to this. For example, a spot light emitted from a spotlight is used. It may be used as a light beam according to the invention. Spot light requires a particularly large amount of light (for example, when the distance between the light source and the illuminated object is large, such as when pointing a large object such as a ruin), or when the illuminated object This is effective when the light reflectance is high (for example, a mirror surface) and a light spot is not formed even when the laser beam is irradiated.

In the above description, the case where the hand pointing device 10 is installed in a public place where an exhibit that can be pointed to by a user is exhibited, such as a museum, has been described as an example. Is not limited to
For example, it can be applied to a presentation. As described above, by applying the present invention,
When the user points to the real object, a light spot is formed at the pointing position on the surface of the pointed object, and when the user points to the inside of the display surface of the display, the light spot is formed at the pointing position inside the display surface. Since seamless (seamless) pointing where cursors etc. are displayed is possible, it can also be applied to product explanations and explanations of historic sites exhibited,
It can contribute to improving the quality of presentations.

[0099]

As described above, according to the inventions of claims 1 and 12, 3 of each part of the designated object or the designated space indicated by the user performing the pointing operation pointing to the desired position is performed. The three-dimensional model representing the three-dimensional coordinates is stored, the situation in which the user is performing the instruction operation is imaged from a plurality of different directions, and based on the plurality of images obtained by the imaging and the three-dimensional model, The position pointed by the user is indicated without judging the pointing position by the pointing operation of the user and irradiating the determined pointing position with the light beam, without operating the device for emitting the light beam. It has an excellent effect of being able to.

According to a third aspect of the present invention, in the first aspect of the present invention, a light beam is emitted toward a pointed object or a pointed space, and a pointing operation is performed near a predetermined position at which a pointing operation is performed by a user. A plurality of images obtained by respectively capturing a designated object or a designated space by a plurality of imaging devices installed at different positions near the height estimated to correspond to the head of the user who is performing the operation Since the three-dimensional model generated by repeating the recognition of the three-dimensional coordinates of the position where the light spot of the light beam is formed based on the above while changing the emission direction of the light beam is stored in the storage means, In addition to the above effects, there is an effect that a three-dimensional model can be obtained by simple processing.

According to a fourth aspect of the present invention, in the third aspect of the present invention, a plurality of three-dimensional models generated in a state in which a plurality of image pickup devices are installed at different heights are stored in a storage means, and image pickup is performed. Since the height of the head of the user is determined based on the plurality of images obtained by, and the designated position is determined using the three-dimensional model corresponding to the determined height among the plurality of three-dimensional models, In addition to the above effects, there is an effect that the position pointed by the user's pointing operation can be accurately determined regardless of the physique of the user.

According to a sixth aspect of the present invention, in the first aspect of the invention, the emission direction control information is associated with the emission direction of the light beam and the three-dimensional coordinates of the position where the light spot is formed by the light beam. Since the emission direction of the light beam for irradiating the light beam to the designated position is determined based on the emission direction control information, it is possible to store the light beam at the designated position in addition to the above effect. It has an effect that it can be easily realized.

According to the invention described in claim 8, in the invention according to claim 1, the emission of the light beam is stopped when the irradiation position of the light beam is out of a predetermined range. The effect that the light beam emitted from the light beam emitting means can be prevented from irradiating the light beam outside the predetermined range regardless of the positional deviation and the erroneous determination of the determining means, and the safety can be ensured Have.

According to a ninth aspect of the invention, in the invention according to the first aspect, when the designated position is within the display surface of the display means, the emission of the light beam is stopped and the designated position within the display surface is reached. In addition to the above-mentioned effect, the image indicating the designated position is displayed at the corresponding position. In addition to the above effect, there is no seam (seamless) regardless of whether the designated position by the user's pointing motion is inside or outside the display surface of the display means. ) There is an effect that it is possible to realize the indication of the designated position.

According to a tenth aspect of the present invention, in the first aspect of the present invention, when the designated position is a position corresponding to a predetermined object, an image representing information on the predetermined object is displayed on the display means. Also, since at least one of outputting the voice representing the information regarding the predetermined object by the voice output means is performed, the effect that the user can easily obtain desired information in addition to the above-mentioned effect is obtained. Have.

[Brief description of drawings]

FIG. 1 is a schematic configuration diagram of a hand pointing device.

FIG. 2 is a block diagram showing a schematic configuration of an electric system of the hand pointing device.

FIG. 3 is a perspective view showing an example of a laser light source and a deflection device.

FIG. 4 is a schematic view showing a state in which a setup device is installed.

FIG. 5 is a flowchart showing the contents of setup processing.

FIG. 6 is a flowchart showing the contents of an instruction determination process.

FIG. 7 is a flowchart showing the contents of feature point / reference point coordinate calculation processing.

FIG. 8 is a conceptual diagram for explaining calculation of three-dimensional coordinates of feature points based on images obtained by image pickup by two video cameras.

9A and 9B are plan views of an information input space, and FIG. 9B is a diagram for explaining determination of a pointed position by a pointing operation of a user.
And (C) are side views of the information input space.

10A is an image diagram for explaining a forward click operation, and FIG. 10B is an image diagram for explaining a backward click operation.

[Explanation of symbols]

10 Hand pointing device 20 display 22 speakers 28A video camera 28B video camera 32 laser light source 34 Deflection device 42 Control unit 44 memory 46 setup equipment 48 lift stage 50 video camera 54 virtual line

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) G06T 1/00 340 G06T 1/00 340Z F term (reference) 2H041 AA12 AB14 AC04 AZ05 5B020 AA15 CC12 CC20 DD06 5B057 BA02 CA08 CA12 CA16 CB20 CC01 DA08 DB02 DB09 DC01 5B068 AA05 AA36 BD09 BE03 BE08 CC06 5B087 AA09 AB09 AE03 BC12 BC13 BC26 BC32 DE07

Claims (12)

[Claims] 1. A storage unit for storing a three-dimensional model representing the three-dimensional coordinates of a designated object or each portion of a designated space pointed by a user performing a pointing operation to point to a desired position, and a plurality of storage units different from each other. To the three-dimensional model stored in the storage means and a plurality of images obtained by the imaging means capturing the situation in which the user is performing the instruction operation. And a light beam emitting unit for emitting a light beam so that the light beam is irradiated to the pointed position determined by the determination unit. Pointing position display device. 2. The judging means judges a pointing direction according to a user's pointing operation based on the plurality of images, and the user makes the judgment based on a three-dimensional model stored in the storage means. 2. The pointing position display device according to claim 1, wherein a nearest object existing on a virtual line extending along the pointing direction is recognized, and an intersection of the recognized object and the virtual line is determined as a pointing position. 3. The height of the three-dimensional model stored in the storage means is estimated to be near a predetermined position at which the user performs the pointing motion and is equivalent to the head of the user performing the pointing motion. In the state where a plurality of image pickup devices are installed at different positions near each other, a light beam is emitted from the light beam emitting means toward the pointed object or the pointed space, and the pointed object is output by the plurality of image pickup devices. Alternatively, recognizing the three-dimensional coordinates of the position where the light spot of the light beam is formed on the basis of a plurality of images obtained by imaging the designated space, The pointing position display device according to claim 1, wherein the pointing position display device is generated by repeating the ejection direction while changing the ejection direction. 4. The storage means stores a plurality of three-dimensional models generated in a state in which the plurality of image pickup devices are installed at different heights, and the determination means stores the plurality of images. The height of the head of the user is determined based on the
The pointing position display device according to claim 3, wherein the pointing position is determined by using a three-dimensional model corresponding to the determined height among the dimensional models. 5. The light beam emitting means emits a light source for emitting a light beam, an emitting direction changing means capable of emitting the light beam incident from the light source in an arbitrary direction, and the emitting direction changing means. 2. An emission direction control unit that controls the emission direction changing unit so that a light beam to be emitted is irradiated to the designated position determined by the determination unit. Pointing position display device. 6. The emission direction control information in which the storage unit associates the emission direction of the light beam by the light beam emission unit with the three-dimensional coordinates of the position where a light spot is formed by the emitted light beam. The light beam emitting means stores the light beam emitting direction for irradiating the light beam to the pointing position determined by the determining means, the light emitting direction control information stored in the storage means. The pointing position display device according to claim 1, wherein the determination is made based on 7. The pointing position display device according to claim 6, wherein the emission direction control information is generated at the same time when the three-dimensional model of the designated object or the designated space is generated. 8. The pointing position display device according to claim 1, wherein the light beam emitting means stops the emission of the light beam when the irradiation position of the light beam deviates from a predetermined range. 9. A display unit capable of displaying an arbitrary image, and when the designated position determined by the determination unit is within the display surface of the display unit, a light beam is emitted from the light beam emitting unit. And a first control unit for controlling so that an image that clearly indicates the designated position is displayed at a position corresponding to the designated position on the display surface. The indicated position display device described. 10. At least one of a display unit capable of displaying an arbitrary image and a voice output unit capable of outputting an arbitrary voice, an image representing information on a predetermined object which can be pointed by a user by an instruction operation, and Data storage means for storing at least one data of voice, and based on the data stored in the data storage means when the designated position determined by the determination means is a position corresponding to the predetermined object Second control means for performing at least one of displaying an image representing information on the predetermined object on the display means and causing a sound representing information on the predetermined object to be output by the audio output means, The pointing position display device according to claim 1, further comprising: 11. An object capable of performing a specific operation in accordance with an instruction from the outside is present at the pointing position determined by the determination means, and the object is obtained by capturing an image by the image capturing means. A third control means for controlling the object to perform the specific operation when it is detected that the user has performed a predetermined operation based on the plurality of captured images. The pointing position display device according to claim 1. 12. A storage unit stores a three-dimensional model representing three-dimensional coordinates of a pointed object pointed by a user performing a pointing operation pointing at a desired position or each part of the pointed space, and the two types are mutually stored. A situation in which the user is performing an instruction operation is captured by an image capturing unit that captures the user from a plurality of different directions, and a plurality of images obtained by the image capturing and a three-dimensional model stored in the storage unit are captured. A pointed position display method for determining a pointed position based on the pointing operation of the user based on the above, and causing a light beam to be emitted from the light beam emitting means so that the light beam is irradiated to the determined pointed position.