JP2010067062A - Input system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To certainly provide a virtual input device even when the surrounding landscape is not fixed or any person performs input operation. <P>SOLUTION: The input device 10 includes: a camera 11 for imaging an object 20 as a virtual input device; an object region recognizing section 12 for recognizing an object region in which the object 20 in the image is positioned based on the imaged image; a feature information acquisition section 14 for acquiring information showing the feature of the object 20; an operation region detecting section 15 for detecting an operation region having a feature different from that of the acquired information in the object region of the imaged image; an input detecting section 16 for detecting user's input operation based on the detected operation region; and an output section 17 for output of information showing the detected user's input operation. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an input system and an input method for inputting information.

In order to virtually provide an input function to an object that does not have an input function for inputting information to a computer or the like, for example, an object is imaged, and a user input to the object from the captured image There is a method of recognizing operations. In order to recognize an input operation, it is necessary to identify an object to be used as a virtual input device and then recognize a tool used for input such as a user's hand. As a method of detecting a moving object, there is a method of detecting from a difference between a past frame and a current frame of an image acquired by a camera as described in Patent Document 1. In addition, as described in Patent Document 2, a technique for recognizing an input operation by analyzing a user's gesture has been proposed.
JP 2006-163452 A JP 2007-87089 A

  However, with the conventional technology described above, it is difficult to recognize a user's input operation in an unrestricted place. For example, in the technique described in Patent Document 1, the backgrounds captured in the past frame and the current frame need to match. Therefore, it cannot be used unless the camera is fixed, and the available place is limited. In Patent Document 2, since the skin color of the hand is recognized, there are cases where the input operation cannot be recognized stably depending on the lighting environment or race. Further, it is impossible to detect an input operation other than a hand.

  The present invention has been made to solve the above problems, and even if the surrounding landscape is not fixed or the input operation is arbitrary, it is ensured. An object of the present invention is to provide an input system and an input method capable of realizing a virtual input device.

  To achieve the above object, an input system according to the present invention includes an imaging unit that images an object, and an object region recognition unit that recognizes an object region where the object in the image is located from an image captured by the imaging unit. A feature information acquisition unit that acquires information indicating the feature of the object, and a feature related to the information acquired by the object feature acquisition unit in the object region recognized by the object region recognition unit in the image captured by the imaging unit An operation area detection means for detecting an operation area having different characteristics from the above, an input detection means for detecting a user input operation based on the operation area detected by the operation area detection means, and a user detected by the input detection means Output means for outputting information indicating the input operation.

  In the input system according to the present invention, an object is imaged, and an object region of the object that is a virtual input device is recognized from the captured image. Subsequently, an operation region having a feature different from the feature of the object is detected in the recognized object region. Subsequently, the user's input operation is detected based on the detected operation area. As described above, in the input system according to the present invention, the recognition of the object region of the object that is a virtual input device is performed, and then the operation region corresponding to the user's input operation on the object region is detected. Therefore, even in an arbitrary situation where the surrounding landscape is not fixed, a virtual input device can be reliably realized. In addition, since the operation of the user can be detected if the feature of the object that is the virtual input device is different from the feature of the operation area according to the user's operation, any device that performs the input operation can be detected. Even so, a virtual input device can be reliably realized.

  It is desirable that the object information acquisition unit acquires information indicating the feature of the object by extracting the feature of the object from the object region recognized by the object region recognition unit in the image captured by the imaging unit. According to this configuration, since the information indicating the characteristics of the object that is a virtual input device is reliably acquired, the present invention can be appropriately implemented.

  The imaging unit images an object over a plurality of times, and the input system further includes a region following unit that recognizes a region corresponding to the region recognized by the object region recognition unit in the time-changed image captured by the imaging unit. It is desirable to provide. According to this configuration, even if the object itself or the imaging direction moves, it is possible to appropriately recognize the object region, thereby appropriately detecting an input operation.

  The input system preferably further includes a projection unit that projects an image in accordance with the object area recognized by the object area recognition unit. According to this configuration, for example, display such as a virtual input device such as a keyboard can be performed, or display corresponding to the detected user operation can be performed. Thereby, a user-friendly virtual input device can be realized.

  The object region recognition means desirably recognizes the object region by detecting a marker provided in advance on the object. The object area recognition means detects a predetermined sound, detects a position corresponding to the object area in the image picked up by the image pickup means at a timing when the sound is detected, and recognizes the object area from the position. It is desirable. It is desirable that the object area recognizing means stores in advance information indicating characteristics related to the object area and recognizes the object area based on the information. According to these configurations, the object region can be reliably recognized, and the present invention can be appropriately implemented.

  It is desirable that the operation area detection unit detects a predetermined sound and detects an operation area in an image captured by the imaging unit at a timing when the sound is detected. According to this configuration, for example, the timing at which the user detects the operation area can be arbitrarily determined, and the user can perform an input operation at an arbitrary timing. Therefore, it is possible to make the user perform an input operation with high operability.

  By the way, the present invention can be described as an invention of an input system as described below, as well as an invention of an input system as described above. This is substantially the same invention only in different categories, and has the same operations and effects.

  That is, the input method according to the present invention shows an imaging step for imaging an object, an object area recognition step for recognizing an object area where the object is located in the image, and an object feature. An operation having a feature different from the feature related to the information acquired in the object feature acquisition step in the object region recognized in the object region recognition step in the feature information acquisition step for acquiring information and the image captured in the imaging step Information indicating an operation region detection step for detecting a region, an input detection step for detecting a user input operation based on the operation region detected in the operation region detection step, and a user input operation detected in the input detection step And an output step for outputting.

  According to the present invention, the object area of the object that is the virtual input device is recognized, and the operation area corresponding to the user's operation on the object area is detected, so the surrounding landscape is fixed. Even in any situation that is not done, a virtual input device can be realized with certainty. In addition, since the operation of the user can be detected if the feature of the object that is the virtual input device is different from the feature of the operation area according to the user's operation, any device that performs the input operation can be detected. Even so, a virtual input device can be reliably realized.

  Hereinafter, preferred embodiments of an input system and an input method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 schematically shows an external configuration of an input device 10 that is an embodiment of an input system according to the present invention. The input device 10 is a glasses-type display (for example, a head mounted display (HDM)), and has a shape that can be worn by the user. The input device 10 has a function of an input device of a computer (not shown). Specifically, the input device 10 projects an image 30 of an input device such as a (virtual) keyboard or touch panel so as to be superimposed on a predetermined object 20, and the image of the input device is projected to the user 40. Recognize The input device 10 detects an input operation (input operation) with a finger or a pen on the video (corresponding to the location of the object 20) by the user 40 and uses it as an input to the computer. The input operation is, for example, an operation of pointing (pointing or clicking) a predetermined portion of the object 20 with a finger or a pen. The computer to which an input operation from the user 40 is input from the input device 10 may be configured integrally with the input device 10 or may be configured separately from the input device 10 (in that case, wireless). Or connected to the input device 10 by wire). As described above, the input device 10 realizes a virtual input device that does not have a physical configuration for input such as a keyboard and a touch panel.

  The predetermined object 20 on which the video 30 of the input device is superimposed preferably has a flat surface, for example, a notebook or notebook held by the user 40 or a fixed wall. Further, the image 30 of the input device may be superimposed on the object 20 by being projected. Further, when the glasses-type display is of an optical see-through type, the surrounding scenery can be seen through the glasses, so that when the user views the object 20 through the lens, the glasses-type display is superimposed. Only the image 30 of the input device may be projected onto the lens. When the eyeglass-type display is a video see-through type, the object 20 captured by the camera may be simultaneously projected onto the lens. At this time, the same image may be projected to both eyes, or an image viewed from each eye may be estimated from the distance between the right eye and the left eye, and the images projected to each eye may be prepared separately. When different videos are projected according to each eye, a video with a three-dimensional effect (depth) can be shown to the user 40.

  Subsequently, functions of the input device 10 will be described. As illustrated in FIG. 2, the input device 10 includes a camera 11, an object region recognition unit 12, a region tracking unit 13, a feature information acquisition unit 14, an operation region detection unit 15, an input detection unit 16, and an output. A unit 17, a video projection unit 18, and a video storage unit 19 are configured.

  The camera 11 is an imaging unit that captures an image of an object 20 that is a virtual input device. The camera 11 images the object 20 over a plurality of times. That is, the camera 11 images the object 20 as a moving image. The camera 11 is provided so that its imaging direction is the same as the user's line-of-sight direction, that is, the optical axis direction of the glasses-type display lens (or the direction of the normal vector of the display surface). Accordingly, the camera 11 captures an image of the object 20 when the user directs his / her line of sight toward the object 20 (when the user directs his / her line of sight toward the object 20, the imaging direction becomes the direction toward the object 20). The camera 11 may be provided integrally with the eyeglass-type display, or may be provided as a separate body around the camera 11 or carried by the user. The camera 11 outputs the captured image (data) to the object region recognition unit 12. Further, the camera 11 outputs the image to the area follower 13 in order to follow an object area described later.

  The object area recognition unit 12 is an object area recognition unit that recognizes an object area where the object 20 is located in the image captured by the camera 11. That is, the object region recognition unit 12 estimates the shape (two-dimensional shape) of the object 20 that is a virtual input device.

  For example, the object region recognition unit 12 recognizes the object region as follows. First, the marker 50 is provided in advance on the object 20 to be recognized. The marker 50 uses an indicia of a characteristic color or shape (for example, an asymmetric shape as shown in FIG. 1). The marker 50 is attached to the object 20 by the user 40 or the like. Further, the object region recognition unit 12 stores in advance information (for example, color and shape information) indicating the characteristics of the marker 50 so that the marker 50 can be recognized. The object region recognition unit 12 recognizes the position and movement of the marker 50 from the image input from the camera 11 based on the stored information indicating the feature of the marker 50. Subsequently, in the image, a set of locations having features (color, edge, optical flow) that move in synchronization with the marker 50 is estimated (recognized) as an object region.

  In addition, since the object 20 exists around the marker 50, the features of the peripheral portion of the marker 50 in the image (a set of colors, edges, and points linked to the movement of the marker 50) are extracted, and the features in the image are extracted. The object region may be estimated (recognized) from a portion (set of features) having The object region may be approximated by a rectangle including a portion having the above-described feature, or may be a plane having a shape connecting the outermost portions of the portion having the above-described feature. The object region determined at this time is set as an initial value, and the positional relationship with the marker 50 is stored.

  Alternatively, the object region may be detected by providing a sufficient number of markers 50 on the object 20 to identify the object region and detecting all the markers 50. In this case, the object area recognition unit 12 stores information indicating the positional relationship between the marker 50 provided on the object 20 and the object area, and detects the object area based on the information.

  Even if the marker 50 is not used, the object region recognition unit 12 can recognize the object region as follows. First, the object area recognition unit 12 stores in advance information indicating the characteristics of the object area (for example, color, shape, and texture information). The object region recognition unit 12 recognizes the object region from the image input from the camera 11 based on the stored information indicating the feature of the object region.

  The object region recognition unit 12 uses information indicating the recognized object region (for example, the position, shape, size, and the like in the image of the object region) as the region tracking unit 13, the feature information acquisition unit 14, the operation region detection unit 15, and the video projection. To the unit 18. Note that the object area recognized by the object area recognition unit 12 does not necessarily match the shape of the object 20, and may be, for example, a part of the object 20 used as a virtual input device.

  The region following unit 13 is a region following unit that recognizes (follows up) the image captured by the camera 11 by detecting a region corresponding to the object region recognized by the object region recognition unit 12. Imaging by the camera 11 is performed while an operation area is detected by an operation area detection unit 15 to be described later and a video is projected by the video projection unit 18, and the captured image changes with time. As described above, the recognized object region in the captured image can be changed by moving the object 20 or the camera 11 in the direction. The detection of the operation area by the operation area detection unit 15 and the projection of the video by the video projection unit 18 are performed according to the recognized object area, and the operation is appropriately performed when video is projected over a plurality of times. This is to enable detection of a region and projection of an image.

Specifically, the region tracking unit 13 acquires information indicating the characteristics of the object region from the image at the time when the object region recognition unit 12 recognizes the object region. For example, an image of the range of the object area estimated by the object area recognition unit 12 is extracted. The object region recognition unit 12 stores the image in the predetermined range as an image (template) indicating the feature of the recognized object region. Subsequently, the area tracking unit 13 detects a part corresponding to the stored image (template) of the predetermined range from the image to be tracked, and detects the part of the object 20 in the image to be tracked. And Specifically, this detection is performed using, for example, a pattern matching (template matching) method or the like. The area follower 13 outputs information indicating the followed shape to the operation area detector 15 and the video projector 18.

  The information indicating the feature of the shape to be compared with the image to be followed does not have to be the image itself. For example, the color information (histogram or average color) of the image in the extracted range may be used as information indicating the characteristics of the shape, and the region having the same color information of the image to be followed may be followed. Further, information indicating the feature of the edge (end) of the object region may be used as a template. Further, the marker 50 detected by the object region recognition unit 12 is followed by a method using an optical flow or the like, and the object region is estimated from the position of the followed marker 50 in the same manner as the object region recognition unit 12. The object area may be tracked.

  Alternatively, when the positional relationship between the marker 50 and the object region is stored as an initial value by the object region recognition unit 12 as described above, the region having the positional relationship with the marker 50 detected by detecting the marker 50 May be the object region to be followed.

  The feature information acquisition unit 14 is an object information acquisition unit that acquires information indicating the characteristics of the object 20. The information indicating the characteristics of the object 20 is, for example, information indicating the color of the object 20 (part corresponding to the object area) (for example, a histogram or an average of colors) or the object 20 (part corresponding to the object area). This corresponds to an image or the like. Information indicating the characteristics of the object 20 is used for detection of an operation region described later. Specifically, the feature information acquisition unit 14 extracts information indicating the above feature from the object region specified by the object region recognition unit 12 in the image captured by the camera 11. The extraction of the information indicating the characteristics is performed when the user 40 does not perform an input operation on the object 20 (the user 40 does not perform the input operation when extracting information).

  When the information acquired in the processing by the object region recognition unit 12 or the region tracking unit 13 includes information indicating the feature of the object 20, the feature information acquisition unit 14 uses the information of the object 20. You may use as information which shows a feature. In addition, when information indicating the feature of the object region is stored in the object region recognition unit 12 in advance, the information may be used as information indicating the feature of the object 20. However, when the information indicating the characteristics of the object 20 included in the information acquired in the processing by the object region recognition unit 12 or the region tracking unit 13 is not appropriate for use in detecting the operation region, etc. The information is not used. Or it is good also as acquiring the information which shows the characteristic of the object 20 by the input from the user 40 grade | etc.,. The feature information acquisition unit 14 outputs information indicating the acquired feature of the object 20 to the operation region detection unit 15.

  The operation region detection unit 15 is an operation region detection unit that detects an operation region having a feature different from the feature of the object region in the object region in the image captured by the camera 11. The object region is recognized by the object region recognizing unit 12 and followed by the region following unit 13. The information on the feature of the object region is input from the feature information acquisition unit 14 and stored in the operation region detection unit 15. The operation area is an area generated on the object area when the user 40 performs an input operation on the object 20. For example, when the user 40 performs an input operation with his / her finger, the region corresponds to the finger on the object 20.

  For example, the operation area detection unit 15 detects an operation area by extracting a part other than the color of the object area extracted by the feature information acquisition unit 14 from the object area. Alternatively, when the image (initial image) of the object region when the user 40 is not performing an input operation is used as the information on the feature of the object region, the initial image and the current image (to be detected in the operation region) ( A difference from the current frame is detected, and the location of the difference is detected as an operation area. When a plurality of operation regions are detected, the region having the largest (area) among the detected regions may be selected, or a region that is equal to or greater than a previously stored threshold value may be selected. Alternatively, an operation area may be detected from a plurality of images, and an area with the largest movement may be selected. The operation area is detected only by searching only the object area portion in the image. That is, it is not necessary to perform a search for a portion other than the object region in the image.

  The operation area detection unit 15 detects the operation area every time an image is captured by the camera 11. The operation region detection unit 15 outputs information indicating the detected operation region (for example, the position, shape, size, and the like of the operation region in the object region) to the input detection unit 16.

  The input detection unit 16 is an input detection unit that detects an input operation of the user 40 based on the operation region detected by the operation region detection unit 15. The input detection unit 16 is, for example, information indicating which part of the object 20 (the object area) the user 40 has performed an input operation of the user 40 (to which part of the object 20 the user 40 has touched). Detect as. The input detection unit 16 detects a predetermined point (for example, the uppermost point) in the operation area in the object area as a point operated by the user 40.

  Moreover, the input detection part 16 is good also as detecting the presence or absence of a user's input operation. In that case, for example, it is detected by determining whether or not the area of the operation region is equal to or greater than a threshold value stored in advance. This is to detect, for example, that the user 40 holds his hand over the object 20 because the operation area is larger than a certain size (in this case, the user 40's hand over the object 20). Hold it as an input operation in advance) Further, the input operation may be detected from the shape of the operation area.

  Further, the input operation may be detected from the time change of the operation area (for example, how the operation area moves). In this case, the detected time and the like are associated with the detected operation area so that a change in time can be detected. In addition, information (input operation pattern) indicating what time change corresponds to what input operation is stored in advance by the input detection unit 16, and the input operation is detected based on the information. . If an input operation is detected by such a pattern, even if information other than the pattern is detected, it is not detected as an input operation, and processing is performed only when a registered pattern is detected, thereby preventing a malfunction. Can do.

  Further, the input detection unit 16 not only detects which part of the object 20 (the object area) has been input as described above, but also, for example, a character “A” is input or a playback button is displayed. It is also possible to detect a more functional input operation of being pressed. In this case, the input detection unit 16 stores the correspondence between the information on the part where the input operation has been performed and the function information (information indicating what character is input or what button is pressed). In addition, the input operation is specified based on the correspondence. The input detection unit 16 outputs information indicating the input operation of the user 40 detected as described above to the output unit 17.

  The output unit 17 is an output unit that outputs information indicating the input operation of the user 40 detected by the input detection unit 16. The output is performed, for example, on a computer connected to the input device 10 as described above. In the present embodiment, the output unit 17 also outputs information indicating the input operation to the video projection unit 18.

  The video projection unit 18 is a projection unit that projects the video 30 in accordance with the object region recognized by the object region recognition unit 12 and the object region tracked by the region tracking unit 13. Specifically, the video projection unit 18 includes a display included in the input device 10. The video projection unit 18 acquires and projects the video to be projected from the video storage unit 19. As described above, the display of the video projection unit 18 is provided, for example, on a lens of a glasses-type display, and the projection of the video is performed so as to be superimposed on the object 20 as described above. The video projection unit 18 converts the video acquired from the video storage unit 19 to fit the object region based on the information input from the object region recognition unit 12 or the region tracking unit 13, and converts the converted video 30. Project. This is because the image is superimposed on the object 20 without a sense of incongruity. For example, the images to be projected stored in the video storage unit 19 are projected with the same size characters arranged in a rectangular range as shown in FIG. 3A, and the object region has a trapezoidal shape. If so, the conversion is performed so that the character on the short side (left side) side becomes smaller than the character on the long side (right side) side, as shown in FIG. In addition, the video projection unit 18 is based on information input from the object region recognition unit 12 or the region tracking unit 13 in a position where the object 20 is located in an image captured by the camera 11 (so that the user can visually recognize the image). ) Project the image 30.

上記の式において、Ｒ_１ｘ，Ｒ_２ｘ，Ｒ_３ｘ，Ｒ_１ｙ，Ｒ_２ｙ，Ｒ_３ｙ，Ｒ_１ｚ，Ｒ_２ｚ，Ｒ_３ｚは回転パラメータであり、ΔＸ，ΔＹ，ΔＺは平行移動パラメータである。変換行例Ｍは、物体２０の物体領域の座標（ｘ，ｙ，ｚ）に合わせて、投影すべき画像の座標（Ｘ，Ｙ，Ｚ）をそれぞれの軸に対して回転移動及び平行移動させるための行列である。ここで、物体領域及び投影すべき画像は二次元であるのでｚ＝Ｚ＝０である。映像投影部１８が、これらのパラメータを、物体領域及び映像蓄積部１９から取得した映像の形状とそれぞれの位置を示す情報とから算出して変換を行う。物体領域が回転している場合は、回転軸毎に回転パラメータＲ部分が、物体領域の回転角度θａから求めることができる。物体領域が平行移動している場合は、平行移動分ΔＸ，ΔＹ，ΔＺのうちその移動軸に沿ったパラメータが設定される。 This conversion and alignment is realized by the image projection unit 18 performing an existing image conversion process for enlarging, reducing, rotating and translating the image acquired from the image storage unit 19. For example, the image conversion process is performed by converting the video (data) acquired from the video storage unit 19 in the following conversion example M.

In the above formula, R _1x , R _2x , R _3x , R _1y , R _2y , R _3y , R _1z , R _2z , R _3z are rotational parameters, and ΔX, ΔY, ΔZ are parallel movement parameters. In the conversion row example M, the coordinates (X, Y, Z) of the image to be projected are rotated and translated with respect to the respective axes in accordance with the coordinates (x, y, z) of the object region of the object 20. Is a matrix for Here, since the object region and the image to be projected are two-dimensional, z = Z = 0. The video projection unit 18 calculates and converts these parameters from the shape of the video acquired from the object region and the video storage unit 19 and information indicating the respective positions. When the object region is rotating, the rotation parameter R portion can be obtained from the rotation angle θa of the object region for each rotation axis. When the object region is translated, parameters along the movement axis are set out of the translation amounts ΔX, ΔY, ΔZ.

  The video projection unit 18 performs the conversion process on the video as described above and projects it. By doing so, the superimposed image 30 is tilted in accordance with the tilt of the object 20 as shown in FIG.

  The image projected by the video projection unit 18 is, for example, an image of an input device. For example, as shown in FIG. 3, a character input by an input operation of the user 40 is projected. Alternatively, a keyboard image may be projected. The user 40 can perform an input operation on the object 20 (its object area) according to the projected image. Further, the image may be an image other than the image of the input device, for example, content that is viewed by the user 40.

  The projection of the video may be performed by a projector. When the glasses-type display is a video see-through display, the video projection unit 18 projects not only the video stored in the video storage unit 19 but also the surrounding video captured by the camera 11.

  Further, the video projection unit 18 may change the video to be projected based on information indicating the input operation of the user 40 input from the output unit 17. Specifically, for example, when the input operation specifies an image to be projected, the image corresponding to the indication is projected.

  The video storage unit 19 stores the video (data) 30 projected by the video projection unit 18, and outputs the video to the video projection unit 18 in accordance with a request from the video projection unit 18.

  As shown in FIG. 4, the input device 10 includes a central processing unit (CPU) 101, a random access memory (RAM) 102 and a read only memory (ROM) 103 that are main storage devices, an auxiliary storage device 104 such as a hard disk, and the like. It is comprised including the computer provided with the hardware. As other hardware, the input device 10 includes the camera 11 and the display 105 described above. The above-described functions of the input device 10 are exhibited when these components operate.

  Subsequently, a process (input method) executed by the input device 10 according to the present embodiment will be described using the flowchart of FIG. This process is executed when the user wears the input device 10 and performs an input operation using the input device 10.

  First, in the input device 10, the imaging of the object 20 that is a virtual input device by the camera 11 is started (S01, imaging step). At this time, when the user wearing the input device 10 faces the object 20, the imaging direction of the camera 11 becomes the direction of the object 20. This imaging is continuously performed during this processing. The captured image is output to the object region recognition unit 12 when recognizing the object region, and thereafter, is output to the region tracking unit 13 for tracking the object region. At the start of imaging, in order to recognize the object area and acquire information indicating the characteristics of the object 20, the object 20 to be imaged is not covered with an input operation such as a finger or a pen of the user 40. The

  Subsequently, the object region recognition unit 12 recognizes the object region from the captured image (S02, object region recognition step). Information indicating the recognized object region is output to the region follower 13, the feature information acquisition unit 14, the operation region detector 15, and the video projector 18. Here, the area following unit 13 acquires and stores information necessary for following the object area (for example, information indicating the positional relationship between the marker 50 and the object area, or information indicating the characteristics of the object area). The

  Subsequently, information indicating the feature of the object 20 is acquired by the feature information acquisition unit 14 (S03, feature information acquisition step). The information indicating the characteristics of the object 20 acquired here is for use in detecting the operation area. The acquisition is performed, for example, by extracting information from the object region portion of the image captured by the camera 11. Information indicating the acquired characteristics of the object 20 is output to the operation region detection unit 15.

  Subsequently, the region tracking unit 13 tracks the object region in the image captured by the camera 11 (after the image that is the object region recognition target by the object region recognition unit 12) (S04, region tracking step). ). Information indicating the tracked object region is output to the video projection unit 18.

  Subsequently, the projected image is acquired from the image storage unit 19 by the image projection unit 18. Subsequently, the video projection unit 18 calculates parameters for converting the video based on information indicating the object region input from the object region recognition unit 12 and the region tracking unit 13. Subsequently, the video projection unit 18 performs conversion processing of an image to be projected using the calculated parameters. Subsequently, the video projection unit 18 projects the converted video 30 so as to be superimposed on the object 20 as described above (S05, projection step).

  In the input device 10, while the above processing is performed, the operation region detection unit 15 detects an operation region in the object region in the image captured by the camera 11 (S 06, operation region detection step). Information indicating the detected operation area is output to the input detection unit 16. Subsequently, the input operation of the user 40 is detected by the input detection unit 16 based on the detected operation area (S07, input detection step). Information indicating the detected input operation is output to the output unit 17, and output is performed from the output unit 17 to a computer or the like that inputs the input operation (S08, output step).

  Information indicating the input operation is also input from the output unit 17 to the video projection unit 18. The video projection unit 18 rewrites the video to be projected according to the input information indicating the input operation (S09). Specifically, for example, when the information indicating the input operation that is input is for reproducing the content, the content is projected.

  Further, in the input device 10, while the imaging by the camera 11 or the projection of the video by the video projection unit 18 is being performed, the above processing of S 04 to S 09 is repeatedly performed. In S07, if the user 40 has not performed an input operation on the object 20 at that time, it is detected that no input operation has been performed. Further, the image rewriting process in S09 is performed according to the input operation of the user 40, and is not necessarily performed. The above is the processing that can be executed by the input device 10 according to the present embodiment.

  As described above, in the input device 10 according to the present embodiment, after the recognition of the object area of the object 20 that is a virtual input device, the operation area corresponding to the operation of the user 40 on the object area is performed. Is detected. Therefore, even if the imaging direction of the camera 11 moves or the background itself moves and the surrounding landscape is not fixed, the range in which processing is performed (attention) follows the object. Only area. Accordingly, a virtual input device can be reliably realized regardless of the deviation of the imaging direction of the camera 11 or the surrounding scenery.

  Further, in the input device 10 according to the present embodiment, if the feature of the object 20 that is a virtual input device is different from the feature of the operation area according to the user's input operation, the user's operation can be detected. it can. Specifically, for example, if the color of the object 20 is different from the color of the finger or pen of the user 40 for performing the operation, the input operation can be detected. For example, since input by a human hand is not determined by skin color, stable recognition can be performed regardless of the lighting environment or race. Therefore, even if an arbitrary input operation is performed, a virtual input device can be reliably realized. From the above, in the input device 10 according to the present embodiment, it is possible to enjoy a video or perform an input operation with the object 20 carried anytime and anywhere.

  Further, as in this embodiment, if information indicating the feature of the object for detecting the operation region is detected from the object region, the information indicating the feature of the object 20 that is a virtual input device is surely obtained. Therefore, the present invention can be appropriately implemented. However, as described above, the feature of the object may be acquired without using this method.

  Further, if the object region is tracked as in the present embodiment, the object region can be recognized properly even if the object 20 itself or the imaging direction of the camera 11 is moved, thereby appropriately performing an input operation. Can be detected. However, when the object 20 or the camera 11 is fixed or when an input operation is detected from a single image in a short time, the above-described configuration is not necessarily required.

  Further, if the image is projected as in the present embodiment, a display such as a virtual input device such as a keyboard can be performed, or a display corresponding to the detected user operation can be performed. Thereby, a user-friendly virtual input device can be realized. However, when it is not necessary to display something for the input operation of the user 40 (for example, when the input operation is to hold the hand over the object 20), the projection of the image is not necessarily required.

  If the object region is recognized by the marker 50, the input device 10 does not need to store information related to the object 20 in advance, so that the object region can be reliably recognized, and the present invention is appropriately implemented. can do. Further, when the object 20 is determined in advance, the object region can be surely recognized even if the features related to the object region are stored in advance and the object region is recognized based on the information. The present invention can be appropriately implemented.

  Note that the input device 10 described above includes a means for detecting sound (for example, a microphone), and may have a function corresponding to the detected sound. Specifically, for example, the operation region detection unit 15 may detect a predetermined sound with a microphone and detect the operation region at a timing when the sound is detected. Thereby, the input operation of the user 40 can be detected only when the user 40 comes into contact with the object 20 and makes a sound for an input operation such as a click operation.

  Specifically, the detection of the predetermined sound is performed by storing in advance information related to the sound of the object 20 being struck in response to the input operation, for example, a sound pattern (rhythm, sound, volume). This is done by determining whether the pattern and the sound input from the microphone match (there is a portion). That is, the sound to be detected is stored (registered) in advance, and the stored sound is detected as a sound generated from the object 20. This registration may be performed by the developer of the input device 10, or the user 40 may register a desired pattern by himself / herself. However, it is not always necessary to store sound information. For example, a sound may be detected when a certain amount of sound (a sound having a volume exceeding a preset threshold) is generated. . When the surroundings are noisy, noise can be removed by analyzing the noise in real time with a microphone and creating a noise model.

  With such a configuration, for example, the timing at which the user 40 detects the operation region can be arbitrarily determined, and the user 40 can perform an input operation at an arbitrary timing. Therefore, it is possible to cause the user 40 to perform an input operation with high operability. Further, the input operation can be detected only when the user 40 really wants to perform the input operation, and the input operation can be recognized with high accuracy.

  Further, the object area of the object 20 may be designated by the user 40 as follows. As shown in FIG. 6, this designation is performed, for example, by making a sound by hitting a corner (feature point) 21 of the shape of the object 20 with a finger. The object region of the designated object 20 is recognized by the object region recognition unit 12 of the input device 10.

  First, the object region recognition unit 12 detects that a specific sound generated from the object 20 is included in the sound collected by the microphone in the same manner as described above. Subsequently, the object region recognition unit 12 detects the position (feature point) where the specific sound is generated in the image captured by the camera 11 at the timing when the specific sound is detected. Specifically, the position is detected from an image input from the camera 11 at a timing when a specific sound is detected. The object region recognizing unit 12 stores in advance information related to what strikes the object 20, and detects the position of the object that strikes the object 20 (the position where the object 20 has been hit) based on the information from the above timing image. Then, the detected position is set as the position where the specific sound is generated. What strikes the object 20 is, for example, a finger of the user 40 or a tool such as a pen used for the user 40, which is designated in advance.

  Detection of the position in the image of an object 20 such as a user's finger is performed by image processing. Specifically, for example, the object region recognizing unit 12 stores in advance the skin color that is the color of the finger as information relating to what strikes the object 20, and the largest region (the number of pixels) of the skin color regions in the image. A large area) is detected, and the uppermost coordinate is detected. Alternatively, the unevenness of the skin color region may be detected, a specific shape (for example, an ellipse) stored in advance by the object region recognition unit 12 may be recognized, and a predetermined point of the shape may be detected. Alternatively, a hand model may be created and stored in the object region recognition unit 12 in advance, and the fingertip may be detected based on the information, and the fingertip point may be detected. The position detection is performed a plurality of times (for example, three times) on the object region of the (one) object 20. As the number of times increases, more detailed recognition of the object region of the object 20 becomes possible.

  Note that the position detection image may be specified based on the time at which a specific sound is detected by associating information about the time taken with the image taken by the camera 11, for example.

  The object area recognition unit 12 follows the detected point until the object area is specified. Imaging by the camera 11 is performed while a plurality of positions are being detected, and the captured image changes with time. If the object 20 is held by hand before the next point is detected after the first position is detected, the position of the detected point in the image may move. Further, the imaging direction of the camera 11 may move, and the position of the detected point in the image may move. The position detection (follow-up) is for appropriately estimating the shape when the position is specified using images over a plurality of times as described above.

  Specifically, the object region recognition unit 12 extracts an image in a predetermined range near the detected coordinates from the image used for position detection. The object region recognition unit 12 stores the image in the predetermined range as an image indicating the feature of the detected position. Subsequently, the object region recognition unit 12 detects a portion corresponding to the stored image in the predetermined range from the image to be tracked and specifies a position (two-dimensional coordinates on the image) tracked from the portion. To do. Specifically, this detection is performed using, for example, a feature point tracking method using an optical flow or the like.

  Subsequently, the object region recognition unit 12 estimates (recognizes) the object region of the object 20 from the detected position and the tracked position. The object region estimated here is a two-dimensional shape (a shape in an image captured by the camera 11). Further, the estimation of the object region here includes estimating the position of the object region (in the image captured by the camera 11). The object region recognition unit 12 estimates the object region when the detected (followed) positions reach a predetermined number (for example, three points). The number of positions used to estimate the object region is stored in advance in the object region recognition unit 12.

  Specifically, for example, the object region recognition unit 12 estimates the object region of the object 20 by connecting the detected positions. Further, the object area recognition unit 12 may be an object area that estimates a figure (for example, a quadrangle) inscribed in a figure that connects the detected positions and a figure that includes all the detected positions. Alternatively, the object area may be estimated by previously setting the size of the above-described projection range on the image and approximating the detected position within the range. The object region recognizing unit 12 stores in advance information (for example, information when a shape such as a rectangle is determined) or a rule for estimating the object region, and uses the information or the rule to determine the object region. Estimate.

  If the recognition of the object region as described above is used, the object region of the object 20 is estimated by imaging the object 20, detecting the sound, and detecting the position where the sound is generated. Therefore, the object region can be recognized easily and reliably, the object region can be recognized, and the present invention can be appropriately implemented.

  In the above-described method, the user 40 hits the object 20 to generate sound from the object 20. However, it is not always necessary to generate sound from the object 20. For example, in the recognition of the input sound of the user 40, when the surroundings are noisy and the sound of hitting the object 20 cannot be acquired, the sound may be detected as the user's voice. In that case, the point to which the user's finger is pointing is detected in the image captured by the camera 11 at the timing when the user's voice is detected. The detected point is a position corresponding to the object region of the object 20.

  Further, the above-described method and detection of the marker 50 may be used in combination. That is, as shown in FIG. 7, the object region is not determined from only the features around the marker 50, but after estimating a certain amount of the object region with the marker 50, the object region is hit with a finger within the estimated object region. If A is designated, the finger can be detected not from the entire image captured by the camera 11 but from an area narrowed to some extent, so that the user's finger can be detected efficiently. Further, an area other than the difference from the past frame or the feature of the object 20 can be used as the object area. In addition, it is possible to accurately specify an object region that the user 40 wants to use as a virtual input device, rather than estimating the object region using only the marker 50. Further, as shown in FIG. 7, a part of the object 20 can be set as an object region.

  In the present embodiment, the image obtained by being picked up by the camera 11 is a normal image (captured by visible light). Anything can be recognized. Specifically, for example, an image by infrared rays, a distance image, an image by thermography (temperature distribution) may be used.

It is a figure which shows typically the external appearance structure of the input device in embodiment of this invention. It is a figure which shows the function structure of the input device in embodiment of this invention. It is a figure which shows the image | video projected by an input device. It is a figure which shows the hardware constitutions of the input device in embodiment of this invention. It is a flowchart which shows the process (input method) performed with the input device in embodiment of this invention. It is a figure which shows a mode that an object is hit by the user when an object area | region is recognized, and a sound is generated. It is a figure which shows another example of a mode that an object is hit by the user and a sound is generated when recognizing an object area.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Input device, 11 ... Camera, 12 ... Object area recognition part, 13 ... Area tracking part, 14 ... Feature information acquisition part, 15 ... Operation area detection part, 16 ... Input detection part, 17 ... Output part, 18 ... Image | video Projection unit, 19 ... video storage unit, 101 ... CPU, 102 ... RAM, 103 ... ROM, 104 ... auxiliary storage device, 105 ... display, 20 ... object, 30 ... video, 40 ... user, 50 ... marker.

Claims (9)

Imaging means for imaging an object;
An object area recognition means for recognizing an object area where the object is located in the image from the image picked up by the image pickup means;
Feature information acquisition means for acquiring information indicating the characteristics of the object;
An operation region for detecting an operation region having a feature different from the feature related to the information acquired by the object feature acquisition unit in the object region recognized by the object region recognition unit in the image captured by the imaging unit. Detection means;
Input detection means for detecting a user's input operation based on the operation area detected by the operation area detection means;
Output means for outputting information indicating the input operation of the user detected by the input detection means;
An input system comprising:   The object information acquisition unit acquires information indicating the feature of the object by extracting the feature of the object from the object region recognized by the object region recognition unit in the image captured by the imaging unit. The input system according to claim 1. The imaging means images the object over a plurality of times,
3. The apparatus according to claim 1, further comprising a region following unit that recognizes a region corresponding to the region recognized by the object region recognition unit in the time-changed image captured by the imaging unit. Input system.   The input system according to any one of claims 1 to 3, further comprising a projecting unit that projects an image according to the object region recognized by the object region recognizing unit.   The input system according to claim 1, wherein the object area recognition unit recognizes the object area by detecting a marker provided in advance on the object.   The object region recognizing unit detects a predetermined sound, detects a position corresponding to the object region in an image captured by the imaging unit at a timing when the sound is detected, and detects the object region from the position. The input system according to any one of claims 1 to 5, wherein the input system is recognized.   The said object area | region recognition means memorize | stores beforehand the information which shows the characteristic which concerns on the said object area | region, and recognizes the said object area | region based on the said information, The Claim 1 characterized by the above-mentioned. The input system described.   The operation region detection unit detects a predetermined sound, and detects the operation region in an image captured by the imaging unit at a timing when the sound is detected. The input system according to claim 1. An imaging step for imaging an object;
An object region recognition step for recognizing an object region where the object in the image is located from the image captured in the imaging step;
A feature information obtaining step for obtaining information indicating the feature of the object;
An operation region for detecting an operation region having a feature different from the feature related to the information acquired in the object feature acquisition step in the object region recognized in the object region recognition step in the image captured in the imaging step. A detection step;
An input detection step of detecting a user's input operation based on the operation region detected in the operation region detection step;
An output step of outputting information indicating the user's input operation detected in the input detection step;
An input method comprising:

Images