JP5765133B2 - Input device, input control method, and input control program - Google Patents

Description

  The present invention relates to an input device that performs input using a virtual object superimposed on an image, an input control method, and an input control program.

  A keyboard is mainly used as a character input device in a computer. On most computers, the keyboard is standard equipment as an input device. The keyboard is a familiar device for many users.

  Therefore, conventionally, an input method for imitating a keyboard using a virtual keyboard has been proposed even when using a computer that is not equipped with a keyboard such as a tablet terminal. For example, the following techniques have been proposed.

  The virtual keyboard system displays a keyboard image on a display screen, and indicates a position of a key that can be currently input by superimposing a finger position acquired by a fingertip sensor on the keyboard image. The key input detection means determines whether an actual input has been performed.

  Also, other virtual keyboard systems can display the keyboard image on the display and display the shape of the hand on the keyboard on the screen using the result of the fingertip / motion detection unit. Indicates the key position.

  The key input processing device is a virtual keyboard in an environment using an optical see-through HMD (Head Mounted Display). The key input processing device determines whether or not the key of the keyboard image projected on the HMD overlaps the position of the fingertip of his / her hand seen through the HMD, and indicates the position of the key that can be currently input.

  The head-mounted display device is a virtual keyboard that uses an HMD, and projects a keyboard image from the HMD onto a surface such as a desk. The keyboard image and your fingertips can be seen with your own eyes directly from the “view section” of the HMD. The HMD is equipped with a camera, and by analyzing an image acquired thereby, it is determined which key is located on which key.

JP 2000-242394 A Japanese Patent No. 4099117 Japanese Patent No. 3214158 JP 2009-104429 A

  In a conventional virtual keyboard, for example, a key that can be input is shown by indicating the position of the fingertip when it is assumed that the viewpoint is directly above the keyboard. In another conventional virtual keyboard, the key at the intersection of the keyboard surface on the line of sight connecting the viewpoint and the position of the fingertip is determined as a key that can be input. The viewpoint is, for example, a shooting point by a camera.

  On the other hand, HMD is called “optical see-through type” that allows you to check actual scenery directly or through a half mirror, and “video see-through type” that shows a display that displays images from a camera taken from a position close to the line of sight. There are types. In order to provide an augmented reality environment, a virtual object is superimposed and displayed at an appropriate position on an object in real space. The augmented reality environment refers to a technology for additionally presenting information to the real environment using a computer, and an environment in which information is additionally presented.

  In the case of an “optical see-through type” HMD, it is necessary to adjust the display position of a virtual object for appropriate superimposed display because the line-of-sight analysis described in Patent Document 1 is necessary, and thus there are many technically difficult problems. . For this reason, the disclosed technique assumes the use of a “video see-through” HMD.

  The “video see-through type” HMD also has some problems. This problem is described in “virtual reality studies” (ISBN: 978-4769351382). For example, since the user sees the image through the camera, there are problems such as viewing angle, resolution, occlusion contradiction, system delay, depth of focus, etc., compared with the naked eye, and the user often feels uncomfortable. .

  For example, there is a technique for removing the display area of the main display device from the display area of the HMD device with respect to the video see-through viewing angle problem (Japanese Patent Laid-Open No. 2006-267604).

  Here, in the augmented reality environment, a case is considered in which a selection object (for example, a virtual keyboard, a selection button, etc.) is displayed as a virtual object superimposed on an image through a camera, and the virtual object is pressed with a user's finger or the like. In this case, the virtual object cannot be pressed down well due to the problem of the discomfort of the appearance regarding the viewing angle and the perception of the depth regarding the depth of focus.

  Therefore, the disclosed technology has been made in view of the above problems, and an input device, an input control method, and an input device that can appropriately press a virtual object intended by a user against a virtual object superimposed and displayed on an image. An object is to provide an input control program.

An input device according to an aspect of the disclosure includes a real object detection unit that acquires an image, a position determination unit that determines a display position of a virtual object to be superimposed on the image, a viewpoint detection unit that detects a viewpoint position of a user, and a determination A position used for determination of pressing of the virtual object based on the superposition processing unit that superimposes and displays the virtual object on the image at the displayed position and the detected position of the viewpoint and the position of the virtual object superimposed and displayed. A pressing determination unit that changes , the real object detection unit detects a real object included in the acquired image, and the pressing determination unit calculates a distance between the real object and the virtual object displayed in a superimposed manner. If the distance is less than or equal to a predetermined value, the position used for the determination of pressing is changed .

  According to the disclosed technology, a virtual object intended by a user can be appropriately pressed against a virtual object superimposed and displayed on an image.

The figure which shows the case where a virtual numeric keypad is input from right above. The figure which shows the case where a virtual keyboard is input from right above. The figure which shows the case where a virtual numeric keypad is input from the diagonal direction. The figure which shows the case (the 1) when inputting a virtual keyboard from the diagonal direction. The figure which shows the case (the 2) when inputting a virtual keyboard from the diagonal direction. The figure which shows the case (the 2) when inputting a virtual keyboard from the diagonal direction. The figure which shows the position distribution at the time of the incorrect input with respect to a correct answer position. The figure for demonstrating the incorrect input with respect to a virtual keyboard. 1 is a diagram illustrating an example of a system in Embodiment 1. FIG. 1 is a block diagram showing an example of a system configuration in Embodiment 1. FIG. The block diagram which shows an example of the function of an input device. The block diagram which shows an example of the function of a virtual object position determination part. The figure which shows an example of a change of a press determination area | region. 5 is a flowchart illustrating an example of input control processing according to the first embodiment. FIG. 9 is a block diagram illustrating an example of functions of an input device according to a second embodiment. The figure which shows the example (the 1) which changes the display position and pressing determination area | region of a virtual key. The figure which shows the example (the 2) which changes the display position and pressing determination area | region of a virtual key. The figure which shows the example (the 1) which changes the press determination area | region of a virtual key. The figure which shows the example (the 2) which changes the press determination area | region of a virtual key. 9 is a flowchart illustrating an example of input control processing according to the second embodiment. The figure which shows the relationship between a reference plane and a threshold value. The flowchart which shows an example of the press determination process by a press determination part. FIG. 10 is a block diagram illustrating an example of functions of an input device according to a third embodiment. 10 is a flowchart illustrating an example of input control processing (part 1) in the third embodiment. 10 is a flowchart illustrating an example of input control processing (part 2) in the third embodiment. The flowchart which shows an example of the process which detects an incorrect input and a correct input. The figure for demonstrating the specific example which detects an incorrect input and a correct input.

  In the embodiment described below, it is assumed that a virtual keyboard as a virtual object is displayed in a three-dimensional space and operated by the user's own fingertip. Since the virtual keyboard has a three-dimensional shape, it is recognized as an input by pressing it correctly from the top of the key.

  Since the viewpoint of the user moves, there is a problem that when the key is viewed through the video see-through HMD, the sense of distance to the key is not correctly recognized. A sense of distance from the actual object (also referred to as an actual object) due to the difference between the camera's imaging range and the visual field of the eye, the depth of focus problem, and the time delay until it is displayed on the screen. It may be difficult to figure out.

  For this reason, the virtual keyboard cannot be operated well due to the difference between the information obtained from the visual sense and the own sense. Specifically, there arises a problem that a key to be input cannot be operated or another key around the target key is operated.

  In response to this problem, the inventors conducted the following experiment. This experiment is an experiment in which 12 subjects use a virtual keyboard to input an instructed key. In this experiment, input is performed while changing the viewpoint and the direction of the keys, and the occurrence rate of erroneous input and the direction of misalignment between the erroneously input key and the designated key are investigated. In this experiment, a marker is attached to the fingertip of the subject, and the subject presses a key while viewing the image (video) through the camera on the display screen.

  The experiment was performed when the input was performed from the viewpoint from right above the key and when the input was performed from the viewpoint from the diagonal of the key.

(View from directly above)
FIG. 1 is a diagram illustrating a case where a virtual numeric keypad is input from directly above. In the example shown in FIG. 1, a case where the numeric keypad 11 of “5” is pressed with a finger with a marker 12 is shown.

  FIG. 2 is a diagram illustrating a case where the virtual keyboard is input from directly above. In the example illustrated in FIG. 2, a case where the “G” key 21 is pressed with a finger with the marker 12 is illustrated. The virtual keyboard is, for example, a QWERTY keyboard.

(An oblique perspective)
FIG. 3 is a diagram illustrating a case where the virtual numeric keypad is input from an oblique direction. In the example shown in FIG. 3, a case where the “6” numeric keypad 13 is pressed with a finger with a marker 12 is shown.

  FIG. 4 is a diagram illustrating a case where the virtual keyboard is input from an oblique direction (part 1). In the example illustrated in FIG. 4, a case where the “C” key 22 is pressed with a finger with the marker 12 is illustrated.

  FIG. 5 is a diagram illustrating a case where the virtual keyboard is input from an oblique direction (part 2). The example shown in FIG. 5 differs from the example shown in FIG. 4 in the angle of the virtual keyboard. In the example shown in FIG. 5, a case where the “X” key 23 is pressed with the finger with the marker 12 is shown.

  FIG. 6 is a diagram illustrating a case where the virtual keyboard is input from an oblique direction (part 2). The example shown in FIG. 6 differs from the example shown in FIGS. 4 and 5 in the angle of the virtual keyboard. In the example shown in FIG. 6, a case where the “F” key 24 is pressed with the finger with the marker 12 is shown. 1 to 6 are examples in which the virtual key is displayed superimposed on the hand when the user presses the virtual key. Moreover, as shown in FIGS. 4-6, about the input from the diagonal direction, the test subject had input from several angles.

(Experimental result)
The experimental results for the input from the viewpoint from directly above are as follows.
Correct input rate: 90.6% Incorrect input rate: 9.4%
The experimental results for input from an oblique perspective are as follows.
Correct input rate: 63.5% Incorrect input rate: 36.5%
From the above experimental results, it can be seen that since the key can be seen in a plane from the viewpoint from directly above, the positional relationship between the fingertip position and the drawn key can be appropriately grasped.

  On the other hand, from an oblique viewpoint, there are many erroneous inputs because the positional relationship between the fingertip position in three dimensions and the drawn key cannot be properly grasped. It was also found that the correct key is often pressed by mistake, regardless of the viewpoint or the key display direction.

  FIG. 7 is a diagram illustrating a position distribution at the time of erroneous input with respect to the correct position. As shown in FIG. 7, when there is an erroneous input, there is a tendency to press the near side in the line-of-sight direction with respect to the correct position.

  FIG. 8 is a diagram for explaining erroneous input to the virtual keyboard. As shown in FIG. 8, when the target key 71 is pressed from a perspective from an oblique direction, there are many cases where an input in front of the line of sight is erroneously input. If this is expressed in a plane, the target key 71 should be pressed, but the key 72 will be pressed.

  In practice, the user sees the virtual key from an oblique direction. Considering this, when the virtual key is superimposed on the image over the camera, the user's operational feeling is improved by superimposing the virtual key from the oblique direction. However, if a virtual key from an oblique direction is superimposed on an image as in the experiment described above, an erroneous input occurs.

  Hereinafter, each embodiment for preventing erroneous input will be described with reference to the accompanying drawings based on the above-described experiment.

[Example 1]
<System>
FIG. 9 is a diagram illustrating an example of a system according to the first embodiment. In the example shown in FIG. 9, the user wears an HMD (Head Mounted Display) with a camera and inputs characters using a virtual keyboard superimposed on the HMD. When the user presses the virtual key, a character corresponding to the virtual key is input, and the HMD wirelessly communicates with the computer to transmit the character to the computer application.

  The application inputs the received character and transmits the display screen information of the application to the HMD. The HMD displays the display screen information of the received application on the HMD, for example.

  In the system shown in FIG. 9, an image on which a virtual keyboard is superimposed may be displayed on a display or a projector instead of being displayed on the HMD. In addition, the character input determination process or the like may be performed not by the HMD but by another machine such as a computer or a smartphone. Further, viewpoint conversion may be performed on an image captured by a camera in the vicinity of the HMD to make the viewpoint from the user, and a virtual keyboard may be superimposed on the image after viewpoint conversion and displayed on the HMD.

  The disclosed system displays a selected object, for example, a virtual keyboard in a three-dimensional space in an augmented reality environment, and is intuitively operated with a user's own finger as if a real keyboard existed there. .

<Configuration>
FIG. 10 is a block diagram illustrating an example of a system configuration according to the first embodiment. In the example illustrated in FIG. 10, the system includes an input device 101, an image input device 110, a viewpoint / line-of-sight detection device 111, a display device 112, and an application 113. Each device may be configured by combining a plurality of devices.

  The image input device 110 is, for example, a camera, is installed at a position close to the user's viewpoint, and captures and acquires an image in the same direction as the line of sight. The acquired image is output to the input device 101. Note that the image input device 110 does not necessarily have to be installed at a position close to the user's viewpoint. If the image input apparatus 110 is at a position different from the user's viewpoint, viewpoint conversion may be performed on the image.

  The viewpoint / line-of-sight detection device 111 includes an imaging unit and the like, and detects the position of the user's viewpoint and the viewing direction (line-of-sight direction). The viewpoint / line-of-sight detection device 111 detects the line of sight by detecting the boundary between the cornea and the sclera (sclera tracker method), detecting the reflected light on the cornea surface (corneal reflection method), There are a method of measuring a potential difference (EOG method), a method of wearing a contact lens with a built-in search coil (search coil method), and the like. For details, see "From eye-gaze interface to eye-gaze communication", Takehiko Ohno, Information Processing Society of Japan Research Report, HI, Human Interface Research Group Report, P.171-178, 2001, and "Man-Machine System Engineering. Analysis (1)-Eye movement-"(http://hydro.energy.kyoto-u.ac.jp/Lab/staff/shimoda/lecture2005/mms/ocular1.pdf).

  In addition, there is a technique for detecting the line of sight by combining the pupil center method and the pupil-corneal reflection method (see Japanese Patent Laid-Open No. 2001-61785). In recent years, there is a technique for detecting a line of sight using a web camera image, and the line of sight is specified using a reflected image of an eyeball using visible light, near infrared light, or the like. Specifically, the gaze direction is estimated from the positions of the pupil and the iris.

  In addition, there is a technology that can detect a line of sight from a distant position (Life Tech Support's line of sight input intention transmission device) and a technology that can detect a line of sight while moving (TalkEye II of Takei Kikai Kogyo Co., Ltd.).

  When the image input device 110 is installed at a position close to the user's viewpoint, the viewpoint / line-of-sight detection device 111 regards the position of the viewpoint as being equal to the position of the image input device 110 (for example, a camera). Further, the viewpoint / line-of-sight detection device 111 can estimate the position of the eyeball from the position of the head when using a sensor that can know the position of the head. For example, there is a head tracking system such as “Track IR” of Mikimoto Co., Ltd.

  The viewpoint / line-of-sight detection apparatus 111 detects the user's viewpoint and line of sight using any known method, and outputs the detected viewpoint to the input apparatus 101.

  The display device 112 displays an image captured by the image input device 110 or an image in which a virtual object is superimposed on the image by the input device 101. Hereinafter, the display device 112 will be described using an HMD as an example, but it may be a normal display or projector. A normal display means a display such as a computer monitor or a television.

  The application 113 is an application executed by a computer, for example, an application such as Word (registered trademark) Word or Excel. The application 113 receives an input command (for example, characters or numbers) from the input device 101. Further, the application 113 may output screen information of a screen displaying the acquired characters and numbers to the input device 101.

  The input device 101 is a device that provides an augmented reality environment, for example. The input device 101 includes a control unit 102, a main storage unit 103, an auxiliary storage unit 104, and a communication unit 105. Each unit is connected via a bus so that data can be transmitted / received to / from each other.

  The control unit 102 is a CPU (Central Processing Unit) that performs control of each device, calculation of data, and processing in a computer. The control unit 102 is an arithmetic device that executes a program stored in the main storage unit 103 or the auxiliary storage unit 104. The control unit 102 receives data from the input unit or the storage unit, calculates, processes the output unit, the storage unit, and the like. To the output.

  The main storage unit 103 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like, and a storage device that stores or temporarily stores programs and data such as an OS and application software that are basic software executed by the control unit 102 It is.

  The auxiliary storage unit 104 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The communication unit 105 communicates with the application 113 in a wired or wireless manner. For example, input commands and screen information are communicated.

<Function>
FIG. 11 is a block diagram illustrating an example of functions of the input device 101. In the example illustrated in FIG. 11, the input device 101 includes a real object detection unit 201, a virtual object position determination unit 202, a viewpoint detection unit 203, a superimposition processing unit 204, a press determination unit 205, and an input command generation unit 206. Each unit of the input device 101 can be realized by the main storage unit 103 as a control unit 102 and a work memory, for example. The input command generation unit 206 can be realized by the communication unit 105, for example, when transmitting an input command.

  The real object detection unit 201 analyzes the image acquired from the image input device 110 and grasps the position and distance of the object. The object is, for example, a hand, a finger, or a desk. This object is called a real object in comparison with a virtual object. In addition, when a virtual object is superimposed, the real object detection unit 201 detects the surface, distance, and the like of the real object in the vicinity of the virtual object region.

  For example, when detecting a real object such as a hand or a finger, the real object detection unit 201 can detect a finger by performing skin color detection or edge extraction. Moreover, it can also detect by attaching a marker etc. to a finger.

  The real object detection unit 201 calculates the relative position between the marker and the image input device 110 from the image (camera image) using, for example, the surface on which the virtual keyboard is to be displayed and the marker attached to the finger.

  For estimation of camera position using markers, see “Development of augmented reality system construction tool ARTToolkit” (by Hirokazu Kato, IEICE technical report, PRMU, pattern recognition, media understanding 101 (652), 79 -86, 2002-02) and "3D characters are born in the real world! Introduction to ARToolkit Augmented Reality Space Programming" (ISBN 978-4048673617).

  Briefly, the real object detection unit 201 assumes that the size of the marker is known, and determines the surface on which the marker exists and the position / posture of the marker from the imaging parameters of the image input device 110 and the shape of the marker being imaged. presume.

The coordinate values of n feature points in the marker coordinate system are (X _i , Y _i , Z _i ) (i = 0, 1,..., N−1), and conversion from the marker coordinate system to the camera coordinate system Let T _{cm be} the matrix and C be the perspective transformation matrix of the camera. At this time, the coordinates (x _i , y _i ) (i = 0, 1,..., N−1) on the image are obtained by Expression (1).

h: Parameter for converting a point on the image from the camera coordinate system R: Rotation element T: Translation element If the real object detection unit 201 knows C, T _cm is calculated by using n feature points. Can be sought. Since the origin of the camera coordinate system is the viewpoint position, the real object detection unit 201 can grasp the positional relationship between the marker and the viewpoint.

Consider a case where two markers are included in an image. Real object detecting unit 201, the point of the camera coordinate system C, the point of the coordinate system of the marker 1 as M _1, a transformation matrix from M ₁ to C to T _1. Similarly, the real object detection unit 201 sets a point in the coordinate system of the marker ₂ as M ₂ and a conversion matrix from M ₂ to C as T ₂ .

In this case, the transformation matrix from the camera coordinate system C to the coordinate system _{M 1} to the marker 1 is opposite matrix _T ^{1 -1} of _{T 1.} Through the camera coordinate system, mutual conversion between the coordinate system of the marker 1 and the coordinate system of the marker 2 can be performed. The real object detection unit 201 converts the point on the coordinate system M ₂ of the marker 2 to the coordinate system M ₁ of the marker ₁ using Expression (2).

The real object detection unit 201 knows the position of the marker 2 in the marker 1 coordinate system. Thereby, the real object detection part 201 estimates the position of a finger | toe using a marker.

  In addition to the above example, the real object detection unit 201 attaches a sensor to a surface on which a virtual object is to be displayed or a fingertip to obtain the distance between the virtual object and the finger, or uses a distance image indicating the depth of the image. The finger position may be obtained by estimating the finger position. The real object detection unit 201 outputs information on the detected real object to the virtual object position determination unit 202, the superimposition processing unit 204, and the press determination unit 205.

  The virtual object position determination unit 202 determines the display position of the virtual object to be superimposed and displayed on the image acquired from the image input device 110. Details of the virtual object position determination unit 202 will be described with reference to FIG.

  FIG. 12 is a block diagram illustrating an example of the function of the virtual object position determination unit 202. The virtual object position determination unit 202 illustrated in FIG. 12 includes a shape determination unit 301 and a position / posture determination unit 302.

  The shape determining unit 301 handles information related to a virtual object to be superimposed and displayed. The shape determination unit 301 also creates 3D shape data of a virtual object to be displayed, reads existing 3D shape data, and the like. The three-dimensional shape data is data indicating the shape of the virtual keyboard, for example.

  The three-dimensional shape data includes a definition of a pressing determination area (a hit determination area). The pressing determination area is an area used for determining whether to press the virtual key. For example, the pressing determination area is a predetermined value above the displayed virtual key and indicates an area corresponding to the size of the virtual key. The reason why the pressing determination area is set to a predetermined value is to facilitate the determination of pressing. Each piece of solid shape data may be a separate file.

  The position / posture determination unit 302 determines the position and orientation for displaying the virtual object determined by the shape determination unit 301 based on the detection result of the real object detection unit 201. The position / posture determination unit 302 performs, for example, recognition of the display position of the virtual object, estimation of the position of the image input device 110, and geometric conversion of the virtual object.

  The position / posture determination unit 302 converts the coordinates of the known shape of the virtual object in accordance with the appearance of the image acquired from the image input device 110. For example, the position / posture determination unit 302 performs coordinate conversion so that, for example, the virtual object has a shape viewed from an oblique direction.

  Specifically, the position / posture determination unit 302 performs coordinate conversion such as rotation, parallel movement, and enlargement / reduction processing to move the virtual object to a predetermined position in the three-dimensional space. This coordinate transformation is a transformation of coordinates in a so-called world coordinate system.

  The world coordinate system is a coordinate system representing the entire space, which is mainly used in the field of three-dimensional graphics. The world coordinate system is a coordinate system for indicating the position of an object in space, and is used for handling display and movement of the object. The world coordinate system is also called a global coordinate system.

  The position / posture determination unit 302 outputs the coordinates of the virtual object determined by performing the coordinate conversion to the superimposition processing unit 204.

  Returning to FIG. 11, the viewpoint detection unit 203, based on the user viewpoint and line-of-sight data acquired from the viewpoint / line-of-sight detection device 111, and the image acquired from the image input device 110, the position of the user viewpoint in the world coordinate system, An attention position in the image (the position of the image corresponding to the line-of-sight direction) is detected. The viewpoint detection unit 203 outputs the detected viewpoint position and line-of-sight direction to the superimposition processing unit 204.

  The superimposition processing unit 204 superimposes and displays the virtual object on the image at the display position determined by the virtual object position determination unit 202. The superimposition processing unit 204 performs perspective transformation on the coordinates of the virtual object acquired from the virtual object position determination unit 202 in accordance with the viewpoint position of the user detected by the viewpoint detection unit 203 and the image acquired from the image input device 110 ( Projective transformation). The coordinate transformation and projective transformation described above are called affine transformation.

  The superimposition processing unit 204 superimposes and displays the virtual object at the coordinate position of the virtual object that has undergone perspective transformation. At this time, the superimposition processing unit 204 performs drawing so that no contradiction arises in the context of the actual object in the image.

  For example, the superimposition processing unit 204 draws a virtual object above the surface of the real object (desk) by a predetermined value. This is to prevent the virtual object from being displayed at a position sinking from the real object or a position that is too floating.

  The press determination unit 205 determines whether the real object (for example, a finger) detected by the real object detection unit 201 and the virtual object superimposed and displayed by the superimposition processing unit 204 intersect in space or within a predetermined distance. Determine if there is any.

  At this time, the press determination unit 205 changes the position used for the virtual object press determination based on the viewpoint position acquired from the viewpoint detection unit 203 and the position of the virtual object displayed in a superimposed manner. The position used for the pressing determination includes the position of the pressing determination region and the coordinate position of the real object when determining whether to press the virtual object.

  The pressing determination unit 205 may change the position of the pressing determination area corresponding to the virtual object based on the position of the virtual object and the viewpoint position. For example, the pressing determination unit 205 shifts the pressing determination area corresponding to the virtual object by a predetermined amount in the viewpoint direction. The viewpoint direction refers to a direction from the position of the virtual object toward the viewpoint position indicating the position of the user's eyes.

  This predetermined amount may be determined based on the distance between the position of the virtual object and the viewpoint position. For example, the predetermined amount may be determined as several percent of the distance between the position of the virtual object and the viewpoint position. Thereby, the position of the press determination area can be shifted using an appropriate value according to the position of the virtual key.

  The press determination unit 205 calculates the distance between the position of the real object and the reference plane, and determines whether or not the virtual object is pressed based on this distance. The reference plane is a surface of a desk that is a real object, a surface of a virtual object that is superimposed, or the like. Here, the reference plane is a desk surface. Note that the surface of the desk is represented by the real object detection unit 201 as one point at a predetermined position of the desk and a normal vector. When the calculated distance is less than the threshold value, the pressing determination unit 205 determines that the button is pressed. The press determination unit 205 can determine contact between a finger and a virtual key, a virtual key and a wall, for example.

  Further, the press determination unit 205 specifies a press determination area including the position of the real object when it is determined to be pressed, and determines that the virtual object corresponding to the press determination area is the pressed object. Since the pressing determination unit 205 can perform the pressing determination in the pressing determination area shifted in the viewpoint direction, it is possible to reduce the user's pressing mistake. This utilizes the tendency to press a key in front of the line of sight. The line-of-sight direction refers to a direction in which an object is viewed from the eyes.

  If the pressing determination area 205 is not changed with respect to the virtual object, the pressing determination unit 205 changes the coordinate position of the real object at the time of pressing determination by a predetermined amount in the direction opposite to the viewpoint direction. Thus, by making use of the tendency to press the key in front of the line-of-sight direction, by changing the coordinates at the time of determination of pressing the real object to the point in the viewpoint direction, it is possible to reduce the user's mistakes in pressing. This predetermined amount may be determined based on the distance between the position of the virtual object and the viewpoint position, as in the pressing determination area.

  FIG. 13 is a diagram illustrating an example of changing the pressing determination area. The white key shown in FIG. 13 indicates the display position 401 of the virtual key, and the gray key indicates the position 402 of the pressing determination area corresponding to the virtual key. The position 402 of the press determination area is shifted in the viewpoint direction with respect to the display position 401 of the virtual key. This is a result of the pressing determination unit 205 changing the pressing determination area in the viewpoint direction in consideration of an erroneous pressing of the virtual key.

  In the example shown in FIG. 13, the press determination area is shown to be below the virtual key, but the press determination area is positioned above the virtual key by a predetermined value, for example. In addition, the pressing determination unit 205 obtains a direction (referred to as a first direction) obtained by projecting the viewpoint direction onto the virtual key plane instead of the direction of the viewpoint direction when changing to the viewpoint direction, and sets a predetermined amount in the first direction. It may be shifted.

  As a result, when the user presses the virtual object while viewing the image on which the virtual object is superimposed, it is possible to reduce a mistake in pressing the virtual object and press the virtual object intended by the user.

  The press determination unit 402 outputs the coordinates of the virtual object determined to be pressed to the input command generation unit 206.

  Returning to FIG. 11, the input command generation unit 206 generates an input command based on the coordinates of the virtual object acquired from the pressing determination unit 205. When the virtual object is a virtual keyboard, the input command generation unit 206 specifies which key code is based on the coordinates of the virtual object, and generates the key code.

  The input command generation unit 206 transmits the generated input command to the application 113 or the like.

<Operation>
Next, the operation of the input device 101 in the first embodiment will be described. FIG. 14 is a flowchart illustrating an example of the input control process according to the first embodiment.

  In step S <b> 101 illustrated in FIG. 14, the input device 101 acquires an image from the image input device 110.

  In step S102, the real object detection unit 201 detects a real object from the acquired image. The real object is, for example, a desk surface or a finger.

  In step S <b> 103, the virtual object position determination unit 202 determines a display position for displaying the virtual object on a desk surface or the like detected by the real object detection unit 201. At this time, the virtual object position determination unit 202 performs the coordinate conversion described above.

  In step S <b> 104, the superimposition processing unit 204 performs superimposition display on the image acquired from the image input apparatus 110 by performing the above-described perspective transformation on the virtual object.

  In step S105, the pressing determination unit 205 changes the pressing determination area corresponding to the virtual object to the viewpoint direction.

  In step S <b> 106, the pressing determination unit 205 determines whether the detected real object has pressed the virtual object displayed in a superimposed manner. Thereby, a user's pushing mistake can be decreased.

  If it is determined that the button has been pressed (step S106—YES), the process proceeds to step S107. If it is determined that the button has not been pressed (step S106—NO), the process returns to step S101.

  In step S107, the input command generation unit 206 generates an input command based on the coordinates of the virtual object determined to be pressed. The generated input command is transmitted to the application 113, for example.

  Note that the pressing determination unit 205 may omit step S105 and shift the coordinates of the real object at the time of pressing in the reverse direction from the viewpoint direction in step S106.

  As described above, according to the first embodiment, the virtual object intended by the user can be appropriately pressed against the virtual object superimposed and displayed on the image.

[Example 2]
Next, an input device according to the second embodiment will be described. In the second embodiment, the input device refers to the error tendency data indicating the data of the correct key and the incorrect key when the user makes a mistake in pressing the virtual key, and the incorrect key held in the error tendency data is pressed. When trying to do so, the position used for the press determination is changed. In the embodiment described below, a virtual keyboard is taken as an example for a virtual object, and a finger is taken as an example for a real object.

<System and configuration>
Since the configuration of the system and the input device in the second embodiment is the same as that in the first embodiment, description thereof is omitted.

<Configuration>
FIG. 15 is a block diagram illustrating an example of functions of the input device 500 according to the second embodiment. In the configuration illustrated in FIG. 15, the same components as those illustrated in FIG. 11 are denoted by the same reference numerals, and description thereof is omitted.

A storage unit 501 illustrated in FIG. 15 stores error tendency data. The storage unit 501 can be realized by the auxiliary storage unit 104, for example. The error tendency data holds the following data.
(1) Key code, viewpoint position, key coordinates, and finger coordinates for correct input (pressing) (2) Key code, viewpoint position, and key coordinates for incorrect input (pressing) corresponding to the correct input key Finger Coordinates The press determination unit 502 sets three thresholds for the distance between the finger and the press determination position. The threshold 1 is a threshold for determining that the finger has approached the virtual key. The threshold 2 is a threshold for determining that the virtual key has been pressed. The threshold 3 is a threshold for determining that the finger has been released after the virtual key is pressed.

  The press determination unit 502 determines whether or not the distance between the finger and the reference surface is less than the threshold value 1. The reference plane is, for example, a desk surface or a surface on which a virtual keyboard is displayed. When this distance is less than the threshold value 1, the pressing determination unit 502 indicates that the virtual key to be pressed (hereinafter referred to as the purpose key) is error tendency data stored in the storage unit 501, and is used as an incorrect key. It is determined whether or not it is stored.

  If the target key is stored as an incorrect key, the press determination unit 502 changes the press determination area corresponding to the target key because the key is easily mistaken. This change process is the same as in the first embodiment. At this time, the pressing determination unit 502 may notify the superimposition processing unit 503 of the coordinates of the target key and request to change the display position of the target key.

  If the target key is not stored as an incorrect key, the press determination unit 502 does not change the position of the target key press determination area. This is because it is a key that is difficult to make mistakes.

  When the distance is less than the threshold value 2, the press determination unit 502 notifies the input command generation unit 206 of the coordinates of the pressed virtual key. When the distance becomes less than the threshold value 2 and becomes larger than the threshold value 3 after the distance becomes less than the threshold value 2, the pressing determination unit 502 notifies the input command generation unit 206 that the pressing of the virtual key is finished. The determination of the threshold 2 and the threshold 3 is also performed in the first embodiment.

  When the superimposition processing unit 503 receives a change request from the pressing determination unit 502, the superimposition processing unit 503 changes the display position of the target key displayed in a superimposed manner. The superimposition processing unit 503 can specify the target key by the coordinates notified by the pressing determination unit 502.

  When the pressing determination area is changed, the superimposition processing unit 503 may change the display position of the target key according to the pressing determination area.

  Thus, when the user tries to press the virtual key, if the virtual key is easy to make a mistake, the position of the pressing determination area can be changed in the viewpoint direction.

<Example of change>
Next, an example of position change in the second embodiment will be described. FIG. 16 is a diagram illustrating an example (part 1) of changing the virtual key display position and the pressing determination area. In the example illustrated in FIG. 16, the user looks at the virtual keyboard from an oblique direction through the display device 111 and attempts to press the virtual key 31. In practice, the virtual key is often pressed in front of the line-of-sight direction. Further, it is assumed that the virtual key 31 is stored in the error tendency data as an erroneous input key.

  At this time, the pressing determination unit 502 shifts the pressing determination area of the virtual key 31 by a predetermined value in the viewpoint direction. This predetermined value may be a predetermined value or may be determined based on the distance between the virtual key 31 and the viewpoint position. The pressing determination unit 502 also changes the pressing determination area corresponding to the virtual keys 32 and 33 in the viewpoint direction from the virtual key 31 to the viewpoint direction. This is to prevent duplication due to movement of the virtual key 31.

  For example, the superimposition processing unit 503 changes the virtual key 31 from the original display position 41 to the display position 42. The change is made after a change request is notified from the press determination unit 502. In addition, the superimposition processing unit 503 does not change only the virtual key 31 but also changes the display position of the virtual keys 32 and 33 as the virtual key 31 is changed. This is because by changing the display position of the virtual key 31, it is better to change the virtual key in the viewing direction as viewed from the virtual key 31 so that it does not overlap with the virtual key 31.

  FIG. 17 is a diagram illustrating an example (part 2) of changing the display position of the virtual key and the pressing determination area. In the example illustrated in FIG. 17, the user looks at the virtual keyboard displayed obliquely from the oblique direction through the display device 111 and presses the virtual key 31. Further, it is assumed that the virtual key 31 is stored in the error tendency data as an erroneous input key.

  At this time, the pressing determination unit 502 shifts the pressing determination area of the virtual key 31 by a predetermined value in the viewpoint direction. This predetermined value may be a predetermined value or may be determined based on the distance between the virtual key 31 and the viewpoint position. The pressing determination unit 502 also changes the pressing determination area corresponding to the virtual keys 34 and 35 in the viewpoint direction from the virtual key 31 to the viewpoint direction.

  For example, the superimposition processing unit 503 changes the virtual key 31 from the original display position 41 to the display position 43. The change is made after a change request is notified from the press determination unit 502. Further, the superimposition processing unit 503 does not change only the virtual key 31 but also changes the display position of the virtual keys 33 and 34 in accordance with the change of the virtual key 31. This is because by changing the display position of the virtual key 31, it is better to change the virtual key in the viewing direction as viewed from the virtual key 31 so that it does not overlap with the virtual key 31.

  FIG. 18 is a diagram illustrating an example (part 1) of changing the virtual key press determination area. The example shown in FIG. 18 is an example in which only the pressing determination area in the example shown in FIG. 16 is changed. In the example shown in FIG. 18, the pressing determination area corresponding to the virtual key 31 is changed to the position 51 in the viewpoint direction. The display position of the virtual key 31 is the position 41. With the change in the position 51 of the press determination area, the positions of the press determination areas corresponding to the virtual keys 32 and 33 are changed to positions 52 and 53 in the viewpoint direction, respectively.

  FIG. 19 is a diagram illustrating an example (part 2) of changing the virtual key press determination area. The example shown in FIG. 19 is an example in which only the pressing determination area in the example shown in FIG. 17 is changed. In the example shown in FIG. 19, the position of the corresponding press determination area corresponding to the virtual key 31 is changed to the position 54 in the viewpoint direction. The display position of the virtual key 31 is the position 41. With the change in the position 51 of the press determination area, the positions of the press determination areas corresponding to the virtual keys 32 and 33 are changed to the positions 55 and 56 in the viewpoint direction, respectively.

  Therefore, the press determination unit 502 changes the position of the press determination area corresponding to another virtual key in the viewpoint direction from the target key in accordance with the change of the position of the press determination area corresponding to the target key. Thus, by changing the position of the press determination area corresponding to the target key, it is possible to prevent overlap with the press determination areas corresponding to other virtual keys.

<Operation>
Next, the operation of the input device 500 in the second embodiment will be described. FIG. 20 is a flowchart illustrating an example of input control processing according to the second embodiment. In step S201 shown in FIG. 20, the virtual object position determination unit 202 reads solid shape data, and the pressing determination unit 502 reads error tendency data.

  In step S <b> 202, the input device 500 acquires an image from the image input device 110.

  In step S203, the real object detection unit 201 analyzes the acquired image.

  In step S204, the real object detection unit 201 determines whether or not the display position of the virtual keyboard is in the image. For example, the real object detection unit 201 determines that there is a display position if a surface of a predetermined size is in the image. The virtual keyboard may be determined in advance to be displayed in a predetermined area in the image.

  If there is a virtual keyboard display position (step S204-YES), the process proceeds to step S205, and if there is no virtual keyboard display position (step S204-NO), the process proceeds to step S212. If the real object detection unit 201 determines that there is a display position, it sets k_flag to 1, and if it determines that there is no display position, it sets k_flag to 0.

  In step S205, the real object detection unit 201 analyzes the acquired image and detects a hand or a finger. The detection of the hand or finger may be performed by detecting skin color, detecting a finger with a marker, or detecting with a sensor attached to the finger.

  In step S206, the real object detection unit 201 determines whether a hand or a finger is included in the image. If a hand or a finger is included in the image (step S206—YES), the process proceeds to step S207. If a hand or a finger is not included in the image (step S206—NO), the process proceeds to step S212. If the real object detection unit 201 determines that the hand or finger is included in the image, the real object detection unit 201 sets h_flag to 1, and if it determines that the hand or finger is not included in the image, sets the h_flag to 0.

  In step S207, the viewpoint detection unit 203 detects a viewpoint position in the space and a line-of-sight direction based on the viewpoint and line-of-sight data acquired from the viewpoint / line-of-sight detection device 111.

  In step S208, the superimposition processing unit 503 determines whether k_flag is 1 and h_flag is 1. If this condition is satisfied, it is determined that the virtual keyboard is superimposed and the process proceeds to step S209. If this condition is not satisfied, it is determined that the virtual keyboard is not superimposed and the process proceeds to step S212.

  In step S209, the pressing determination unit 502 calculates the distance between the virtual keyboard (virtual object) and the hand or finger.

  In step S210, the press determination unit 502 determines whether it is necessary to change the display of the virtual key based on the calculated distance. When the change is necessary (step S210-YES), the process proceeds to step S211. When the change is not necessary (step S210-NO), the process proceeds to step S212.

  The necessity for change means that the calculated distance is less than the threshold value 1 and that a virtual key (target key) that is less than the threshold value 1 is held as an erroneous key of error tendency data. This condition indicates that a finger or hand approaches a virtual key (object key) and that the object key is frequently pressed incorrectly.

In step S211, the press determination unit 502 shifts the position of the target key press determination area in the viewpoint direction. Further, the superimposition processing unit 503 changes the display position of the target key to be superimposed in accordance with the press determination area by a display change command from the press determination unit 502. (See FIGS. 16 and 17)
In step S <b> 212, the superimposition processing unit 503 displays the image acquired from the image input device 110.

  In step S213, the superimposition processing unit 503 determines whether k_flag is 1. If k_flag is 1 (step S213-YES), the process proceeds to step S214, and if k_flag is 0 (step S213-NO), the process returns to step S202.

  In step S214, the superimposition processing unit 503 superimposes and displays the virtual keyboard that has undergone coordinate transformation and perspective transformation on the image. At this time, if the hand or finger is closer to the virtual key than the threshold 1, the virtual key is displayed in a superimposed state with the position of the display position of the virtual key changed.

  In step S215, the pressing determination unit 502 determines whether or not the calculated distance is less than the threshold value 2. If the distance is less than the threshold 2 (step S215-YES), it is determined that the virtual key has been pressed, and the process proceeds to step S216. If the distance is greater than or equal to the threshold 2 (step S215-NO), the virtual key has not been pressed. The determination is made, and the process returns to step S202.

  In step S216, the input command generation unit 206 generates a key code corresponding to the virtual key determined to be pressed.

  Although the display position of the virtual key is changed in step S211, this change is not necessarily a necessary process. The input device 500 may change only the position of the virtual key press determination area (see FIGS. 18 and 19).

  FIG. 21 is a diagram illustrating the relationship between the reference plane and the threshold value. In the example shown in FIG. 21, the threshold value 1 (TH1) indicates the distance at which it is determined that the finger has approached the virtual key. The threshold 2 (TH2) indicates a distance at which it can be determined that the virtual key is pressed. A threshold value 3 indicates a distance for determining that the virtual key has been released after the virtual key is pressed. Here, the distance between the reference plane and the finger is d. The reference plane is, for example, a real object plane or a virtual keyboard plane.

  FIG. 22 is a flowchart illustrating an example of a press determination process performed by the press determination unit 502. In step S301 illustrated in FIG. 22, the pressing determination unit 502 determines whether the distance d is less than TH1. The pressing determination unit 502 sets k_near to 1 if the distance d is less than TH1, and sets k_near to 0 if the distance d is equal to or greater than TH1.

  In step S302, the pressing determination unit 502 determines whether the distance d is less than TH2. The pressing determination unit 502 sets k_down to 1 if the distance d is less than TH2, and does nothing if the distance d is greater than TH2.

  In step S303, the press determination unit 502 determines whether the distance d is greater than TH3 and k_down is 1. The press determination unit 502 sets k_down to 0 if this condition is satisfied, and does nothing if this condition is not satisfied. When the process of step S303 ends, the pressing determination unit 502 performs the next image process.

  Thereby, when the k_near is 1, the press determination unit 502 changes the position of the press determination region, and requests the superimposition processing unit 503 to change the display position. When the k_down is 1, the press determination unit 502 determines that the press has been performed, and notifies the input command generation unit 206 of information on the pressed virtual key.

  Note that the press determination unit 502 determines continuous input if k_down does not become 0 within a predetermined time, and performs the next press determination when k_down becomes 0 within a predetermined time.

  As described above, according to the second embodiment, when a real object (for example, a finger) approaches a virtual key, it is possible to minimize the pressing determination area for changing the position. Further, according to the second embodiment, the display position of the virtual key can be changed in accordance with the change of the position of the pressing determination area. In addition, according to the second embodiment, since error tendency data is referred to, the position of the pressing determination area or the display position of the virtual key can be not changed for a virtual key in which an erroneous input does not occur.

[Example 3]
Next, an input device according to the third embodiment will be described. In the third embodiment, the input device updates the error tendency data used in the second embodiment and indicating the data of the correct key and the wrong key when the user makes a mistake in pressing the virtual key. In the embodiment described below, a virtual keyboard is taken as an example for a virtual object, and a finger is taken as an example for a real object.

<System and configuration>
Since the configuration of the system and the input device in the third embodiment is the same as that in the first embodiment, description thereof is omitted.

<Configuration>
FIG. 23 is a block diagram illustrating an example of functions of the input device 600 according to the third embodiment. In the configuration shown in FIG. 23, the same components as those shown in FIGS. 11 and 15 are denoted by the same reference numerals, and the description thereof is omitted. The input command generation unit 601 outputs the generated key code to the learning unit 602.

  The learning unit 602 detects an erroneous input by using the coordinates of the finger and the virtual key when the virtual key is input and information obtained by deleting the character. The learning unit 602 learns error tendency data for each virtual key using an erroneously input key, a re-input key, and the coordinates and viewpoint position when those keys are pressed.

  For example, the learning unit 602 can detect that an erroneous input has occurred when the key code of the BackSpace key is acquired. The learning unit 602 can determine that the next input key code is a correct input.

  Note that the learning unit 602 may assume various cases in which the input after re-inputting is incorrect or continuously deleted. In the third embodiment, a case where it is possible to detect whether the input after deletion is wrong will be described later.

  The learning unit 602 stores the key code, viewpoint position, virtual key coordinates, and finger coordinates in the case of an erroneous input in the buffer as error data.

  In addition, the learning unit 602 stores the key code, viewpoint position, virtual key coordinates, and finger coordinates for correct input in the buffer as correct data. The learning unit 602 associates the error data with the correct data and updates the error tendency data stored in the storage unit 603. The key code is acquired from, for example, the input command generation unit 601, and the viewpoint position, virtual key coordinates, and finger coordinates are acquired from, for example, the press determination unit 502.

  The storage unit 603 stores initial error tendency data in advance. The initial error tendency data may be set by a prior experiment or the like. Moreover, since the error tendency data is preferably changed for each user, the error tendency data may be initialized if there is no update for a certain period of time.

  Thereby, error tendency data can be learned according to the user, and the position of the press determination area and the display position of the virtual key can be appropriately changed for the user.

<Operation>
Next, the operation of the input device 600 according to the third embodiment will be described. First, the input control method according to the third embodiment will be described with reference to FIGS. FIG. 24 is a flowchart illustrating an example of input control processing (part 1) in the third embodiment.

  In step S401 illustrated in FIG. 24, the virtual object position determination unit 202 reads solid shape data, and the pressing determination unit 502 reads initial error tendency data.

  Steps S402 to S416 are the same as steps S202 to S216 shown in FIG.

  In step S417, the learning unit 602 stores the current key code, the finger coordinates at the time of input, the virtual key coordinates, and the viewpoint position in a buffer. Hereinafter, the key code, finger coordinates at the time of input, virtual key coordinates, and viewpoint position data are collectively referred to as input data.

  In step S418, the learning unit 602 determines whether the current input is an erroneous input. If it is incorrect input (step S418-YES), it will progress to S419, and if it is not incorrect input (step S418-NO), it will progress to step S420. Whether it is an erroneous input can be determined, for example, based on whether the key code of the BackSpace key.

  In step S419, the learning unit 602 holds the previous input data stored as error data. After step S419, the process returns to step S402.

  In step S420, the learning unit 602 determines whether correct input after erroneous input has been performed. If correct input is performed (step S420—YES), the process proceeds to step S421, and if normal input is performed (step S420—NO), the process returns to step S402.

  In step S421, the learning unit 602 adds error data and correct input data in association with each other to the error tendency data.

  In step S422, the learning unit 602 updates the error tendency data based on the added data.

  As a result, the error tendency data is updated in accordance with the user's erroneous input habit, so that the pressing determination area position or the virtual key display position changing process can be performed in accordance with the user's erroneous input habit. it can.

  FIG. 26 is a flowchart illustrating an example of a process for detecting an erroneous input and a correct input. The example shown in FIG. 26 shows a process that can be determined as an error even when an input after an erroneous input is wrong.

  In step S501, the learning unit 602 initializes parameters d_flag and n_flag to 0. d_flag is a parameter for finding an erroneous input corresponding to a correct input. n_flag is a parameter representing the number of consecutive keys until deleted.

  In step S502, the learning unit 602 inputs a key from the input command generation unit 601. In step S503, the learning unit 602 stores input data in a buffer.

  In step S504, the learning unit 602 determines whether the key code represents a deletion key (for example, a BackSpace key). If it is a delete key (step S504-YES), it will progress to step S505, and if it is not a delete key (step S504-NO), it will progress to step S509.

  In step S505, the learning unit 602 detects that the previous input is incorrect.

  In step S506, the learning unit 602 determines whether d_flag is 0 or not. If d_flag is 0 (step S506-YES), the process proceeds to step S507, and if d_flag is not 0 (step S506-NO), the process proceeds to step S508.

  In step S507, the learning unit 602 sets d_flag to 1 and n_flag to 0. After step S507, the process returns to step S502.

  In step S508, the learning unit 602 adds 1 to d_flag and sets n_flag to 0. After step S508, the process returns to step S502.

  In step S509, the learning unit 602 determines whether d_flag is not 0 and n_flag is greater than 1. When this condition is satisfied (step S509—YES), the process proceeds to step S511, and when this condition is not satisfied (step S509—NO), the process proceeds to step S510.

  In step S509, it is determined whether a key other than the delete key has been followed twice after the delete key. Thereby, after inputting the character key after the delete key, when the delete key is further pressed, it is possible to determine that the character key between the delete keys is an erroneous input.

  In step S510, the learning unit 602 adds 1 to n_flag. After the process of step S510, the process proceeds to step S502.

  In step S511, the learning unit 602 determines that the previous input is a correct answer (correct input).

  In step S512, the learning unit 602 determines whether d_flag is greater than zero. If d_flag is greater than 0 (step S512—YES), the process proceeds to step S513. If d_flag is 0 (step S512—NO), the process returns to step S501.

  In step S513, the learning unit 602 determines that 2 × d_flag + 1 is an erroneous input corresponding to the current correct answer.

  In step S514, the learning unit 602 updates error tendency data with input data at the time of correct answer and error. In step S515, the learning unit 602 subtracts 1 from d_flag. After the process of step S515, the process returns to step S512.

<Specific example of detecting erroneous input and correct input>
A specific example based on the processing of FIG. 26 will be described. FIG. 27 is a diagram for explaining a specific example of detecting an erroneous input and a correct input. In the example illustrated in FIG. 27, it is assumed that characters are input in the order of a, b, and d, and then deletion is input.

  The learning unit 602 determines that d is an error when deletion is input. Next, e is input and deletion is input. The learning unit 602 determines that e is an error when deletion is input.

  Next, it is assumed that characters are input twice in the order of c and d. The learning unit 602 determines that c immediately before d is a correct answer (correct input). The learning unit 602 determines that the error corresponding to c is e and d. Therefore, the learning unit 602 associates the input data of e and d as the error data of the input data corresponding to c, and updates the error tendency data.

  The algorithm shown in FIG. 26 is merely an example, and the algorithm may be appropriately changed in consideration of the case where deletion is continuously input.

  As described above, according to the third embodiment, the error tendency data is updated in accordance with the user's erroneous input habit. Change processing can be performed.

[Modification]
Next, a modified example of each embodiment will be described. In the above embodiment, the virtual keyboard is described as an example of the virtual object. However, the virtual object may be a selection object. For example, a selection button or a balloon may be used. This speech balloon is a speech balloon issued from a real object such as a word or an object of a text.

  When this balloon is selected, the application 113 checks the detailed information of the word or object that is the balloon. The superimposition processing unit can also superimpose and display this detailed information on the image.

  In addition, real objects are not limited to fingers and hands. For example, an indicator bar that can specify a three-dimensional position may be used.

  In addition, by recording a program for realizing the input control processing described in each of the above-described embodiments on a recording medium, the input control processing in each of the embodiments can be performed by a computer. For example, it is possible to record the program on a recording medium, and cause the computer or portable device to read the recording medium on which the program is recorded, thereby realizing the above-described input control processing.

  The recording medium is a recording medium for recording information optically, electrically or magnetically, such as a CD-ROM, flexible disk, magneto-optical disk, etc., and information is electrically recorded such as ROM, flash memory, etc. Various types of recording media such as a semiconductor memory can be used.

  Although the embodiments have been described in detail above, the invention is not limited to the specific embodiments, and various modifications and changes can be made within the scope described in the claims. It is also possible to combine all or a plurality of the constituent elements of the above-described embodiments.

In addition, the following additional notes are disclosed regarding each of the above embodiments.
(Appendix 1)
A real object detection unit for acquiring an image;
A position determining unit that determines a display position of a virtual object to be superimposed on the image;
A viewpoint detection unit for detecting a user's viewpoint position;
A superimposition processing unit that superimposes and displays the virtual object on the image at the determined display position;
A pressing determination unit that changes a position used for pressing determination of the virtual object based on the detected viewpoint position and the position of the virtual object superimposed and displayed;
An input device comprising:
(Appendix 2)
The real object detection unit is
Detecting a real object contained in the image;
The pressing determination unit
The input device according to appendix 1, wherein a distance between the real object and the virtual object superimposed and displayed is calculated, and when the distance is equal to or less than a predetermined value, a position changing process used for the pressing determination is performed.
(Appendix 3)
The pressing determination unit
The input device according to claim 1 or 2, wherein a position of the virtual object press determination area indicating an area used for the virtual object press determination is changed by a predetermined amount from the virtual object in the viewpoint position direction.
(Appendix 4)
The pressing determination unit
The input device according to claim 1 or 2, wherein when determining to press the virtual object, the position of the coordinates of the real object is changed by a predetermined amount from the virtual object in a direction opposite to the viewpoint position.
(Appendix 5)
The input device according to appendix 3 or 4, wherein the predetermined amount is determined based on a distance between the viewpoint position and the virtual object.
(Appendix 6)
The superimposition processing unit
The input device according to supplementary note 3, wherein when the position of the pressing determination area is changed, the display position of the virtual object corresponding to the pressing determination area is changed according to the position of the pressing determination area.
(Appendix 7)
The virtual object is a virtual keyboard;
First data including the coordinates of the virtual key when the input is correct and the coordinates of the real object when the virtual key is pressed, the coordinates of the virtual key when the input is incorrect, and when the virtual key is pressed A storage unit that stores third data associated with second data including the coordinates of the real object in
The pressing determination unit
Additional notes 2 to 6 for performing a process of changing the position used for the pressing determination when a virtual key whose distance from the real object is within the predetermined value is a virtual key stored in the storage unit. The input device according to item.
(Appendix 8)
Detecting an erroneous input to the virtual key of the virtual keyboard, the coordinates of the virtual key when erroneously input, the coordinates of the real object when the virtual key is pressed, and the coordinates of the virtual key when re-input, The input device according to appendix 7, further comprising: a learning unit that updates the third data using the coordinates of the real object when the virtual key is pressed.
(Appendix 9)
Get an image,
Determining the display position of the virtual object to be superimposed on the image;
Detect the user's viewpoint position,
Displaying the virtual object superimposed on the image at the determined display position;
An input control method in which a computer executes a process of changing a position used for determination of pressing of the virtual object based on the detected viewpoint position and the position of the virtual object superimposed and displayed.
(Appendix 10)
Get an image,
Determining the display position of the virtual object to be superimposed on the image;
Detect the user's viewpoint position,
Displaying the virtual object superimposed on the image at the determined display position;
An input control program for causing a computer to execute a process of changing a position used for determination of pressing of the virtual object based on the detected viewpoint position and the position of the virtual object superimposed and displayed.

101, 500, 600 Input device 102 Control unit 103 Main storage unit 104 Auxiliary storage unit 105 Communication unit 110 Image input device 111 Viewpoint / line-of-sight detection device 112 Display device 112 Application 201 Real object detection unit 202 Virtual object position determination unit 203 Viewpoint detection Units 204 and 503 Superimposition processing units 205 and 502 Press determination unit 206 and 601 Input command generation unit 301 Shape determination unit 302 Position and orientation determination units 501 and 603 Storage unit 602 Learning unit

Claims (8)

A real object detection unit for acquiring an image;
A position determining unit that determines a display position of a virtual object to be superimposed on the image;
A viewpoint detection unit for detecting a user's viewpoint position;
A superimposition processing unit that superimposes and displays the virtual object on the image at the determined display position;
A pressing determination unit that changes a position used for pressing determination of the virtual object based on the detected viewpoint position and the position of the virtual object superimposed and displayed;
Equipped with a,
The real object detection unit is
Detect real objects contained in the acquired image,
The pressing determination unit
An input device that calculates a distance between the real object and the virtual object displayed in a superimposed manner, and performs a process of changing a position used for the pressing determination when the distance is equal to or less than a predetermined value . The pressing determination unit
Wherein the position of pressing determining region of the virtual object that indicates the area used for pressing determining the virtual object, an input apparatus according to claim 1, wherein changing a predetermined amount in the viewpoint position direction from the virtual object. The pressing determination unit
Wherein the position of coordinates of the actual object when the judgment of pressing against the virtual object, an input apparatus according to claim 1, wherein changing a predetermined amount in a direction opposite to the said viewpoint position from the virtual object. The superimposition processing unit
The input device according to claim 2 , wherein when the position of the press determination area is changed, the display position of the virtual object corresponding to the press determination area is changed according to the position of the press determination area. The virtual object is a virtual keyboard;
First data including the coordinates of the virtual key when the input is correct and the coordinates of the real object when the virtual key is pressed, the coordinates of the virtual key when the input is incorrect, and when the virtual key is pressed A storage unit that stores third data associated with second data including the coordinates of the real object in
The pressing determination unit
Virtual key distance from the actual object is within the predetermined value, when a virtual key that is stored in the storage unit, according to claim 1 to 4 or performs a process of changing the location to be used for the pressing determining The input device according to one item. Detecting an erroneous input to the virtual key of the virtual keyboard, the coordinates of the virtual key when erroneously input, the coordinates of the real object when the virtual key is pressed, and the coordinates of the virtual key when re-input, The input device according to claim 5 , further comprising: a learning unit that updates the third data using the coordinates of the real object when the virtual key is pressed. Get an image,
Detect real objects contained in the acquired image,
Determining the display position of the virtual object to be superimposed on the image;
Detect the user's viewpoint position,
Displaying the virtual object superimposed on the image at the determined display position;
A distance between the real object and the virtual object displayed in a superimposed manner is calculated, and when the distance is equal to or less than a predetermined value, based on the detected viewpoint position and the position of the virtual object displayed in a superimposed manner, An input control method in which a computer executes a process of changing a position used for determining whether or not a virtual object is pressed. Get an image,
Detect real objects contained in the acquired image,
Determining the display position of the virtual object to be superimposed on the image;
Detect the user's viewpoint position,
Displaying the virtual object superimposed on the image at the determined display position;
A distance between the real object and the virtual object displayed in a superimposed manner is calculated, and when the distance is equal to or less than a predetermined value, based on the detected viewpoint position and the position of the virtual object displayed in a superimposed manner, An input control program for causing a computer to execute a process of changing a position used for determining whether or not a virtual object is pressed.