JP2020071588A - Display device and method for controlling display device - Google Patents

Abstract

To improve a user's operability.SOLUTION: An HMD 100 comprises: a transmission type image display unit 20 that is attached to the head of a user and transmits light; a myoelectric sensor 200 that detects the motion of the hand of the user; an operation estimation unit 143 that estimates an operation to select one image MA for input from a plurality of images MN for input based on a detection value from the myoelectric sensor 200; a determination unit 144 that determines a result of estimation made by the operation estimation unit 143; and a learning unit 145 that machine-learns a condition for specifying the one image MA for input from the detection value from the myoelectric sensor 200 based on a learning data set 147a including the detection value from the myoelectric sensor 200, the one image MA for input, and a result of determination made by the determination unit 144.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a display device and a display device control method.

Conventionally, various interface devices that receive a user's operation are known (for example, refer to Patent Document 1).
For example, the interface device described in Patent Document 1 is attached to the operator's hand. The interface device includes a member that comes into contact with at least a part of one palm of an operator, a camera, and a detection unit. The camera is located on the side of the member opposite the palm-facing side. The detection means detects the motion of one hand based on the image captured by this camera.

JP2012-73830A

  An object of the present invention is to provide a display device capable of improving operability for users.

  One mode for achieving the above object is a transmissive display unit that transmits light, and is overlapped with a display unit mounted on a user's head and the user's hand visually recognized through the display unit. At the position, a display control unit for displaying a plurality of input images on the display unit, a motion sensor for detecting a motion of the user's hand, and a plurality of input images among the plurality of input images based on a detection value of the motion sensor. From the operation estimation unit that estimates an operation of selecting one input image from the determination unit, the determination unit that determines the estimation result of the operation estimation unit, the detection value of the motion sensor, the one input image, and the determination A learning unit that machine-learns a condition for specifying the one input image from the detection value of the motion sensor based on a learning data set including a combination with a determination result of the unit.

  The display device may include an imaging unit, and the determination unit may be configured to determine whether the estimation result of the operation estimation unit is correct based on an image captured by the imaging unit.

  In the display device, a voice input unit that receives a voice input from the user is provided, and the determination unit determines whether the estimation result of the operation estimation unit is correct based on the voice input to the voice input unit. It may be configured.

  In the display device, each of the plurality of input images may indicate a different character, and the operation estimation unit may be configured to estimate a character to be input based on a detection value of the motion sensor. ..

  In the display device, each of the plurality of input images is associated with a region separated by a joint in each finger of the user other than the thumb of the hand, and the operation of selecting the one input image is performed as described above. The operation may be such that the thumb of the user's hand is pressed against a region divided by the joint.

  In the above display device, two or more predetermined number of input images included in the plurality of input images are associated with a region separated by a joint in each finger of the user other than the thumb, The operation of selecting is an operation of pressing the thumb of the user's hand to a region delimited by the joint, and selecting the one input image from the predetermined number of input images associated with the region. The operation to be performed may be an operation of moving the thumb of the user's hand in a predetermined direction.

  In the above display device, the display control unit may be configured to display a first guide image for guiding the position of the user's hand on the display unit.

  In the display device, the display control unit tracks the position of the user's hand visually recognized through the display unit, and according to the movement of the user's hand, each of the plurality of input images is displayed. The display position may be adjusted.

  In the display device, the motion sensor may include a myoelectric sensor attached to the wrist of the user.

  Another aspect of achieving the above object is a transmissive display unit that transmits light, and includes a display unit mounted on a user's head and a motion sensor that detects a motion of the user's hand. A method for controlling a display device, comprising: an operation estimation step of estimating an operation of selecting one input image from the plurality of input images from a detection value of the motion sensor; and an estimation result of the operation estimation step. Based on a learning data set including a combination of a determination step of determining the motion sensor, a detection value of the motion sensor, the one input image, and a determination result of the determination step. And a learning step of machine-learning a condition for specifying one input image, the control method of the display device.

Explanatory drawing which shows the external appearance structure of HMD. The figure which shows the structure of the optical system of an image display part. FIG. 3 is a perspective view showing a main configuration of an image display unit. The schematic diagram which shows the relationship between the position of an object and an imaging range. The figure which shows the structure of each part which comprises HMD. The figure which shows the structure of a control part. The figure which shows a guide image. The figure which shows the image for input. The flowchart which shows the process of a control part. The flowchart which shows the machine learning process of a control part. The flowchart which shows the process of the control part after a learning process. The figure which shows the other form of the image for input.

Hereinafter, the present embodiment will be described with reference to the drawings.
FIG. 1 is an explanatory diagram showing an external configuration of the HMD 100.
As shown in FIG. 1, an HMD (Head Mounted Display) 100 is a so-called see-through type () HMD, and includes a control device 10 and an image display unit 20. The HMD 100 corresponds to an example of a “display device”. The see-through type can also be referred to as a “transmissive type”.
The control device 10 includes a flat box-shaped case 10A. The case 10A includes various switches that receive user's operations, the track pad 14, and the like, and the user operates these to cause the control device 10 to function as a control device that controls the HMD 100. Further, the case 10A has a built-in functional unit that controls the HMD 100. The case 10A corresponds to an example of a housing.

Further, the control device 10 of the HMD 100 is connected to the myoelectric sensor 200 in a wirelessly communicable manner. The myoelectric sensor 200 is worn on the user's wrist.
The myoelectric sensor 200 detects an electric signal generated according to the movement of the muscle of the user's finger. In the present embodiment, the myoelectric sensor 200 mainly detects an electric signal generated according to the movement of the muscles of the thumb of the user.
The myoelectric sensor 200 is attached to the user's wrist as described later with reference to FIG. 7. The myoelectric sensor 200 is formed in a cylindrical shape, and a plurality of sensors are arranged on the inner surface side thereof. The “inner surface side” refers to the side in contact with the outer peripheral surface of the user's wrist. The myoelectric sensor 200 is configured to be able to detect movements of a plurality of muscles by the plurality of sensors.
The myoelectric sensor 200 includes a battery, a communication unit, a control unit, and an input / output unit. The communication unit can communicate with the control device 10 of the HMD 100 by wireless communication such as Bluetooth (registered trademark).
The myoelectric sensor 200 corresponds to an example of a “motion sensor”. The "motion sensor" may be a gyro sensor attached to the user's finger. Further, the “motion sensor” may include the myoelectric sensor 200 and the gyro sensor.

The image display unit 20 is a wearing body worn on the head of the user, and has a spectacle shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27. .. The image display unit 20 corresponds to an example of a “display unit”.
The right holding unit 21 and the left holding unit 23 extend rearward from both ends of the front frame 27, and hold the image display unit 20 on the user's head like temples of glasses. Here, of both ends of the front frame 27, an end located on the right side of the user in the mounted state of the image display unit 20 is an end ER, and an end located on the left side of the user is an end EL. The right holding unit 21 is provided so as to extend from the end ER of the front frame 27 to a position corresponding to the right side of the user's head in the mounted state of the image display unit 20. The left holding part 23 is provided so as to extend from the end part EL to a position corresponding to the left side head of the user in the mounted state of the image display part 20.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is located in front of the right eye of the user when the image display unit 20 is attached, and allows the right eye to visually recognize the image. The left light guide plate 28 is located in front of the left eye of the user when the image display unit 20 is attached, and allows the left eye to visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other. Corresponding to. The front frame 27 may be provided with a nose pad that comes into contact with the user's nose when the image display unit 20 is attached, at the connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad, the right holding unit 21, and the left holding unit 23. Further, a belt (not shown) that contacts the user's occipital region when the image display unit 20 is attached may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 is connected to the user's head by the belt. Can be held at

  The right display unit 22 is a unit related to the display of an image by the right light guide plate 26, is provided in the right holding portion 21, and is located in the vicinity of the right side head of the user in the mounted state. The left display unit 24 is a unit related to the display of an image by the left light guide plate 28, is provided in the left holding portion 23, and is located in the vicinity of the user's left head in the mounted state.

  The right light guide plate 26 and the left light guide plate 28 of the present embodiment are optical parts formed of a light-transmissive resin or the like, and are, for example, prisms, and generate image light output from the right display unit 22 and the left display unit 24. , To the eyes of the user.

  Further, an electronic shade (not shown) having a dimming function may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The electronic shade has a terminal (not shown) for inputting a voltage, and a shade body (not shown) whose light transmittance changes according to the voltage between the terminals, and adjusts the applied voltage under the control of the control unit 141 described later. It is possible. The electronic shade may have a configuration in which the transmittance in the entire wavelength range including visible light changes, or a configuration in which the transmittance varies depending on the wavelength range of light. The electronic shade is arranged, for example, so as to cover the front side of the front frame 27, which is the side opposite to the eye side of the user. By adjusting the optical characteristics of the electronic shade, it is possible to adjust the amount of external light that enters the right light guide plate 26 and the left light guide plate 28 from the outside and that passes through the right light guide plate 26 and the left light guide plate 28.

  The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively, and allows the user to visually recognize a virtual image by the image light, thereby displaying an image. indicate. When external light passes through the right light guide plate 26 and the left light guide plate 28 from the front of the user and enters the user's eye, the image light and the external light forming a virtual image enter the user's eye. The visibility of the virtual image is affected by the intensity of outside light. Therefore, for example, by mounting the electronic shade on the front frame 27 and appropriately selecting or adjusting the optical characteristics of the electronic shade, the visibility of the virtual image can be adjusted.

  The camera 61 is arranged on the front frame 27 of the image display unit 20. It is desirable that the camera 61 capture an image of the outside view direction that the user visually recognizes with the image display unit 20 mounted, and blocks outside light that passes through the right light guide plate 26 and the left light guide plate 28 on the front surface of the front frame 27. It is provided in a position that does not exist. In the example of FIG. 1, the camera 61 is arranged on the end portion ER side of the front frame 27. The camera 61 may be arranged on the end EL side, or may be arranged on the connecting portion between the right light guide plate 26 and the left light guide plate 28. The camera 61 corresponds to an example of an “imaging unit”.

The camera 61 is a digital camera including an image pickup element such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor), an image pickup lens, and the like. The camera 61 of this embodiment is a monocular camera, but may be a stereo camera. The camera 61 captures an outside view of at least a part of the front side direction of the HMD 100, in other words, the user's view direction in the state where the HMD 100 is mounted. The outside scene corresponds to the real space. In other words, it can be said that the camera 61 captures a range or a direction overlapping with the user's field of view, and captures a direction in which the user is gazing. The width of the angle of view of the camera 61 can be set as appropriate, but in the present embodiment, as will be described later, the user includes the outside world visually recognized through the right light guide plate 26 and the left light guide plate 28. More preferably, the imaging range of the camera 61 is set so that the entire visual field of the user, which is visible through the right light guide plate 26 and the left light guide plate 28, can be imaged.
The camera 61 executes the imaging process according to the instruction of the control unit 141 and outputs the captured image data to the display control unit 142 and the determination unit 144. Each of the display control unit 142 and the determination unit 144 will be described later with reference to FIG.

  The HMD 100 includes a distance sensor 64 that detects a distance to a measurement object located in a preset measurement direction. The distance sensor 64 may be configured to detect a distance to a measurement object located in front of the user, and in the present embodiment, the distance between the right light guide plate 26 and the left light guide plate 28 in the front frame 27 is set. It is placed in the connecting part. In this example, in the mounted state of the image display unit 20, the position of the distance sensor 64 is substantially in the middle of both eyes of the user in the horizontal direction and above the both eyes of the user in the vertical direction. The measurement direction of the distance sensor 64 can be, for example, the front side direction of the front frame 27, in other words, the direction overlapping with the imaging direction of the camera 61.

  The distance sensor 64 has, for example, a light source such as an LED (Light Emitting Diode) or a laser diode, and a light receiving unit that receives the reflected light that the light emitted from the light source reflects on the measurement target. The distance sensor 64 may execute the triangular distance measuring process or the distance measuring process based on the time difference according to the instruction of the control unit 141. Further, the distance sensor 64 may be configured to include a sound source that emits ultrasonic waves and a detection unit that receives the ultrasonic waves reflected by the measurement target. In this case, the distance sensor 64 may execute the distance measurement process based on the time difference until the reflection of the ultrasonic wave according to the instruction of the control unit 141.

  The control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector 42 provided at the end of the case 10A. That is, the case 10A is provided with a connector 42 into / from which the connection cable 40 can be inserted / removed, and the connection cable 40 is connected to the connector 42 when the image display unit 20 is used.

The connection cable 40 connects from the tip of the left holding unit 23 to various circuits provided inside the image display unit 20. The connection cable 40 includes a metal cable that transmits digital data or an optical fiber cable, and may have a metal cable that transmits an analog signal. A connector 46 is provided in the middle of the connection cable 40.
The connector 46 is a jack for connecting a stereo mini plug, and the connector 46 and the control device 10 are connected by a line for transmitting an analog audio signal, for example. The connector 46 may be called an audio connector. In the configuration example shown in FIG. 1, the headset 30 including the right earphone 32 and the left earphone 34 that configure the stereo headphones, and the microphone 63 is connected to the connector 46.

  For example, as shown in FIG. 1, the microphone 63 is arranged such that the sound collecting unit of the microphone 63 faces the direction of the user's line of sight, collects sound, and outputs a sound signal to the sound interface 182 shown in FIG. .. The microphone 63 may be, for example, a monaural microphone or a stereo microphone, a directional microphone, or an omnidirectional microphone.

  The control device 10 includes a track pad 14, an up / down key 15, an LED display unit 17, and a power switch 18 as operated parts operated by a user. These operated parts are arranged on the surface of the case 10A. These operated parts are operated, for example, by the fingers of the user.

  The track pad 14 is an area on the front surface of the case 10A for a user to touch with a finger and perform a touch operation. The track pad 14 may be the same plane as the front surface of the case 10A, but it is preferable that the user can distinguish between the track pad 14 and other areas. For example, the line indicating the edge of the track pad 14 may be formed by printing or unevenness, or the track pad 14 may be subjected to a surface treatment that makes the feel of the surface of the track pad 14 different from other areas. ..

  The control device 10 can detect a touch operation by the user on the track pad 14 on the front surface of the case 10A by the touch sensor 13 shown in FIG. When the touch sensor 13 detects a contact operation, the control device 10 specifies the position where the operation is detected. The track pad 14 can be used for an operation of inputting an absolute position or a relative position on the track pad 14.

  An LED display unit 17 is installed on the front surface of the case 10A. The LED display unit 17 is located on the track pad 14, and the surface of the LED display unit 17 is not different from other regions on the front surface of the case 10A. The LED display unit 17 has a transmissive part (not shown) capable of transmitting light, and emits light so that a user can visually recognize a symbol or the like by lighting one or a plurality of LEDs installed immediately below the transmissive part. .. In the example of FIG. 1, three symbols Δ, symbol ◯, and symbol □ appear by turning on the LED of the LED display unit 17. The symbol Δ indicates a triangle. The symbol ○ indicates a circle. The symbol □ indicates a rectangle.

  The control device 10 can detect the touch operation of the user's finger on the LED display unit 17 by the touch sensor 13 and specify the operation position. Therefore, for example, it is possible to specify which of the symbols appearing on the LED display unit 17 corresponds to the operation position. Therefore, the LED display unit 17 functions as a software button. For example, by associating the symbol appearing on the LED display unit 17 with the function of the HMD 100, the touch operation on the LED display unit 17 can be detected as the operation for the function. In the example of FIG. 1, the HMD 100 can assign the symbol ◯ to the home button. In this case, when the touch operation is performed at the position of the symbol ◯, the control unit 141 detects the operation of the home button. The symbol □ can be assigned to the history button. In this case, the control unit 141 detects the touch operation of the symbol □ as the operation of the history button. Similarly, the symbol Δ can be assigned to the back button. The control unit 141 detects the touch operation of the symbol Δ as the operation of the return button.

The up / down key 15 is arranged on the side surface of the case 10A and includes a pair of keys for detecting a pressing operation. The up / down key 15 is used for inputting an instruction to increase / decrease the volume output from the right earphone 32 and the left earphone 34, and to input an instruction to increase / decrease the display brightness of the image display unit 20.
The power switch 18 is a switch that switches on and off the power of the HMD 100.

  A USB (Universal Serial Bus) connector 188 shown in FIG. 5 is provided on the same side surface of the case 10A as the power switch 18. The USB connector 188 is an interface for connecting the control device 10 to an external device, and in the present embodiment, a connector conforming to the USB standard is exemplified as an example of the interface. The USB connector 188 is, for example, a connector having a shape and size conforming to the microUSB standard, and specifications such as transfer speed are arbitrary.

  The control device 10 has a battery 132 shown in FIG. 5, and the control device 10 and the image display unit 20 operate by the power supplied by the battery 132. The battery 132 can be charged by supplying power to the USB connector 188. The HMD 100 can be charged by removing the control device 10 and the image display unit 20 and connecting only the control device 10 to a dedicated charging device.

FIG. 2 is a main part plan view showing a configuration of an optical system included in the image display unit 20. FIG. 2 illustrates the left eye LE and the right eye RE of the user for the sake of explanation.
As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured symmetrically. The right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 that emits image light as a configuration that allows the user's right eye RE to visually recognize an image. Further, the right optical system 251 including a lens group that guides the image light L emitted from the OLED unit 221 is provided. The image light L is guided to the right light guide plate 26 by the right optical system 251.

The OLED unit 221 has an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display panel configured by arranging light emitting elements that emit R, G, and B color lights by organic electroluminescence to be arranged in a matrix. R indicates red. G indicates green. B shows blue. The OLED panel 223 includes a plurality of pixels with a unit including one element of R, G, and B as one pixel, and an image is formed by the pixels arranged in a matrix. Under the control of the control unit 141, the OLED drive circuit 225 selects a light emitting element included in the OLED panel 223 and energizes the light emitting element to cause the light emitting element of the OLED panel 223 to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back surface of the light emitting surface, by bonding or the like. The OLED drive circuit 225 is composed of, for example, a semiconductor device that drives the OLED panel 223, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 223. The temperature sensor 217 shown in FIG. 5 is mounted on this board.
The OLED panel 223 may have a structure in which light emitting elements that emit white light are arranged in a matrix and color filters corresponding to the colors R, G, and B are arranged in an overlapping manner. Further, an OLED panel 223 having a WRGB structure including light emitting elements that emit W light in addition to light emitting elements that emit R, G, and B color lights respectively may be used. W indicates white.

  The right optical system 251 has a collimating lens that turns the image light L emitted from the OLED panel 223 into a parallel light flux. The image light L converted into a parallel light flux by the collimator lens enters the right light guide plate 26. Inside the right light guide plate 26, a plurality of reflecting surfaces that reflect the image light L are formed in an optical path that guides the light. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 located in front of the right eye RE is formed on the right light guide plate 26. The half mirror 261 corresponds to an example of a reflecting surface. The image light L is reflected by the half mirror 261 and emitted from the right light guide plate 26 toward the right eye RE, and the image light L forms an image on the retina of the right eye RE to allow the user to visually recognize the image.

  The left display unit 24 has a left optical system 252 including a OLED unit 241 that emits image light, a lens group that guides the image light L emitted by the OLED unit 241, and the like so that the left eye LE of the user can visually recognize the image. With. The image light L is guided to the left light guide plate 28 by the left optical system 252.

  The OLED unit 241 has an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. The OLED panel 243 is a self-luminous display panel configured similarly to the OLED panel 223. The OLED drive circuit 245 selects a light emitting element included in the OLED panel 243 and energizes the light emitting element according to an instruction from the control unit 141 to cause the light emitting element of the OLED panel 243 to emit light. The OLED drive circuit 245 is fixed to the back surface of the OLED panel 243, that is, the back surface of the light emitting surface, by bonding or the like. The OLED drive circuit 245 is composed of, for example, a semiconductor device that drives the OLED panel 243, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 243. A temperature sensor 239 is mounted on this board.

  The left optical system 252 has a collimating lens that converts the image light L emitted from the OLED panel 243 into a parallel light flux. The image light L converted into a parallel light flux by the collimator lens enters the left light guide plate 28. The left light guide plate 28 is an optical element having a plurality of reflecting surfaces that reflect the image light L, and is, for example, a prism. The image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 28. A half mirror 281 located in front of the left eye LE is formed on the left light guide plate 28. The half mirror 281 corresponds to an example of a reflecting surface. The image light L is reflected by the half mirror 281 and emitted from the left light guide plate 28 toward the left eye LE, and the image light L forms an image on the retina of the left eye LE to allow the user to visually recognize the image.

With this configuration, the HMD 100 functions as a transmissive display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 enter the right eye RE of the user. Further, the image light L reflected by the half mirror 281 and the external light OL transmitted through the half mirror 281 are incident on the left eye LE. As described above, the HMD 100 allows the image light L of the internally processed image and the external light OL to overlap and enter the user's eye, and allows the user to see the outside view through the right light guide plate 26 and the left light guide plate 28. The image by the image light L is visually recognized in a manner superimposed on the outside scene.
The half mirrors 261 and 281 are image extraction units that reflect the image lights output by the right display unit 22 and the left display unit 24 to extract images, and can be referred to as display units.

  The left optical system 252 and the left light guide plate 28 are collectively referred to as “left light guide portion”, and the right optical system 251 and the right light guide plate 26 are collectively referred to as “right light guide portion”. The configurations of the right light guide section and the left light guide section are not limited to the above examples, and any method can be used as long as a virtual image is formed in front of the user's eyes by using image light. For example, a diffraction grating is used. Alternatively, a semi-transmissive reflective film may be used.

  3 and 4 are diagrams showing a configuration of a main part of the image display unit 20. FIG. 3 is a perspective view of a main part of the image display unit 20 viewed from the user's head side. The connection cable 40 is not shown in FIG. FIG. 4 is a schematic diagram showing the relationship between the position of the object and the imaging range of the camera 61.

FIG. 3 is a side of the image display unit 20 that is in contact with the user's head, in other words, a side visible to the right eye RE and the left eye LE of the user. In other words, the back sides of the right light guide plate 26 and the left light guide plate 28 are visible in FIG.
In FIG. 3, the half mirror 261 that irradiates the right eye RE of the user with the image light and the half mirror 281 that irradiates the left eye LE of the image light with each other appear as substantially rectangular areas. Further, the entire right light guide plate 26 and left light guide plate 28 including the half mirrors 261 and 281 transmit outside light as described above. Therefore, the user sees the outside scene through the entire right light guide plate 26 and the left light guide plate 28, and sees the rectangular display image at the positions of the half mirrors 261 and 281.

  The camera 61 is arranged at the right end of the image display unit 20 as described above, and images the direction in which both eyes of the user face, that is, the front of the user. In FIG. 4, the position of the camera 61 is schematically shown in a plan view together with the right eye RE and the left eye LE of the user. The angle of view of the camera 61 is indicated by C. The angle of view C corresponds to an example of the imaging range. Although the horizontal view angle C is shown in FIG. 4, the actual view angle of the camera 61 extends in the vertical direction as in a general digital camera.

  The optical axis of the camera 61 is a direction including the line-of-sight directions of the right eye RE and the left eye LE. The external view that the user can see while wearing the HMD 100 is not limited to infinity. For example, as shown in FIG. 4, when the user gazes at the object OB with both eyes, the user's line of sight is directed to the object OB as indicated by reference characters RD and LD in the figure. In this case, the distance from the user to the object OB is often about 30 cm to 10 m, and more often about 1 m to 4 m. Therefore, for the HMD 100, the upper limit and the lower limit of the distance from the user to the object OB during normal use may be set. This standard may be obtained by research or experiment, or may be set by the user. When the distance to the object OB during normal use corresponds to the set upper limit and when the distance to the object OB corresponds to the set lower limit, the object OB is It is preferably set so as to be included in the angle of view.

  Generally, a human viewing angle is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction, and the effective field of view having excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. Furthermore, the stable gazing point where a gazing point to which a human gazes quickly and stably is set to about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. In this case, when the gazing point is the object OB of FIG. 4, the effective visual field is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction with the lines of sight RD and LD as the centers. Further, the stable fixation field is 60 to 90 degrees in the horizontal direction and 45 to 70 degrees in the vertical direction, and the viewing angle is approximately 200 degrees in the horizontal direction and 125 degrees in the vertical direction. Further, an actual visual field that the user sees through the image display unit 20 and the right light guide plate 26 and the left light guide plate 28 can be referred to as a real field of view (FOV). In the configuration of the present embodiment shown in FIG. 3, the actual visual field corresponds to the actual visual field that the user sees through the right light guide plate 26 and the left light guide plate 28. The real field of view is narrower than the viewing angle and the stable fixation field, but wider than the effective field of view.

  The angle of view C of the camera 61 is preferably capable of capturing a wider range than the visual field of the user, and specifically, the angle of view C is preferably at least wider than the effective visual field of the user. It is more preferable that the angle of view C is wider than the actual visual field of the user. More preferably, the angle of view C is wider than the stable fixation field of view of the user, and most preferably the angle of view C is wider than the viewing angles of both eyes of the user.

  The camera 61 may be provided with a so-called wide-angle lens as an imaging lens so that a wide field angle can be imaged. The wide-angle lens may include a lens called an ultra-wide-angle lens or a quasi-wide-angle lens, may be a single-focus lens or a zoom lens, and the camera 61 includes a lens group including a plurality of lenses. It may be.

  Further, the distance sensor 64 is arranged facing the front in the center of the right light guide plate 26 and the left light guide plate 28. For example, the distance sensor 64 is configured to be able to detect the distance from the center position of the image display unit 20 to an object located in the front direction of the user, such as the object OB shown in FIG. Since the user wearing the HMD 100 turns his or her head in the gaze direction, it can be considered that the gaze target is in front of the image display unit 20. Therefore, if the distance sensor 64 arranged in the center of the image display unit 20 sets the front direction of the image display unit 20 as the detection direction 64A, the distance to the target watched by the user can be detected.

Further, as shown in FIG. 3, an inner camera 68 is arranged on the user side of the image display unit 20. A pair of inner cameras 68 are provided at the center positions of the right light guide plate 26 and the left light guide plate 28 so as to correspond to the user's right eye RE and left eye LE, respectively. The inner cameras 68 are a pair of cameras that image the right eye RE and the left eye LE of the user, respectively. The inner camera 68 captures an image according to an instruction from the control unit 141. The control unit 141 analyzes the captured image data of the inner camera 68. For example, the control unit 141 detects the reflected light on the eyeball surface of the right eye RE and the left eye LE and the image of the pupil from the imaged image data of the inner camera 68, and specifies the user's gaze direction. Further, the control unit 141 may obtain a change in the line-of-sight direction of the user, and may detect the eye movements of the right eye RE and the left eye LE.
Here, the movement of the user's line of sight can also be regarded as the movement of the user's virtual viewpoint.

  The control unit 141 may also extract eyelid images of the user's right eye RE and left eye LE from the imaged image data of the inner camera 68 to detect eyelid movements, or detect eyelid states. Good. The eyelid is sometimes called the eyelid. In the present embodiment, the image display unit 20 includes a pair of inner cameras 68, 68 as an example, but one inner camera 68 may be provided at the center position of the image display unit 20, for example. In this case, it is preferable that one inner camera 68 has an angle of view capable of picking up the right eye RE and the left eye LE. However, for example, even if only one of the right eye RE and the left eye LE is picked up by the inner camera 68. Good. That is, the control unit 141 may detect the line-of-sight direction of one of the right eye RE and the left eye LE, eye movement, eyelid movement, eyelid state, and the like.

  Further, the control unit 141 can obtain the vergence angles of the right eye RE and the left eye LE when detecting the line-of-sight directions of the right eye RE and the left eye LE from the image captured by the inner camera 68. In FIG. 4, the vergence angle is indicated by the symbol PA. The vergence angle PA corresponds to the distance to the object OB that the user gazes at. That is, when a user stereoscopically views an image or an object, the vergence angles of the right eye RE and the left eye LE are determined according to the distance to the object to be viewed. Therefore, by detecting the angle of convergence, it is possible to determine the distance the user is gazing. In addition, stereoscopic vision can be guided by displaying an image so as to guide the vergence angle of the user.

  The vergence angle can be obtained from the imaged image data of the inner camera 68, for example. For example, the line-of-sight direction of the right eye RE is obtained from the captured image data by the inner camera 68, and the angle LA of the line-of-sight direction of the right eye RE with respect to the front direction of the right eye RE is obtained from this line-of-sight direction. Similarly, the line-of-sight direction of the left eye LE is obtained from the imaged image data of the inner camera 68, and the angle RA of the line-of-sight direction of the left eye LE with respect to the front direction of the left eye LE is obtained based on this line-of-sight direction. The vergence angle PA is equal to the sum of the angle LA and the angle RA, and the vergence angle PA can be easily obtained.

FIG. 5 is a diagram showing a configuration of each unit that constitutes the HMD 100.
As shown in FIG. 5, the control device 10 includes a main processor 140 that executes a program to control the HMD 100. A memory 118 and a non-volatile storage unit 121 are connected to the main processor 140. The operation unit 110 is connected to the main processor 140 as an input device. A 6-axis sensor 111, a magnetic sensor 113, and a GPS 115 are connected to the main processor 140 as sensors. Further, the main processor 140 is connected with the communication unit 117, the audio codec 180, the external connector 184, the external memory interface 186, the USB connector 188, the sensor hub 192, and the FPGA 194. These function as an interface with the outside.

  The main processor 140 is mounted on the controller board 120 included in the control device 10. In addition to the main processor 140, the memory 118, the non-volatile storage unit 121, and the like may be mounted on the controller board 120. In this embodiment, the 6-axis sensor 111, the magnetic sensor 113, the GPS 115, the communication unit 117, the memory 118, the non-volatile storage unit 121, the audio codec 180, etc. are mounted on the controller board 120. Further, the external connector 184, the external memory interface 186, the USB connector 188, the sensor hub 192, the FPGA 194, and the interface 196 may be mounted on the controller board 120.

  The memory 118 constitutes a work area for temporarily storing a control program to be executed and data to be processed when the main processor 140 executes the control program. The non-volatile storage unit 121 is composed of a flash memory or an eMMC (embedded Multi Media Card). The non-volatile storage unit 121 stores a program executed by the main processor 140 and various data processed by the main processor 140 by executing the program.

The main processor 140 detects a touch operation on the operation surface of the track pad 14 based on an operation signal input from the operation unit 110, and acquires an operation position.
The operation unit 110 includes a button 11, a touch sensor 13, and an LED display unit 17. The touch sensor 13 detects a touch operation on the track pad 14 and specifies the operation position of the detected touch operation. In this case, the operation unit 110 outputs an operation signal including data indicating the touch position on the track pad 14 to the main processor 140. When the button 11 is operated and when the touch sensor 13 detects a touch operation, an operation signal is output from the operation unit 110 to the main processor 140.
The LED display unit 17 includes an LED (not shown) arranged immediately below the track pad 14 shown in FIG. 3, and a drive circuit for lighting the LED. The LED display unit 17 turns on, blinks, and turns off the LED under the control of the main processor 140.

The 6-axis sensor 111 is a motion sensor including a 3-axis acceleration sensor and a 3-axis gyro sensor. The gyro sensor corresponds to an example of the angular velocity sensor. The motion sensor may be called an inertial sensor. The 6-axis sensor 111 may employ an IMU (Internal Measurement Unit) in which the above sensor is modularized.
The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor.
A GPS (Global Positioning System) 115 includes a GPS antenna (not shown), receives a radio signal transmitted from a GPS satellite, and detects the coordinates of the current position of the control device 10.
The 6-axis sensor 111, the magnetic sensor 113, and the GPS 115 output the detected value to the main processor 140 according to a sampling cycle designated in advance. Alternatively, the 6-axis sensor 111, the magnetic sensor 113, and the GPS 115 output detection values to the main processor 140 at a timing designated by the main processor 140 in response to a request from the main processor 140.

  The communication unit 117 executes wireless communication with an external device. In the present embodiment, the communication unit 117 executes wireless communication with the myoelectric sensor 200. The communication unit 117 is configured to include an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like, or a device in which these are integrated. The communication unit 117 performs wireless communication in compliance with standards such as Bluetooth (registered trademark) and wireless LAN. The wireless LAN includes Wi-Fi (registered trademark).

The voice interface 182 is an interface that inputs and outputs a voice signal. In the present embodiment, the audio interface 182 includes the connector 46 shown in FIG. 3 provided on the connection cable 40. The audio codec 180 is connected to the audio interface 182, and encodes and decodes an audio signal input / output via the audio interface 182. The audio codec 180 may include an A / D converter that converts an analog audio signal into digital audio data, and a D / A converter that performs the reverse conversion. For example, in the HMD 100 of this embodiment, sound is output by the right earphone 32 and the left earphone 34, and is collected by the microphone 63. The audio codec 180 converts the digital audio data output by the main processor 140 into an analog audio signal and outputs the analog audio signal via the audio interface 182. The audio codec 180 also converts an analog audio signal input to the audio interface 182 into digital audio data and outputs the digital audio data to the main processor 140.
The microphone 63, the voice interface 182, and the voice codec 180 correspond to an example of “voice input unit”.

The external connector 184 is a connector for connecting an external device that communicates with the main processor 140. The external connector 184 is, for example, an interface that connects an external device to the main processor 140 when the program executed by the main processor 140 is debugged or the operation log of the HMD 100 is collected. Is.
The external memory interface 186 is an interface to which a portable memory device can be connected, and includes, for example, a memory card slot into which a card type recording medium can be attached and which can read data, and an interface circuit. The size, shape, and standard of the card-type recording medium in this case are not limited and can be changed as appropriate.

  The USB connector 188 includes a connector conforming to the USB standard and an interface circuit, and can be connected to a USB memory device, a smartphone, a computer, or the like. The size and shape of the USB connector 188 and the compatible version of the USB standard can be appropriately selected and changed. When connecting the HMD 100 and the remote controller 350 with a USB cable, the USB connector 188 is connected to the remote controller 350.

  Further, the HMD 100 includes a vibrator 19. The vibrator 19 includes a motor (not shown), an eccentric rotor (not shown), etc., and generates vibration under the control of the main processor 140. The HMD 100 causes the vibrator 19 to vibrate in a predetermined vibration pattern when, for example, an operation on the operation unit 110 is detected, or when the HMD 100 is powered on and off.

  The image display unit 20 is connected to the sensor hub 192 and the FPGA 194 via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20 and outputs the detection values to the main processor 140. Further, the FPGA 194 executes processing of data transmitted and received between the main processor 140 and each unit of the image display unit 20, and transmission via the interface 196.

  The right display unit 22 and the left display unit 24 of the image display unit 20 are connected to the control device 10, respectively. As shown in FIG. 3, in the HMD 100, the connection cable 40 is connected to the left holding portion 23, the wiring connected to the connection cable 40 is laid inside the image display unit 20, and the right display unit 22 and the left display unit 24 are respectively provided. Are connected to the control device 10.

The right display unit 22 has a display unit substrate 210. An interface (I / F) 211 connected to the interface 196, a receiver (Rx) 213 for receiving data input from the control device 10 via the interface 211, and an EEPROM 215 are mounted on the display unit substrate 210. It The EEPROM 215 corresponds to an example of a storage unit.
The interface 211 connects the receiving unit 213, the EEPROM 215, the temperature sensor 217, the camera 61, the illuminance sensor 65, and the LED indicator 67 to the control device 10.

  An EEPROM (Electrically Erasable Programmable ROM) 215 stores various data so that the main processor 140 can read the data. The EEPROM 215 stores, for example, data regarding emission characteristics and display characteristics of the OLED units 221 and 241 included in the image display unit 20, data regarding characteristics of a sensor included in the right display unit 22 or the left display unit 24, and the like. Specifically, parameters for gamma correction of the OLED units 221, 241 and data for compensating the detection values of the temperature sensors 217, 239 are stored. These data are generated by the factory inspection of the HMD 100 and written in the EEPROM 215, and after the shipment, the main processor 140 can perform processing by using the data of the EEPROM 215.

The camera 61 performs imaging according to a signal input via the interface 211, and outputs captured image data or a signal indicating an imaging result to the control device 10.
As shown in FIG. 3, the illuminance sensor 65 is provided at the end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the amount of received light. The amount of received light may be referred to as received light intensity.
As shown in FIG. 3, the LED indicator 67 is arranged near the camera 61 at the end ER of the front frame 27. The LED indicator 67 lights up while the image pickup by the camera 61 is being performed, and notifies that the image pickup is in progress.

  The temperature sensor 217 detects temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 217 is mounted on the back surface side of the OLED panel 223 shown in FIG. The temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225, for example. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223.

  The inner camera 68 performs imaging according to a signal input from the control device 10 via the interface 211, and outputs captured image data or a signal indicating an imaging result to the control device 10. The pair of inner cameras 68, 68 shown in FIG. 3 may operate simultaneously. Further, each of the pair of inner cameras 68, 68 may be connected to the interface 211 and operate independently.

  The distance sensor 64 performs distance detection according to a signal input from the control device 10 via the interface 211, and outputs a signal indicating the detection result to the control device 10. The pair of distance sensors 64, 64 shown in FIG. 3 may operate simultaneously. Further, each of the pair of distance sensors 64, 64 may be connected to the interface 211 and operate independently.

  The receiving unit 213 receives data transmitted by the main processor 140 via the interface 211. When receiving the image data of the image displayed by the OLED unit 221, the receiving unit 213 outputs the received image data to the OLED drive circuit 225 shown in FIG.

The left display unit 24 has a display unit substrate 210. An interface (I / F) 231 connected to the interface 196, and a receiver 233 that receives data input from the control device 10 via the interface 231 are mounted on the display unit substrate 210. The receiving unit may be described as “Rx”. A 6-axis sensor 235 and a magnetic sensor 237 are mounted on the display unit substrate 210. The 6-axis sensor 235 corresponds to an example of “motion sensor”.
The interface 231 connects the receiving unit 233, the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 to the control device 10.

The 6-axis sensor 235 is a motion sensor including a 3-axis acceleration sensor and a 3-axis gyro sensor. The 6-axis sensor 235 may employ an IMU (Internal Measurement Unit) in which the above sensor is modularized.
The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor.

The temperature sensor 239 detects the temperature and outputs the voltage value or the resistance value corresponding to the detected temperature as the detected value. The temperature sensor 239 is mounted on the back surface side of the OLED panel 243 shown in FIG. The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243.
Further, the temperature sensor 239 may be built in the OLED panel 243 or the OLED drive circuit 245. Further, the substrate may be a semiconductor substrate. Specifically, when the OLED panel 243 is mounted as a Si-OLED together with the OLED drive circuit 245 and the like as an integrated circuit on an integrated semiconductor chip, the temperature sensor 239 may be mounted on this semiconductor chip.

The camera 61, the distance sensor 64, the illuminance sensor 65, the inner camera 68, the temperature sensor 217 included in the right display unit 22 and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 included in the left display unit 24 are connected to the sensor hub 192. To be done. The sensor hub 192 sets and initializes the sampling cycle of each sensor under the control of the main processor 140. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling cycle of each sensor. Further, the sensor hub 192 outputs the detection value of each sensor included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a function of temporarily holding the detection value of each sensor in synchronization with the timing of output to the main processor 140. Further, the sensor hub 192 may have a function of converting to a signal format or a data format of an output value of each sensor, converting the data into a unified data format, and outputting the data to the main processor 140.
Further, the sensor hub 192 starts and stops energization of the LED indicator 67 under the control of the main processor 140, and turns on or blinks the LED indicator 67 at the timing when the camera 61 starts and ends image capturing.

  The control device 10 includes a power supply unit 130 and operates by electric power supplied from the power supply unit 130. The power supply unit 130 includes a rechargeable battery 132 and a power supply control circuit 134 that detects the remaining capacity of the battery 132 and controls charging of the battery 132. The power supply control circuit 134 is connected to the main processor 140 and outputs the detected value of the remaining capacity of the battery 132 or the detected value of the voltage to the main processor 140. Further, the control device 10 may supply power to the image display unit 20 based on the power supplied by the power supply unit 130. Further, the main processor 140 may control the power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20.

  The HMD 100 may include an interface (not shown) that connects various external devices that are content supply sources. For example, the interface may be a wired interface such as a USB interface, a micro USB interface, or a memory card interface, or may be a wireless communication interface. The external device in this case is an image supply device that supplies an image to the HMD 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used. In this case, the HMD 100 can output images and sounds based on the content data input from these external devices.

FIG. 6 is a diagram showing the configuration of the control unit 141.
As shown in FIG. 6, the control unit 141 includes a display control unit 142, an operation estimation unit 143, a determination unit 144, a learning unit 145, a signal storage unit 146, a learning data storage unit 147, and an image storage unit. And 148. Specifically, the main processor 140 executes the control program of the memory 118 to function as the display control unit 142, the operation estimation unit 143, the determination unit 144, and the learning unit 145. Further, the main processor 140 executes the control program of the memory 118 to cause the memory 118 to function as the signal storage unit 146, the learning data storage unit 147, and the image storage unit 148.

The display control unit 142 causes the image display unit 20 to display the plurality of input images MN at a position overlapping the user's hand which is visible through the image display unit 20.
Specifically, the display control unit 142 specifies the position of the user's hand included in the outside scene, based on the image data acquired by the camera 61. Then, the display control unit 142 causes the image display unit 20 to display the plurality of input images MN so as to overlap the specified position of the user's hand. That is, the display control unit 142 determines the display position of the plurality of input images MN to be the position of the identified user's hand.
The display control unit 142 also tracks the position of the user's hand that is visually recognized through the image display unit 20, and adjusts the display position of the plurality of input images MN according to the movement of the user's hand.
Specifically, each of the plurality of input images MN shows different characters. The plurality of input images MN will be described later in detail with reference to FIG.
The display control unit 142 also causes the image display unit 20 to display the first guide image GL1 that guides the position of the user's hand. The display control unit 142 also causes the image display unit 20 to display the second guide image GL2 that guides the position of the myoelectric sensor 200. The first guide image GL1 and the second guide image GL2 will be described later in detail with reference to FIG. 7.

The signal storage unit 146 stores the reference value of the myoelectric sensor 200.
Specifically, the signal storage unit 146 stores the reference value of the myoelectric sensor 200 corresponding to the operation of selecting each of the plurality of input images MN.
The reference value of the myoelectric sensor 200 stored in the signal storage unit 146 is obtained by, for example, pre-calibration. For example, the reference value of the myoelectric sensor 200 is stored in association with the information indicating the user. In other words, the reference value of the myoelectric sensor 200 is stored for each user.
Further, the reference value of the myoelectric sensor 200 stored in the signal storage unit 146 is repeatedly detected with the user wearing the myoelectric sensor 200 at an appropriate position. Then, the average value is stored in the signal storage unit 146. Further, instead of the average value, the range of the reference value of the myoelectric sensor 200 may be stored in the signal storage unit 146.

  The image storage unit 148 stores the image displayed by the display control unit 142. Specifically, the image storage unit 148 stores images such as the plurality of input images MN, the first guide image GL1, and the second guide image GL2.

The operation estimating unit 143 acquires the detection value of the myoelectric sensor 200, and estimates the operation of selecting one input image MA from the plurality of input images MN based on the detection value of the myoelectric sensor 200. ..
More specifically, in the period until the learning unit 145 generates the determination criterion, the operation estimation unit 143 performs the operation of selecting one input image MA from the plurality of input images MN as follows. To estimate. That is, the operation estimation unit 143, based on the detected value of the myoelectric sensor 200 and the reference value of the myoelectric sensor 200 stored in the signal storage unit 146, selects one input image from the plurality of input images MN. The operation of selecting the image MA is estimated.
In the period after the learning unit 145 has generated the determination criterion, the operation estimation unit 143 estimates the operation of selecting one input image MA from the plurality of input images MN as follows. That is, the operation estimation unit 143 estimates the operation of selecting one input image MA from the plurality of input images MN based on the detection value of the myoelectric sensor 200 and the determination criterion.

The determination unit 144 determines the estimation result of the operation estimation unit 143.
Specifically, the determination unit 144 determines whether the estimation result of the operation estimation unit 143 is correct based on the image captured by the camera 61.
More specifically, the camera 61 generates a moving image of the user's hand including the position where each of the plurality of input images MN is displayed by the display control unit 142. Then, the determination unit 144 determines whether the estimation result of the operation estimation unit 143 is correct based on the moving image of the user's hand.

That is, when the moving image of the user's hand indicates the operation of selecting one input image MA from the plurality of input images MN, the determination unit 144 determines that the estimation result of the operation estimation unit 143 is correct. judge. When the moving image of the user's hand does not indicate the operation of selecting one input image MA from the plurality of input images MN, the determination unit 144 determines that the estimation result of the operation estimation unit 143 is incorrect. judge.
When the moving image of the user's hand does not show the operation of selecting one input image MA from the plurality of input images MN, the following two cases are included. The first case is a case where the moving image of the user's hand indicates an operation of selecting an input image other than one input image MA from the plurality of input images MN. The second case is a case where the moving image of the user's hand does not indicate the operation of selecting one of the input images MN.
In the present embodiment, the determination unit 144 determines whether the estimation result of the operation estimation unit 143 is correct based on the image captured by the camera 61, but is not limited to this. The determination unit 144 may determine whether the estimation result of the operation estimation unit 143 is correct, for example, based on the operation by the user. Further, the determination unit 144 may determine whether the estimation result of the operation estimation unit 143 is correct, for example, based on the voice from the user.

  The learning data storage unit 147 stores the learning data set 147a. The learning data set 147a includes a combination of the detection value of the myoelectric sensor 200, one input image MA, and the determination result of the determination unit 144.

The learning unit 145 machine-learns the conditions for specifying one input image MA from the detection value of the myoelectric sensor 200 based on the learning data set 147a.
The condition for specifying one input image MA from the detection values of the myoelectric sensor 200 is, for example, that the detection value of the myoelectric sensor 200 selects one input image MA from a plurality of input images MN. This is a criterion for determining whether or not to indicate an operation to be performed. The machine learning process executed by the learning unit 145 will be described later in detail with reference to FIG. 9.
In addition, the learning unit 145 stores the learning data set 147a in the learning data storage unit 147. Specifically, the learning unit 145 stores the learning data set 147 a in the learning data storage unit 147 every time the determination unit 144 determines whether the estimation result of the operation estimation unit 143 is correct.

FIG. 7 is a diagram showing the guide image GL.
In FIG. 7, a visual field area AR1 and an image area AR2 are shown. The visual field area AR1 indicates the range of the scenery in the real space that can be recognized by the user by the external light transmitted through the right light guide plate 26 and the external light transmitted through the left light guide plate 28. In other words, the visual field area AR1 indicates the visual field range of the user wearing the HMD 100. The visual field area AR1 includes the right hand HP of the user. A myoelectric sensor 200 is attached to the wrist position PS1 of the user's right hand HP.
The image area AR2 indicates the range of the image that can be recognized by the user by the image light guided by the right light guide plate 26 and the image light guided by the left light guide plate 28. In other words, the image area AR2 shows the range of the image that can be recognized by the user wearing the HMD 100. The image area AR2 includes the first guide image GL1 and the second guide image GL2.

The first guide image GL1 guides the position of the user's hand. Specifically, the first guide image GL1 indicates the position where the user's right hand HP should be placed in the real space. As shown in FIG. 7, the user's right hand HP is arranged at the position indicated by the first guide image GL1.
The second guide image GL2 shows a mounting position of the myoelectric sensor 200 on the wrist of the user's right hand HP for detecting an appropriate electric signal by the myoelectric sensor 200. In FIG. 7, the second guide image GL2 is closer to the palm than the position PS1 indicating the position of the myoelectric sensor 200 in the real space.
In this way, when the position PS1 of the myoelectric sensor 200 does not overlap the second guide image GL2, the display control unit 142 blinks the second guide image GL2.
In addition, the user needs to bring the myoelectric sensor 200 close to the palm so that the position PS1 of the myoelectric sensor 200 overlaps the second guide image GL2.

  In the present embodiment, when the position PS1 of the myoelectric sensor 200 does not overlap the second guide image GL2, the display control unit 142 blinks and displays the second guide image GL2, but the present invention is not limited to this. When the position PS1 of the myoelectric sensor 200 does not overlap with the second guide image GL2, the display control unit 142 may change the display form of the second guide image GL2. For example, the display control unit 142 may change the display color of the second guide image GL2.

FIG. 8 is a diagram showing the input image MN according to the present embodiment.
Similar to FIG. 7, FIG. 8 shows a visual field area AR1 and an image area AR2.
The image area AR2 includes the right hand HP of the user. A myoelectric sensor 200 is attached to the wrist position PS1 of the user's right hand HP.

Further, in the present embodiment, a case where the HMD 100 receives an input of a user ID will be described. The guide image GP1 is displayed in the image area AR2 of FIG. “ID: 123_” is displayed in the guide image GP1.
The guide image GP1 shows that the numbers “1”, “2”, and “3” have been input as user IDs. Further, "_" indicating the cursor on the guide image GP1 indicates that the user ID is waiting for the input of the fourth character.

Each of the plurality of input images MN indicates different characters. As shown in FIG. 8, for example, each of the plurality of input images MN includes numbers “0” to “9”, a symbol “−”, a determination command, and a return command.
The decision command indicates a command to decide the input of the user ID. The return command indicates a command to erase one of the characters input as the user ID.
Specifically, each of the plurality of input images MN is associated with a region demarcated by a joint in the finger of the user's right hand HP.
More specifically, each of the plurality of input images MN is associated with a region separated by a joint in each finger of the user's right hand HP except the thumb FP1.

The user's right hand HP includes a thumb FP1, an index finger FP2, a middle finger FP3, a ring finger FP4, and a little finger FP5.
The index finger FP2 includes a first joint FP21, a second joint FP22, and a third joint FP23.
In the index finger FP2, the input image M21 is displayed in the region on the tip side of the first joint FP21. The input image M21 indicates the number "1". The tip side indicates the side away from the palm.
That is, the display control unit 142 places the input image M21 at a position overlapping the region on the distal end side of the first joint FP21 of the index finger FP2 of the user's right hand HP that is visible through the image display unit 20. 20 to display.
Specifically, the display control unit 142 specifies the position of the region on the tip side of the first joint FP21 of the index finger FP2 of the user's right hand HP included in the outside scene, based on the image data acquired by the camera 61. Then, the display position of the input image M21 is determined to be the position of the region on the distal end side of the first joint FP21 of the index finger FP2 of the identified right hand HP of the user.
The input image M21 is included in the plurality of input images MN.
Further, for the input images MN excluding the input images M21 included in the plurality of input images MN, reference numeral MN indicating the plurality of input images MN is omitted in FIG.

An input image M22 is displayed in the area between the first joint FP21 and the second joint FP22. The input image M22 indicates the number "5".
That is, the display control unit 142 positions the input image M22 so as to overlap a region between the first joint FP21 and the second joint FP22 of the index finger FP2 of the user's right hand HP that is visually recognized through the image display unit 20. Then, the image is displayed on the image display unit 20.
On the index finger FP2, the input image M23 is displayed in a region between the second joint FP22 and the third joint FP23. The input image M23 indicates the number "9".
That is, the display control unit 142 positions the input image M23 so as to overlap with a region between the second joint FP22 and the third joint FP23 of the index finger FP2 of the user's right hand HP that is visually recognized through the image display unit 20. Then, the image is displayed on the image display unit 20.
An input image M24 is displayed in the area of the palm of the index finger FP2 that is close to the third joint FP23. The input image M24 shows a decision command.
That is, the input image M24 shows a command for confirming the input of each number and symbol forming the user ID. Further, the input image M24 shows a command to confirm the user ID. The input image M24 shows the character string of “determination”.
In addition, the display control unit 142 displays the input image M24 on the image display unit 20 at a position overlapping the third joint FP23 of the user's right hand HP visible through the image display unit 20 and the area of the palm adjacent to the third joint FP23. Display it.

The middle finger FP3 includes a first joint FP31, a second joint FP32, and a third joint FP33.
In the middle finger FP3, the input image M31 is displayed in the region on the tip side of the first joint FP31. The input image M31 shows the number “2”.
In the middle finger FP3, the input image M32 is displayed in the area between the first joint FP31 and the second joint FP32. The input image M32 indicates the number "6".
In the middle finger FP3, the input image M33 is displayed in the area between the second joint FP32 and the third joint FP33. The input image M33 indicates the number "0".
The display control unit 142 images each of the input image M31 to the input image M33 at a position that overlaps a region separated by a joint in the middle finger FP3 of the user's right hand HP that is visually recognized through the image display unit 20. It is displayed on the display unit 20. The regions delimited by the joints are a region on the distal end side of the first joint FP31, a region between the first joint FP31 and the second joint FP32, and a region between the second joint FP32 and the third joint FP33. including.

The ring finger FP4 includes a first joint FP41, a second joint FP42, and a third joint FP43.
In the ring finger FP4, the input image M41 is displayed in the region on the tip side of the first joint FP41. The input image M41 indicates the number "3".
On the ring finger FP4, an input image M42 is displayed in the area between the first joint FP41 and the second joint FP42. The input image M42 indicates the number "7".
On the ring finger FP4, the input image M43 is displayed in the region between the second joint FP42 and the third joint FP43. The input image M43 shows the symbol "-".
The display control unit 142 places each of the input image M41 to the input image M43 at a position that overlaps with a region separated by a joint in the ring finger FP4 of the user's right hand HP that is visually recognized through the image display unit 20. It is displayed on the display unit 20. The regions delimited by the joints include a region on the distal end side of the first joint FP41, a region between the first joint FP41 and the second joint FP42, and a region between the second joint FP42 and the third joint FP43. including.

The little finger FP5 includes a first joint FP51, a second joint FP52, and a third joint FP53.
In the little finger FP5, the input image M51 is displayed in the region on the tip side of the first joint FP51. The input image M51 indicates the number "4".
In the little finger FP5, the input image M52 is displayed in the area between the first joint FP51 and the second joint FP52. The input image M52 indicates the number "8".
In the little finger FP5, the input image M53 is displayed in the area between the second joint FP52 and the third joint FP53. The input image M53 shows a command to erase one of the characters input as the user ID. The input image M53 shows the character “return”.
The display control unit 142 places each of the input image M51 to the input image M53 at a position that overlaps with a region demarcated by a joint in the little finger FP5 of the user's right hand HP that is seen through the image display unit 20. It is displayed on the display unit 20. The regions delimited by the joints are a region on the distal end side of the first joint FP51, a region between the first joint FP51 and the second joint FP52, and a region between the second joint FP52 and the third joint FP53. including.

The operation estimation unit 143 estimates the operation of selecting one input image MA from the plurality of input images MN based on the detection value of the myoelectric sensor 200.
Specifically, in the operation of selecting one input image MA, the thumb FP1 of the user's right hand HP is pressed against the regions delimited by the joints of the index finger FP2, the middle finger FP3, the ring finger FP4, and the little finger FP5. It is an operation.
In other words, the gesture of selecting one input image MA is a gesture of pressing the thumb FP1 of the user's right hand HP to the joint-separated area of each of the index finger FP2, the middle finger FP3, the ring finger FP4, and the little finger FP5. Is.
Since this gesture is detected by the operation estimation unit 143 based on the detection value of the myoelectric sensor 200, it may be described as “myoelectric gesture” so as to be distinguishable from a general gesture.
The thumb FP1 of the user's right hand HP includes a first joint FP11 and a second joint FP12. A guide image M1 is displayed in the region of the thumb FP1 on the distal end side of the first joint FP11. The guide image M1 shows, for example, a symbol of a circle “◯”.
The guide image M1 indicates that one input image MA is selected by the operation of pressing the region on the distal end side of the first joint FP11 to the region divided by the joint.

For example, in the case of selecting the input image M21 indicating the number “1”, the user sets the region on the tip side of the first joint FP11 of the thumb FP1 to the region on the tip side of the first joint FP21 of the index finger FP2. Perform pressing operation.
Further, for example, when selecting the input image M32 indicating the number “6”, the user sets the region on the tip side of the first joint FP11 of the thumb FP1 to the first joint FP31 and the second joint FP3 of the middle finger FP3. The operation of pressing the area between the FP 32 and the FP 32 is performed.
Further, for example, when selecting the input image M43 indicating the symbol “−”, the user sets the region on the tip side of the first joint FP11 of the thumb FP1 to the second joint FP42 and the third joint of the ring finger FP4. The operation of pressing the area between the FP 43 and the FP 43 is performed.
Further, for example, when selecting the input image M24 indicating the determination command, the user sets the region on the tip side of the first joint FP11 of the thumb FP1 to the region of the palm adjacent to the third joint FP23 of the index finger FP2. Perform pressing operation.

Further, the operation estimation unit 143 estimates the operation of selecting one input image MA from the plurality of input images MN based on the detection value of the myoelectric sensor 200.
The plurality of input images MN include numbers “0” to “9”, a symbol “−”, a determination command, and a return command. Specifically, the plurality of input images MN include the input image M21 to the input image M24, the input image M31 to the input image M33, the input image M41 to the input image M43, and the input image M51 to the input image M51. It corresponds to the image M53 for use. That is, the plurality of input images MN represent the 13 input images MN shown in FIG. The operation estimation unit 143 estimates the operation of selecting one input image MA from the 13 input images MN based on the detection value of the myoelectric sensor 200.

The signal storage unit 146 stores the reference value of the myoelectric sensor 200 corresponding to the operation of selecting each of the 13 input images MN.
The reference value of the myoelectric sensor 200 stored in the signal storage unit 146 is an operation for selecting each of the 13 input images MN for each user while the user wears the myoelectric sensor 200 at an appropriate position. It is repeatedly detected every time. Then, the average value is stored in the signal storage unit 146 as the reference value of the myoelectric sensor 200 in association with the operation of selecting each of the 13 input images MN. Further, instead of the average value, the range of the detection value of the myoelectric sensor 200 may be stored in the signal storage unit 146.

The operation estimation unit 143, based on the detection value of the myoelectric sensor 200 and the reference value of the myoelectric sensor 200 stored in the signal storage unit 146, selects one input image from the 13 input images MN. Estimate the operation of selecting MA.
For example, when the detected value of the myoelectric sensor 200 indicates an operation of pressing the region on the tip side of the first joint FP11 of the thumb FP1 to the region on the tip side of the first joint FP21 of the index finger FP2, the operation estimation unit 143 estimates the operation of selecting the input image M21.
Further, for example, the detection value of the myoelectric sensor 200 indicates an operation of pressing the region on the tip side of the first joint FP11 of the thumb FP1 to the region between the first joint FP31 and the second joint FP32 of the middle finger FP3. In that case, the operation estimation unit 143 performs the following processing. That is, the operation estimation unit 143 estimates the operation of selecting the input image M32.
In addition, for example, the detection value of the myoelectric sensor 200 indicates an operation of pressing the region on the tip side of the first joint FP11 of the thumb FP1 to the region between the second joint FP42 and the third joint FP43 of the ring finger FP4. In that case, the operation estimation unit 143 performs the following processing. That is, the operation estimation unit 143 estimates the operation of selecting the input image M43.
Further, for example, in the case where the detection value of the myoelectric sensor 200 indicates an operation of pressing the region on the tip side of the first joint FP11 of the thumb FP1 to the region of the palm of the index finger FP2 that is close to the third joint FP23, The operation estimation unit 143 performs the following processing. That is, the operation estimation unit 143 estimates the operation of selecting the input image M24.

  In this way, the operation estimation unit 143 selects from the 13 input images MN based on the detected value of the myoelectric sensor 200 and the reference value of the myoelectric sensor 200 stored in the signal storage unit 146. The operation of selecting one input image MA is estimated. As a result, the user ID is input.

FIG. 9 is a flowchart showing the processing of the control unit 141. In the present embodiment, a case where the user's right hand HP is arranged at the position indicated by the first guide image GL1 will be described.
As shown in FIG. 9, first, in step S101, the display control unit 142 displays the guide image GL. The guide image GL includes a first guide image GL1 that guides the position of the user's right hand HP and a second guide image GL2 that guides the position of the myoelectric sensor 200.
Next, in step S103, the display control unit 142 determines, based on the second guide image GL2, whether or not the position PS1 at which the myoelectric sensor 200 is attached is appropriate.
When the display control unit 142 determines that the position PS1 where the myoelectric sensor 200 is attached is appropriate (YES in step S103), the process proceeds to step S107. When the display control unit 142 determines that the position PS1 where the myoelectric sensor 200 is attached is not appropriate (NO in step S103), the process proceeds to step S105.
Then, in step S105, the user corrects the position PS1 at which the myoelectric sensor 200 is attached. In other words, the display control unit 142 detects that the position PS1 where the myoelectric sensor 200 is attached is not appropriate, and displays the second guide image GL2 in a blinking manner. Then, the display control unit 142 detects that the position PS1 where the myoelectric sensor 200 is attached is corrected. Then, the process returns to step S103.

  If YES in step S103, the display control unit 142 causes the image display unit 20 to display the plurality of input images MN in step S107. Specifically, the display control unit 142 causes the image display unit 20 to display the plurality of input images MN at a position overlapping the right hand HP of the user who is visually recognized through the image display unit 20.

Next, in step S109, the operation estimation unit 143 determines whether or not the detection value of the myoelectric sensor 200 is acquired.
When the operation estimation unit 143 determines that the detection value of the myoelectric sensor 200 is not acquired (NO in step S109), the process is in a standby state. When the operation estimation unit 143 determines that the detection value of the myoelectric sensor 200 is acquired (YES in step S109), the process proceeds to step S111.
Then, in step S111, the determination unit 144 acquires the moving image of the user's right hand HP generated by the camera 61. The moving image of the user's right hand HP indicates an operation of selecting one of the input images from the plurality of input images MN. Therefore, in this embodiment, it may be described as an “operation image”.
Next, in step S113, the operation estimation unit 143 estimates the operation of selecting one input image MA from the plurality of input images MN based on the detection value of the myoelectric sensor 200.
Next, in step S115, the determination unit 144 determines whether the estimation result of the operation estimation unit 143 is correct, based on the moving image of the user's right hand HP generated by the camera 61.
That is, when the determination unit 144 determines that the operation of selecting one input image MA from the plurality of input images MN is performed, the determination unit 144 determines that the estimation result of the operation estimation unit 143 is correct. To do. Further, when the determination unit 144 determines that the operation of selecting one input image MA from the plurality of input images MN is not performed, the estimation result of the operation estimation unit 143 is incorrect. 144 determines.
Next, in step S117, the learning unit 145 executes the "machine learning process", and the process ends. In the “machine learning process”, it is determined whether or not the learning unit 145 indicates an operation in which the detection value of the myoelectric sensor 200 selects one input image MA from the plurality of input images MN based on the learning data set 147a. A process for generating the determination criterion will be described.

  Note that step S107 corresponds to an example of “display control step”. Step S113 corresponds to an example of “operation estimation step”. Step S115 corresponds to an example of a "determination step". Step S117 corresponds to an example of a "learning step".

FIG. 10 is a flowchart showing the machine learning process of the control unit 141.
As shown in FIG. 10, in step S201, the learning unit 145 classifies the learning data set 147a for each input image MA included in the plurality of input images MN and stores the learning data set 147a in the learning data storage unit 147. The learning data set 147a includes the detection value of the myoelectric sensor 200, one input image MA, and the determination result of the determination unit 144.
That is, in the learning data storage unit 147, the learning data set 147a is classified and stored for each input image MA.
Next, in step S203, the learning unit 145 determines whether the determination unit 144 determines that the estimation result of the operation estimation unit 143 is correct.
When the determination unit 144 determines that the estimation result of the operation estimation unit 143 is incorrect (NO in step S203), the process proceeds to step S211. If the determination unit 144 determines that the estimation result of the operation estimation unit 143 is correct (YES in step S203), the process proceeds to step S205.

Then, in step S205, the learning unit 145 extracts the first feature amount from the detection value of the myoelectric sensor 200. The first feature amount indicates the feature of the detection value of the myoelectric sensor 200 when the determination unit 144 determines that the estimation result of the operation estimation unit 143 is correct.
For example, the first feature amount includes either the frequency or the amplitude of the detection value of the myoelectric sensor 200. Specifically, the first feature amount includes the maximum value of frequency, the average value of frequency, the minimum value of frequency, the maximum value of amplitude, the average value of amplitude, and the minimum value of amplitude in the detection value of the myoelectric sensor 200. At least one is included.
Further, the first feature amount may indicate, for example, a feature of change over time in the detection value of the myoelectric sensor 200.
Next, in step S207, the learning unit 145 stores the first feature amount extracted in step S205 in the learning data storage unit 147. Specifically, the learning unit 145 associates the first feature amount with the learning data storage unit 147 by associating it with the one input image MA selected by the operation estimated by the operation estimating unit 143 in step S113 of FIG. Remember.
Next, in step S209, the learning unit 145 totalizes the first characteristic amount, and the process proceeds to step S217.
Specifically, the learning data storage unit 147 aggregates a plurality of first feature amounts stored in association with one input image MA. For example, the learning unit 145 calculates at least one of the average value, standard deviation, maximum value, and minimum value of the first feature amount.

In the case of NO in step S203, the learning unit 145 extracts the second feature amount from the detection value of the myoelectric sensor 200 in step S211. The second feature amount indicates the feature of the detection value of the myoelectric sensor 200 when the determination unit 144 determines that the estimation result of the operation estimation unit 143 is incorrect.
For example, the second feature amount includes either the frequency or the amplitude of the detection value of the myoelectric sensor 200. Specifically, the second feature amount includes the maximum value of frequency, the average value of frequency, the minimum value of frequency, the maximum value of amplitude, the average value of amplitude, and the minimum value of amplitude in the detected value of the myoelectric sensor 200. At least one is included.
In addition, the second feature amount may indicate, for example, a feature of change over time in the detection value of the myoelectric sensor 200.
Next, in step S213, the learning unit 145 stores the second feature amount extracted in step S211 in the learning data storage unit 147. Specifically, the learning unit 145 associates the second feature amount with the learning data storage unit 147 by associating it with the one input image MA selected by the operation estimated by the operation estimating unit 143 in step S113 of FIG. Remember.
Next, in step S215, the learning unit 145 totalizes the second feature amount, and the process proceeds to step S217.
Specifically, the learning data storage unit 147 aggregates a plurality of second feature amounts stored in association with one input image MA. For example, the learning unit 145 calculates at least one of the average value, standard deviation, maximum value, and minimum value of the second feature amount.

Then, in step S217, the learning unit 145 generates a criterion for determining whether or not the detection value of the myoelectric sensor 200 indicates an operation of selecting one input image MA from the plurality of input images MN, The process ends.
For example, when the minimum value of the first characteristic amount is larger than the maximum value of the second characteristic amount, the learning unit 145 generates the determination criterion indicating that the value of the characteristic amount is equal to or larger than the minimum value of the first characteristic amount. To do.
Further, for example, when the maximum value of the first feature amount is smaller than the minimum value of the second feature amount, the learning unit 145 determines that the value of the feature amount is equal to or less than the maximum value of the first feature amount. To generate.

FIG. 11 is a flowchart showing the processing of the control unit 141 after the learning processing.
Since each of steps S301 to S311 shown in FIG. 11 is the same as each of steps S101 to S111 shown in FIG. 8, description thereof will be omitted.
In step S313, the learning unit 145 extracts the feature amount from the detection value of the myoelectric sensor 200. The feature amount corresponds to the first feature amount and the second feature amount.
Next, in step S315, the operation estimation unit 143 estimates the operation of selecting one input image MA from the plurality of input images MN based on the detection value of the myoelectric sensor 200 and the determination criterion. To do. Then, a process progresses to step S317.
Specifically, the operation estimation unit 143 estimates the operation of selecting one input image MA from the plurality of input images MN based on the feature amount extracted in step S313 and the determination criterion. ..
Since each of steps S317 to S319 shown in FIG. 11 is the same as each of steps S115 to S117 shown in FIG. 8, description thereof will be omitted.

FIG. 12 is a diagram showing another form of the input image MN. Similar to FIG. 7, FIG. 12 shows the visual field region AR1 and the image region AR2.
The image area AR2 includes the right hand HP of the user. A myoelectric sensor 200 is attached to the wrist position PS1 of the user's right hand HP.

As described with reference to FIG. 8, the HMD 100 accepts an input of a numeral, whereas in FIG. 12, the HMD 100 accepts an input of an alphabet or the like, which is a difference.
Each of the plurality of input images MN indicates different characters. For example, each of the plurality of input images MN includes alphabets "A" to "Z", an "@" mark, a backspace command, a return command, a character type change command, and an enter command.
The backspace command indicates a command to delete one input character. The return command indicates a command to cancel one operation performed immediately before. The character type change command indicates a command to change the type of input character. The enter command is a command for confirming the input operation.
The plurality of input images MN are associated with the area delimited by the joint in the finger of the user's right hand HP.
More specifically, the plurality of input images MN are associated with the regions delimited by joints in each finger of the user's right hand HP except the thumb FP1.

The user's right hand HP includes a thumb FP1, an index finger FP2, a middle finger FP3, a ring finger FP4, and a little finger FP5.
The index finger FP2 includes a first joint FP21, a second joint FP22, and a third joint FP23. The middle finger FP3 includes a first joint FP31, a second joint FP32, and a third joint FP33. The ring finger FP4 includes a first joint FP41, a second joint FP42, and a third joint FP43. The little finger FP5 includes a first joint FP51, a second joint FP52, and a third joint FP53. Reference numerals corresponding to these joints are omitted in the drawing.

The two or more predetermined number of input images included in the plurality of input images MN are associated with the areas delimited by the joints of the fingers of the user's right hand HP except the thumb.
For example, in the index finger FP2, three input images L21 are displayed in the region on the distal end side of the first joint FP21. The three input images L21 show the alphabet “D”, the alphabet “E”, and the alphabet “F”. In this case, the predetermined number is “3”.
Further, for example, in the index finger FP2, three input images L22 are displayed in a region between the first joint FP21 and the second joint FP22. The three input images L22 show the alphabet “M”, the alphabet “N”, and the alphabet “O”. In this case, the predetermined number is “3”.
Further, for example, in the index finger FP2, four input images L23 are displayed in the region between the second joint FP22 and the third joint FP23. The four input images L23 show the alphabet "W", the alphabet "X", the alphabet "Y", and the alphabet "Z". In this case, the predetermined number is "4".

The operation estimation unit 143 estimates the operation of selecting one input image MA from the plurality of input images MN based on the detection value of the myoelectric sensor 200.
Specifically, the operation estimation unit 143 selects one of the areas selected by the joint based on the detection value of the myoelectric sensor 200, and one of the predetermined number of input images MB associated with the area. The operation of selecting the input image MA is estimated.
First, the operation of selecting the area delimited by the joint is an operation of pressing the thumb FP1 of the user's right hand HP to the area delimited by the joint in each of the index finger FP2, the middle finger FP3, the ring finger FP4, and the little finger FP5. .. A predetermined number of input images MB are selected from the plurality of input images MN by the operation of selecting the area delimited by the joints.
Next, the operation of selecting one input image MA from the predetermined number of input images MB associated with the area is an operation of moving the thumb FP1 of the user's right hand HP in a predetermined direction.

Further, more specifically, for example, when the area between the second joint FP22 and the third joint FP23 is selected on the index finger FP2, one input image MA is selected as follows. It
That is, when selecting the alphabet “W”, the user performs an operation of moving the thumb FP1 of the right hand HP in the direction toward the tip of the index finger FP2 (upward). When selecting the alphabet “X”, the user performs an operation of moving the thumb FP1 of the right hand HP in a direction away from the middle finger FP3 (right direction). When selecting the alphabet “Y”, the user performs an operation of moving the thumb FP1 of the right hand HP in a direction (downward) toward the base end of the index finger FP2. When selecting the alphabet "Z", the user performs an operation of moving the thumb FP1 of the right hand HP in the direction toward the middle finger FP3 (to the left).

  In this way, the operation estimation unit 143 performs an operation of selecting an area demarcated by the joint based on the detection value of the myoelectric sensor 200, and one from the predetermined number of input images MB associated with the area. The operation of selecting the input image MA is estimated. Therefore, the user can input the alphabet.

As described above with reference to FIGS. 1 to 12, in the present embodiment, the operation estimation unit 143, based on the detection value of the myoelectric sensor 200 that is an example of the motion sensor, the plurality of input images MN. The operation of selecting one input image MA from among the above is estimated. The determination unit 144 determines the estimation result of the operation estimation unit 143. The learning unit 145 machine-learns the conditions for specifying one input image MA from the detection value of the myoelectric sensor 200 based on the learning data set 147a. The condition for specifying one input image MA indicates, for example, a criterion for determining the feature amount of the detection value of the myoelectric sensor 200 for specifying one input image MA. The learning data set 147a includes a combination of the detection value of the myoelectric sensor 200, one input image MA, and the determination result of the determination unit 144.
Therefore, since the condition for specifying one input image MA is machine-learned from the detected value of the myoelectric sensor 200, the accuracy with which the operation estimation unit 143 estimates the operation of selecting one input image MA can be improved. .. Therefore, the operability of the HMD 100 can be improved.

Further, the camera 61 is provided, and the determination unit 144 determines whether the estimation result of the operation estimation unit 143 is correct based on the image captured by the camera 61.
Therefore, the camera 61 can generate an image of the right hand HP of the user. Therefore, the determination unit 144 can accurately determine whether or not the operation by the movement of the user's right hand HP is the operation of selecting one input image MA.

Further, each of the plurality of input images MN indicates different characters. The operation estimation unit 143 estimates a character to be input based on the detection value of the myoelectric sensor 200.
Therefore, the user can easily perform the operation of inputting characters. Therefore, the operability of the HMD 100 can be further improved.

Each of the plurality of input images MN is associated with a region separated by a joint in each finger of the user's right hand HP except the thumb FP1. The operation of selecting one input image MA is an operation of pressing the thumb FP1 of the user's right hand HP to a region delimited by joints.
Therefore, the myoelectric sensor 200 can accurately detect the operation of selecting one input image MA. Further, the user can perform an operation of selecting one input image MA using only the right hand HP. Therefore, the operability of the HMD 100 can be further improved.

The two or more predetermined number of input images MB included in the plurality of input images MN are associated with the regions separated by the joints in each finger of the user's right hand HP except the thumb FP1. The operation of selecting an area is an operation of pressing the thumb FP1 of the user's right hand HP to an area delimited by joints. The operation of selecting one input image MA from the predetermined number of input images MB associated with the area is an operation of moving the thumb FP1 of the user's right hand HP in a predetermined direction.
Therefore, the myoelectric sensor 200 can accurately detect the operation of selecting one input image MA. Further, the user can perform an operation of selecting one input image MA from a large number of input images MN such as alphabets by using only the right hand HP. Therefore, the operability of the HMD 100 can be further improved.

Further, the display control unit 142 causes the image display unit 20 to display the first guide image GL1 that guides the position of the right hand HP of the user.
Therefore, for example, by displaying the second guide image GL2 for guiding the position of the myoelectric sensor 200 on the image display unit 20, the user can mount the myoelectric sensor 200 at an appropriate position. Therefore, the operation of selecting one input image MA can be detected more accurately. As a result, the operability of the HMD 100 can be further improved.

The display control unit 142 also tracks the position of the user's hand visually recognized through the image display unit 20, and adjusts the display position of each of the plurality of input images MN according to the movement of the user's hand. To do.
Therefore, even when the user's hand moves, each of the plurality of input images MN can be displayed at an appropriate position. Therefore, the operability of the HMD 100 can be further improved.

In addition, the motion sensor includes a myoelectric sensor 200 worn on the user's wrist.
Therefore, the operation of selecting one input image MA can be detected with a simple configuration.

  In the present embodiment, the display control unit 142 displays the input image M at a position overlapping the user's right hand HP, but the present invention is not limited to this. The display control unit 142 may display the plurality of input images MN at the position overlapping the user's hand. For example, the display control unit 142 may display the plurality of input images MN at a position overlapping the user's left hand. Further, for example, the display control unit 142 may display a plurality of input images MN at positions overlapping both hands of the user. In this case, as compared with the case where the input image MN is displayed at a position overlapping the right hand HP of the user, a large number of input images MN can be displayed.

  Further, in the present embodiment, a case has been described in which each of the plurality of input images MN indicates a number, but the present invention is not limited to this. Each of the plurality of input images MN only needs to show a symbol. Here, the "symbol" includes characters, mathematical symbols, graphic symbols, and the like. The characters include alphabets, hiragana, and katakana. For example, each of the plurality of input images MN may indicate one character of the alphabet. In this case, the text can be input.

Further, in the present embodiment, the case where the motion sensor is the myoelectric sensor 200 worn on the wrist of the user's right hand HP has been described, but the present invention is not limited to this. The motion sensor may be a gyro sensor attached to the finger of the user's right hand HP. The finger of the right hand HP is, for example, the thumb FP1. Further, the motion sensor may include a myoelectric sensor 200 worn on the wrist and a gyro sensor worn on the user's finger.
Further, for example, the sensor worn on the finger may be a triaxial acceleration sensor or a 6-axis sensor. The 6-axis sensor is a motion sensor including a 3-axis acceleration sensor and a 3-axis gyro sensor. The 6-axis sensor may employ an IMU in which the above sensor is modularized.
Further, for example, the gyro sensor may be attached to a finger other than the thumb FP1. Specifically, the gyro sensor may be attached to the index finger FP2, the middle finger FP3, the ring finger FP4, and the little finger FP5 of the right hand HP.

  Further, in the present embodiment, the determination unit 144 determines whether the estimation result of the operation estimation unit 143 is correct, but the present invention is not limited to this. The determination unit 144 may determine the estimation result of the operation estimation unit 143. For example, when the estimation result of the operation estimation unit 143 is incorrect, the determination unit 144 may determine the degree to which the estimation result of the operation estimation unit 143 is incorrect. For example, when the operation estimating unit 143 estimates the operation of selecting one input image MA, and the determining unit 144 determines that the operation of selecting one input image MB different from the input image MA is performed. The determination unit 144 may execute the following processing. That is, the determination unit 144 may determine the degree to which the estimation result of the estimation unit 143 is incorrect based on the distance between the input image MA and the input image MB.

Further, in the present embodiment, the condition for specifying one input image MA indicates the criterion for the feature amount of the detection value of the myoelectric sensor 200 for specifying one input image MA. Not limited to. The condition for specifying one input image MA may be a condition related to the detection value of the myoelectric sensor 200 for specifying one input image MA.
For example, the condition for specifying one input image MA may be the range of the characteristic amount of the detection value of the myoelectric sensor 200 for specifying one input image MA. The feature amount of the detected value of the myoelectric sensor 200 includes either the frequency or the amplitude of the detected value of the myoelectric sensor 200. Specifically, the feature amount is at least 1 of the maximum value of the frequency, the average value of the frequency, the minimum value of the frequency, the maximum value of the amplitude, the average value of the amplitude, and the minimum value of the amplitude in the detected value of myoelectric sensor 200. Including one.

Note that the configuration is not limited to the configuration of the present embodiment, and can be implemented in various modes without departing from the gist thereof.
For example, in the present embodiment, the configuration in which the control device 10 is connected to the image display unit 20 by wire is illustrated, but the configuration is not limited to this, and the image display unit 20 is configured to be wirelessly connected to the control device 10. You may. The wireless communication method in this case may be the method exemplified as the communication method supported by the communication unit 117, or may be another communication method.

  Further, some functions of the control device 10 may be provided in the image display unit 20, or the control device 10 may be realized by a plurality of devices. For example, instead of the control device 10, a wearable device that can be attached to the user's body, clothing, or clothing worn by the user may be used. In this case, the wearable device may be, for example, a watch type device, a ring type device, a laser pointer, a mouse, an air mouse, a game controller, a pen type device, or the like.

  Furthermore, in the present embodiment, the configuration in which the image display unit 20 and the control device 10 are separated and connected via the connection cable 40 has been described as an example. The present invention is not limited to this, and the control device 10 and the image display unit 20 may be integrally configured and mounted on the user's head.

  Further, in the present embodiment, the configuration in which the user sees the outside scene through the display unit is not limited to the configuration in which the right light guide plate 26 and the left light guide plate 28 transmit the outside light. For example, it can be applied to a display device that displays an image in a state where the outside scene cannot be visually recognized. Specifically, it can be applied to a display device that displays a captured image of the camera 61, an image or CG generated based on this captured image, video data stored in advance, video based on video data input from the outside, or the like. .. This type of display device may include a so-called closed type display device in which the outside scene cannot be visually recognized. For example, if the image display unit 20 displays a composite image in which the image of the outside scene captured by the camera 61 and the display image are combined together, the user can see the outside scene even if the image display unit 20 does not transmit external light. And the image can be visually displayed. Of course, it is also possible to apply to such a so-called video see-through type display device.

  Further, for example, instead of the image display unit 20, an image display unit of another system such as an image display unit worn like a hat may be adopted, and an image is displayed corresponding to the left eye LE of the user. It suffices to include a display unit that displays the image and a display unit that displays an image corresponding to the right eye RE of the user. Further, the display device may be configured as an HMD mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as an HMD built in a body protector such as a helmet. In this case, the part that positions the position with respect to the body of the user and the part that is positioned with respect to the part can be the mounting part.

Further, as an optical system for guiding the image light to the eyes of the user, a configuration in which a virtual image is formed by the half mirrors 261 and 281 on a part of the right light guide plate 26 and the left light guide plate 28 has been illustrated. The present invention is not limited to this, and the image may be displayed in a display area having an area that occupies the entire surface or most of the right light guide plate 26 and the left light guide plate 28. In this case, the operation of changing the display position of the image may include a process of reducing the image.
Further, the optical element is not limited to the right light guide plate 26 and the left light guide plate 28 having the half mirrors 261, 281 and may be any optical component that allows image light to enter the user's eye. , A prism, or a holographic display section may be used.

  Further, at least some of the functional blocks shown in FIGS. 5 and 6 may be realized by hardware, or may be realized by the cooperation of hardware and software. It is not limited to a configuration in which independent hardware resources are arranged as described above. The program executed by the control unit 141 may be stored in the non-volatile storage unit 121 or another storage device (not shown) in the control device 10. Alternatively, the program stored in an external device may be acquired via the communication unit 117 or the external connector 184 and executed. Further, among the components formed on the control device 10, the operation unit 110 may be formed as a user interface (UI). Moreover, the configuration formed in the control device 10 may be duplicated in the image display unit 20. For example, a processor similar to the main processor 140 may be arranged in the image display unit 20, or the main processor 140 included in the control device 10 and the processor of the image display unit 20 may be configured to execute separate functions. Good.

  10 ... Control device, 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display unit, 23 ... Left holding part, 24 ... Left display unit, 26 ... Right light guide plate, 27 ... Front frame , 28 ... Left light guide plate, 30 ... Headset, 32 ... Right earphone, 34 ... Left earphone, 40 ... Connection cable, 61 ... Camera (imaging unit), 63 ... Microphone (part of voice input unit), 100 ... HMD (Display device), 110 ... Operation unit, 117 ... Communication unit, 118 ... Memory, 120 ... Controller board, 121 ... Nonvolatile storage unit, 130 ... Power supply unit, 140 ... Main processor, 141 ... Control unit, 142 ... Display control Section, 143 ... operation estimation section, 144 ... determination section, 145 ... learning section, 146 ... signal storage section, 147 ... learning data storage section, 147a ... learning data set, 148 ... image storage section, 18 ... voice codec (part of voice input unit), 182 ... voice interface (part of voice input unit), 200 ... myoelectric sensor (motion sensor), AR1 ... field of view area, AR2 ... image area, FP1 ... thumb, FP11 ... 1st joint, FP12 ... 2nd joint, FP2 ... Index finger, FP21 ... 1st joint, FP22 ... 2nd joint, FP23 ... 3rd joint, FP3 ... Middle finger, FP31 ... 1st joint, FP32 ... 2nd joint, FP33 ... 3rd joint, FP4 ... ring finger, FP41 ... 1st joint, FP42 ... 2nd joint, FP43 ... 3rd joint, FP5 ... little finger, FP51 ... 1st joint, FP52 ... 2nd joint, FP53 ... 3rd joint, GL ... guide image, GL1 ... first guide image, GL2 ... second guide image, HP ... right hand, MA ... input image, MN ... input image.

Claims (10)

A transmissive display unit that transmits light, which is mounted on the user's head,
A display control unit that causes the display unit to display a plurality of input images at a position overlapping the user's hand that is visible through the display unit;
A motion sensor for detecting movement of the user's hand,
An operation estimation unit that estimates an operation of selecting one input image from the plurality of input images based on a detection value of the motion sensor;
A determination unit that determines the estimation result of the operation estimation unit,
The one input image is identified from the detection value of the motion sensor based on a learning data set including a combination of the detection value of the motion sensor, the one input image, and the determination result of the determination unit. A learning unit that machine-learns the conditions for
A display device including. Equipped with an imaging unit,
The determination unit determines whether the estimation result of the operation estimation unit is correct based on an image captured by the imaging unit,
The display device according to claim 1. A voice input unit for receiving voice input from the user,
The determination unit determines whether the estimation result of the operation estimation unit is correct based on the voice input to the voice input unit,
The display device according to claim 1. Each of the plurality of input images indicates different characters,
The operation estimation unit estimates a character to be input based on a detection value of the motion sensor,
The display device according to claim 1. Each of the plurality of input images is associated with an area separated by a joint in each finger of the user's hand excluding the thumb,
The operation of selecting the one input image is an operation of pressing the thumb of the user's hand to a region delimited by the joint.
The display device according to claim 1. A predetermined number of two or more input images included in the plurality of input images are associated with a region separated by a joint in each finger of the user excluding the thumb of the hand,
The operation of selecting the region is an operation of pressing the thumb of the user's hand to a region delimited by the joint,
The operation of selecting the one input image from the predetermined number of input images associated with the area is an operation of moving the thumb of the user's hand in a predetermined direction.
The display device according to claim 1. The display control unit causes the display unit to display a first guide image for guiding the position of the user's hand.
The display device according to claim 1. The display control unit tracks the position of the user's hand visually recognized through the display unit, and adjusts the display position of each of the plurality of input images according to the movement of the user's hand. ,
The display device according to claim 1. The motion sensor includes a myoelectric sensor worn on the user's wrist,
The display device according to claim 1. A method of controlling a display device, comprising a transmissive display unit that transmits light, the display unit being mounted on a user's head, and a motion sensor that detects a motion of the user's hand,
A display control step of displaying a plurality of input images on the display unit at a position overlapping with the user's hand visually recognized through the display unit;
An operation estimation step of estimating an operation of selecting one input image from the plurality of input images based on the detection value of the motion sensor;
A determination step of determining the estimation result of the operation estimation step,
The one input image is specified from the detection value of the motion sensor based on a learning data set including a combination of the detection value of the motion sensor, the one input image, and the determination result of the determination step. A learning step of machine learning the conditions for
A method for controlling a display device, including:

Images