JP2017016599A - Display device, display device control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To readily allow a person who acts for a place or a thing as an object to see information on movement.SOLUTION: An HMD 100 comprises an image display unit 20 for displaying an image in a visually recognizable manner by letting an outer scene pass through. The HMD 100 comprises: a position detection unit 162 for recognizing input; and a control unit 140 for having the image display unit 20 display items, and changes a display on the image display unit 20 according to the input recognized by the position detection unit 162.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a display device, a display device control method, and a program.

  Conventionally, a method for supporting execution of a work procedure according to a scenario including information on the work procedure is known (for example, see Patent Document 1). In the method described in Patent Document 1, a work list or the like is displayed according to a scenario including an action sequence related to details of a work procedure, and the execution of the work procedure is supported. The execution status is displayed on the screen.

JP 2013-29985 A

In the method described in Patent Document 1, a worker who performs work performs work by looking at a screen on which a work list or the like is displayed. In this case, the worker has to look at both the place and the target object for the work and the screen. As described above, in order to show information related to the operation to a person who operates with respect to the place or object, the target person (for example, the worker) You must move your line of sight to see what information you want to see. Therefore, there has been a demand for a method that can easily show information related to an operation when operating on a place or an object.
The present invention has been made in view of the above circumstances, and an object of the present invention is to make it possible to easily show information regarding an operation to a person who operates on a place or an object.

In order to achieve the above object, a display device according to the present invention includes a display unit that displays an image so as to be visible through an outside scene, an input unit that recognizes input, and displays items on the display unit. And a control unit that changes the display on the display unit according to the input recognized by the unit.
According to the present invention, a person viewing a display device can change the display of an item by viewing the displayed item and performing an input that can be recognized by the input unit. Thereby, for example, a person who operates with respect to a place or an object can see the items on the display unit, and the place or object that is the target of the operation can be seen through the display unit. As described above, the object and the item existing in the real space which is the outside scene can be seen and the display can be changed without performing a movement with a large load such as a large movement of the line of sight.

Further, the present invention is characterized in that, in the display device, the control unit displays the item on the display unit based on item data including the item.
According to the present invention, various numbers and various types of items can be displayed.

In the display device according to the present invention, the item data includes a plurality of the items corresponding to operations, and the order of the items corresponding to the execution order of the operations is set in each of the items. It is characterized by.
According to the present invention, it is possible to notify the viewer of the display device of the execution order of the operations according to the display mode of the items.

Further, in the display device according to the present invention, the control unit arranges a plurality of items in a list format to be displayed on the display unit, and the items are arranged according to an order set in the items. It is characterized by.
According to the present invention, it is possible to notify the viewer of the display device of the execution order of operations by arranging a plurality of items displayed.

Further, the present invention is characterized in that, in the display device, the control unit sequentially displays the items in the order set for the items.
According to the present invention, it is possible to notify the viewer of the display device of the execution order of operations according to the display order of items.

In the display device according to the present invention, the control unit causes the display unit to display an image based on order data including information related to an execution order of operations corresponding to the items included in the item data. It is characterized by.
According to the present invention, it is possible to inform the viewer of the display device of the execution order of the operations with an image.

The display device may further include an order data storage unit that stores the item data and the order data in association with each other.
According to the present invention, it is possible to easily read and use data relating to items relating to operations and data relating to the execution order of operations.

In the display device according to the aspect of the invention, the input unit may detect an input in a range that is visible through the display unit and specify an input position, and the control unit may be specified by the input unit. The display on the display unit is changed corresponding to the input position.
ADVANTAGE OF THE INVENTION According to this invention, when the input which a input part can recognize is performed, a display can be changed corresponding to an input position.

Further, the present invention is characterized in that, in the display device, the control unit displays an index on the item being displayed at a position corresponding to the input position specified by the input unit.
According to the present invention, it is possible to change the display mode of displayed items by performing input.

The display device may further include a state detection unit that detects an execution state of an operation corresponding to the item based on a captured image of the outside scene or a sound collected from the outside scene, and the control unit includes: The display state of the item being displayed by the display unit is changed according to the execution state detected by the state detection unit.
According to the present invention, the execution state of an operation can be detected, and the display state can be changed corresponding to the detected execution state.

The display device may further include an execution data storage unit that stores execution data including information related to an execution state of the operation, and the control unit operates an operation corresponding to the item included in the item data. The execution data is stored in the execution data storage unit in association with the item data, and when the execution data is read from the execution data storage unit, the display unit displays the execution data based on the read execution data. The display state of the item is controlled.
According to the present invention, data relating to the execution state of the operation is stored, and by reading this data and reflecting it in the display, it is possible to record the execution state and call the recorded execution state.

The display device may further include a target detection unit that detects a target in the outside scene, and the control unit may visually recognize the target detected by the target detection unit through the display unit. The item included in the item data is displayed in correspondence with the position.
According to the present invention, it is possible to easily visually recognize both a place or an object and an item to be operated.

According to the present invention, the display device includes a communication unit that communicates with an external device, and the item data is received and acquired from the external device by the communication unit.
According to the present invention, data relating to items can be acquired from the outside.

In the display device according to the present invention, the communication unit performs short-range wireless communication.
According to the present invention, data related to items can be acquired from the outside by short-range wireless communication. In addition, since the distance at which short-range wireless communication can be performed is limited, data associated with a place can be acquired.

In order to achieve the above object, the present invention provides a control method for a display device having a display unit that displays an image so as to be visible through the outside scene, and obtains data of a list including a plurality of items. A position corresponding to the specified input position by detecting an input in a range that is visible through the display unit, specifying an input position, and displaying the item on the display unit based on item data including the item. The display state of the item being displayed is changed.
According to the present invention, a person viewing a display device can change the display of an item by viewing the displayed item and performing an input that can be recognized by the input unit. Thereby, for example, a person who operates with respect to a place or an object can see the items on the display unit, and the place or object that is the target of the operation can be seen through the display unit. As described above, the object and the item existing in the real space which is the outside scene can be seen and the display can be changed without performing a movement with a large load such as a large movement of the line of sight.

  In addition, the present invention can be configured as a program that can be executed by a computer that controls a display device having a display unit that displays an image so as to be visible through the outside scene. The program detects an input in a range that is visible through the display unit by the computer, specifies an input position, displays the item on the display unit based on item data including the item, and specifies This program changes the display state of the item being displayed at a position corresponding to the input position. Moreover, you may comprise this invention as a storage medium which memorize | stored this program.

Explanatory drawing which shows the external appearance structure of HMD of 1st Embodiment. The figure which shows the structure of the optical system of an image display part. The figure which shows the principal part structure of an image display part. The functional block diagram of each part which comprises HMD. It is a schematic diagram which shows the structural example of the data which a memory | storage part memorize | stores, (A) shows the structural example of scenario data and historical data, (B) shows another structural example of scenario data and historical data. It is explanatory drawing of the input method of HMD, (A) shows the example of input which uses a virtual keyboard, (B) shows the example of a position input. The figure which shows the example of a display of HMD. The flowchart which shows operation | movement of HMD of 1st Embodiment. The flowchart which shows operation | movement of HMD of 2nd Embodiment.

[First Embodiment]
FIG. 1 is an explanatory diagram showing an external configuration of an HMD (Head Mounted Display) 100 according to an embodiment to which the present invention is applied.
The HMD 100 is a display device that includes an image display unit 20 that allows a user to visually recognize a virtual image while being mounted on the user's head, and a control device 10 that controls the image display unit 20. The control device 10 also functions as a controller for the user to operate the HMD 100.

  The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a right camera. 61 (imaging part), left camera 62 (imaging part), and microphone 63 are provided. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. The right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as “display drive unit”, and the right optical image display unit 26 and the left optical image display unit 28 are simply referred to as “optical image display unit”. Also called.

The display driving units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 described later with reference to FIGS.
The right optical image display unit 26 and the left optical image display unit 28 include light guide plates 261 and 262 (FIG. 2) and a light control plate 20A. The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. The light control plate 20A is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. As the light control plate 20A, various materials such as a material that has almost no light transmission, a material that is nearly transparent, a material that attenuates the amount of light and transmits light, and a material that attenuates or reflects light of a specific wavelength may be used. it can. By appropriately selecting the optical characteristics (light transmittance, etc.) of the light control plate 20A, the external light amount incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to visually recognize the virtual image. You can adjust the ease. In the present embodiment, a case will be described in which at least a light control plate 20A having a light transmittance that allows a user wearing the HMD 100 to visually recognize an outside scene is described. The light control plate 20A protects the right light guide plate 261 and the left light guide plate 262, and suppresses damage to the right light guide plate 261 and the left light guide plate 262, adhesion of dirt, and the like.
The dimming plate 20A may be detachable from the right optical image display unit 26 and the left optical image display unit 28, or may be mounted by replacing a plurality of types of dimming plates 20A, or may be omitted. .

The right camera 61 is disposed at the end on the right holding unit 21 side on the front surface of the HMD 100. Further, the left camera 62 is disposed at the end on the left holding unit 23 side on the front surface of the HMD 100. The right camera 61 and the left camera 62 are digital cameras including an imaging element such as a CCD or a CMOS and an imaging lens, and may be a monocular camera or a stereo camera.
The right camera 61 and the left camera 62 capture at least a part of the outside scene in the front side direction of the HMD 100, in other words, in the user's viewing direction when the HMD 100 is worn. In another expression, at least one of the right camera 61 and the left camera 62 captures a range or direction that overlaps the user's field of view. More specifically, at least one of the right camera 61 and the left camera 62 images the direction in which the user gazes. The width of the angle of view of the right camera 61 and the left camera 62 can be set as appropriate, but in the present embodiment, as will be described later, the user visually recognizes through the right optical image display unit 26 and the left optical image display unit 28. An angle of view including the outside world. Furthermore, it is more preferable that the imaging ranges of the right camera 61 and the left camera 62 are set so that the entire field of view of the user through the light control plate 20A can be imaged.

  The right camera 61 and the left camera 62 execute imaging according to the control of the imaging control unit 161 (FIG. 4) included in the control unit 140, and output captured image data to the imaging control unit 161.

Here, the HMD 100 may include a distance sensor (not shown) that detects a distance to a measurement object positioned in a preset measurement direction. The distance sensor is disposed, for example, at the boundary between the right optical image display unit 26 and the left optical image display unit 28. In this case, when the user wears the image display unit 20, the position of the distance sensor 64 is substantially in the middle of both eyes of the user in the horizontal direction and above the eyes of the user in the vertical direction. is there. The measurement direction of the distance sensor 64 can be, for example, a direction overlapping the imaging direction of the right camera 61 and the left camera 62 in the front side direction of the HMD 100.
The distance sensor can be configured to include, for example, a light source such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured. The distance sensor may perform a triangulation process or a distance measurement process based on a time difference according to the control of the control unit 140. The distance sensor may be configured to include a sound source that emits ultrasonic waves and a detection unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance sensor may perform the distance measurement process based on the time difference until the reflection of the ultrasonic wave according to the control of the control unit 140.

FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. FIG. 2 shows the user's left eye LE and right eye RE for explanation.
The left display driving unit 24 includes a left backlight 222 having a light source such as an LED and a diffusion plate, a transmissive left LCD 242 disposed on an optical path of light emitted from the diffusion plate of the left backlight 222, and a left LCD 242. A left projection optical system 252 including a lens group that guides the image light L that has passed through the lens. The left LCD 242 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

The left projection optical system 252 includes a collimator lens that converts the image light L emitted from the left LCD 242 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimator lens is incident on the left light guide plate 262. The left light guide plate 262 is a prism formed with a plurality of reflecting surfaces that reflect the image light L, and the image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 262. The left light guide plate 262 is formed with a half mirror 262A (reflection surface) positioned in front of the left eye LE.
The image light L reflected by the half mirror 262A is emitted from the left optical image display unit 28 toward the left eye LE, and this image light L forms an image on the retina of the left eye LE so that the user can visually recognize the image.

  The right display driving unit 22 is configured to be symmetrical with the left display driving unit 24. The right display driving unit 22 includes a right backlight 221 having a light source such as an LED and a diffusion plate, a transmissive right LCD 241 disposed on an optical path of light emitted from the diffusion plate of the right backlight 221, and a right LCD 241. A right projection optical system 251 including a lens group that guides the image light L that has passed through the lens. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

The right projection optical system 251 includes a collimator lens that converts the image light L emitted from the right LCD 241 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimator lens is incident on the right light guide plate 261. The right light guide plate 261 is a prism formed with a plurality of reflecting surfaces that reflect the image light L, and the image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 261. The right light guide plate 261 is formed with a half mirror 261A (reflection surface) located in front of the right eye RE.
The image light L reflected by the half mirror 261A is emitted from the right optical image display unit 26 toward the right eye RE, and this image light L forms an image on the retina of the right eye RE to make the user visually recognize the image.

  The image light L reflected by the half mirror 261A and the external light OL transmitted through the light control plate 20A are incident on the right eye RE of the user. The image light L reflected by the half mirror 262A and the external light OL transmitted through the light control plate 20A are incident on the left eye LE. In this way, the HMD 100 superimposes the image light L of the internally processed image and the external light OL on the user's eyes and allows the user to see the external scene through the light control plate 20A. The image by the image light L is visually recognized. As described above, the HMD 100 functions as a see-through display device.

  The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “left light guide unit”, and the right projection optical system 251 and the right light guide plate 261 are collectively referred to as “right light guide unit”. The configuration of the right light guide and the left light guide is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light. It may be used, or a semi-transmissive reflective film may be used.

  The image display unit 20 is connected to the control device 10 via the connection unit 40. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control device 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the housing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and is connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collecting unit of the microphone 63 faces the user's line of sight, collects sound, and sends the sound signal to the sound processing unit 187 (FIG. 4). Output. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

  The right cord 42, the left cord 44, and the main body cord 48 only need to be capable of transmitting digital data, and can be constituted by, for example, a metal cable or an optical fiber. Further, the right cord 42 and the left cord 44 may be combined into a single cord.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. The end of the main body cord 48 opposite to the connecting member 46 and the control device 10 are provided with connectors (not shown) that fit together. By fitting the connector of the main body cord 48 and the connector of the control device 10 or removing this fitting, the control device 10 and the image display unit 20 can be contacted and separated.

  The control device 10 controls the HMD 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 that is operated by a user with fingers.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 includes a light source such as an LED (Light Emitting Diode), and notifies the operation state (for example, power ON / OFF) of the HMD 100 according to the lighting state of the light source. The display switching key 13 outputs, for example, a signal instructing switching of the image display mode in response to the pressing operation.

  The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or the like can be adopted. The luminance switching key 15 outputs a signal instructing increase / decrease of the luminance of the image display unit 20 in response to the pressing operation. The direction key 16 outputs an operation signal in response to a pressing operation on a key corresponding to the up / down / left / right direction. The power switch 18 is a switch for switching on / off the power of the HMD 100.

  3A and 3B are diagrams showing the configuration of the main part of the image display unit 20, where FIG. 3A is a main part perspective view of the image display unit 20 viewed from the user's head side, and FIG. 3B is the right camera 61 and the left side. It is explanatory drawing of the angle of view of the camera 62. FIG. In FIG. 3A, illustration of the right cord 42 and the left cord 44 connected to the image display unit 20 is omitted.

FIG. 3A shows the side of the image display unit 20 in contact with the user's head, in other words, the side visible to the user's right eye RE and left eye LE. In other words, the back sides of the right optical image display unit 26 and the left optical image display unit 28 are visible.
In FIG. 3A, the half mirror 261A that irradiates the user's right eye RE with the image light and the half mirror 262A that irradiates the left eye LE with the image light appear as substantially rectangular areas. Further, the entire right optical image display unit 26 and left optical image display unit 28 including the half mirrors 261A and 262A transmit external light as described above. For this reason, the user sees the outside scene through the entire right optical image display unit 26 and left optical image display unit 28, and visually recognizes a rectangular display image at the positions of the half mirrors 261A and 262A.

As described above, the right camera 61 is arranged at the end on the right holding unit 21 side so as to face the front of the image display unit 20, and the left camera 62 is arranged at the end on the left holding unit 23 side.
FIG. 3B is a diagram schematically showing the positions of the right camera 61 and the left camera 62 in plan view together with the right eye RE and the left eye LE of the user. The angle of view (imaging range) of the right camera 61 is indicated by CR, and the angle of view (imaging range) of the left camera 62 is indicated by CL. Note that FIG. 3B shows horizontal field angles CR and CL, but the actual field angles of the right camera 61 and the left camera 62 expand in the vertical direction as in a general digital camera.

The angle of view CR and the angle of view CL are substantially bilaterally symmetric with respect to the center position of the image display unit 20, and both the field angle CR and the angle of view CL include the frontal direction of the center position of the image display unit 20. For this reason, the angles of view CR and CL overlap in front of the central position of the image display unit 20.
For example, as shown in FIG. 3B, when there is a target OB in the front direction of the image display unit 20, the target OB is included in both the view angle CR and the view angle CL. For this reason, the target OB is captured in both the captured image of the right camera 61 and the captured image of the left camera 62. Here, when the user gazes at the target OB, the user's line of sight is directed toward the target OB as indicated by reference signs RD and LD in the figure. In general, the viewing angle of a human is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Among them, the effective field of view with excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. Furthermore, the stable gaze field in which the gaze point that the human gazes at appears to be quickly and stably is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction.
Therefore, when the gazing point is the target OB, the effective visual field is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction around the lines of sight RD and LD, and 60 to 90 degrees in the horizontal direction and 45 degrees to the vertical direction. A stable visual field is about 70 degrees, and a viewing angle is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction.

  In addition, an actual field of view (FOV: Field Of View) is the actual field of view that the user wearing the HMD 100 sees through the image display unit 20 and the right optical image display unit 26 and the left optical image display unit 28. Call it. In the configuration of the present embodiment shown in FIGS. 1 and 2, the real field of view corresponds to an actual field of view that the user sees through the right optical image display unit 26 and the left optical image display unit 28. The real field of view is narrower than the viewing angle and stable focus field described with reference to FIG.

  The right camera 61 and the left camera 62 are preferably capable of imaging a wider range than the user's field of view. Specifically, the entire view angles CR and CL are at least wider than the effective field of view of the user. Is preferred. Further, it is more preferable that the entire angles of view CR and CL are wider than the actual field of view of the user. More preferably, the entire angle of view CR, CL is wider than the stable viewing field of the user, and most preferably, the entire angle of view CR, CL is wider than the viewing angle of the user.

  For this reason, the right camera 61 and the left camera 62 are arranged such that the field angle CR and the field angle CL overlap each other in front of the image display unit 20 as illustrated in FIG. Further, the right camera 61 and the left camera 62 may be configured by wide angle cameras. That is, the right camera 61 and the left camera 62 may include a so-called wide-angle lens as an imaging lens and can capture a wide angle of view. The wide-angle lens may include a lens called a super-wide-angle lens or a quasi-wide-angle lens, and may be a single focus lens or a zoom lens. 62 may be provided. Further, the angle of view CR of the right camera 61 and the angle of view CL of the left camera 62 may not be the same angle. Further, the imaging direction of the right camera 61 and the imaging direction of the left camera 62 do not have to be completely parallel. When the captured image of the right camera 61 and the captured image of the left camera 62 are superimposed, it is only necessary to capture a range wider than the visual field of the user.

FIG. 4 is a functional block diagram of each part constituting the HMD 100.
The HMD 100 includes an interface 125 for connecting various external devices OA that are content supply sources. As the interface 125, for example, an interface corresponding to wired connection such as a USB interface, a micro USB interface, a memory card interface, or the like may be used, and the interface 125 may be configured by a wireless communication interface. The external device OA is an image supply device that supplies an image to the HMD 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used.

The control device 10 includes a control unit 140, an input information acquisition unit 110, a storage unit 120, a transmission unit (Tx) 51, and a transmission unit (Tx) 52.
The input information acquisition unit 110 is connected to the operation unit 135. The operation unit 135 includes the track pad 14, the direction key 16, the power switch 18, and the like. The input information acquisition unit 110 acquires input contents based on a signal input from the operation unit 135. The control device 10 includes a power supply unit (not shown), and supplies power to each unit of the control device 10 and the image display unit 20.

The storage unit 120 is a nonvolatile storage device, and stores various computer programs and data related to these programs. The storage unit 120 may store still image data or moving image data to be displayed on the image display unit 20.
The storage unit 120 stores setting data 121. The setting data 121 includes various setting values used by the control unit 140. The setting value included in the setting data 121 may be a value input in advance by operating the operation unit 135, or a setting value from an external device OA or another device (not shown) via the communication unit 117 or the interface 125. May be received and stored.
The storage unit 120 also stores scenario data 123 and history data 124. The scenario data 123 is data used to guide the work procedure and the like to the user wearing the HMD 100. The history data 124 is data including information related to the progress and execution history of work related to the scenario data 123. The scenario data 123 and the history data 124 will be described later. The scenario data 123 includes item data and order data. The storage unit 120 corresponds to an order data storage unit. The storage unit 120 stores history data 124 (execution data), and thus corresponds to an execution data storage unit.

A three-axis sensor 113, a GPS 115, a communication unit 117, a short-range wireless communication unit 119, and a voice recognition unit 114 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 acquires the detection value of the triaxial sensor 113. With the three-axis sensor 113, the control unit 140 can detect the movement of the control device 10, and can detect an operation such as shaking the control device 10, for example. The 3-axis sensor 113 may be replaced with a 9-axis sensor. In this case, the control unit 140 can acquire detection values of the three-axis acceleration sensor, the three-axis angular velocity sensor, and the three-axis geomagnetic sensor, and can detect the attitude, orientation, and movement of the control device 10, for example.
The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and calculates the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. The GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140.

The communication unit 117 executes wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), or Bluetooth (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and causes the image display unit 20 to display an image. On the other hand, when the external device OA is wired to the interface 125, the control unit 140 acquires content data from the interface 125 and causes the image display unit 20 to display an image. The communication unit 117 and the interface 125 function as a data acquisition unit DA that acquires content data from the external device OA.

  The short-range wireless communication unit 119 is a communication unit that performs short-range wireless data communication under the control of the control unit 140. The short-range wireless communication unit 119 includes a wireless communication interface including an antenna, an RF circuit, a baseband circuit, and the like (not shown). The short-range wireless communication unit 119 performs wireless communication called NFC (Near Field radio Communication), for example. Specifically, short-range wireless communication such as FeliCa (registered trademark), ISO / IEC 14443, and ISO / IEC 18092 is performed. Further, the short-range wireless communication unit 119 may be configured to execute wireless communication conforming to the Bluetooth standard. In this case, for example, it is preferable that the short-range wireless communication unit 119 includes an antenna or an RF circuit (not shown) compatible with Bluetooth class 3 (output 1 mw) and is suitable for short-range wireless communication. The short-range wireless communication unit 119 may be configured as a so-called IC tag reader, and may acquire data from the IC tag by performing wireless communication with an external IC tag (not shown).

  The control unit 140 stores a CPU (not shown) for executing a program, a RAM (not shown) for temporarily storing a program and data executed by the CPU, and a basic control program and data executed by the CPU in a nonvolatile manner. ROM (not shown). The control unit 140 reads and executes the computer program stored in the storage unit 120, and operates the operating system (OS) 150, the image processing unit 160, the imaging control unit 161, the position detection unit 162, the AR display control unit 163, and the state detection unit. 164, functioning as a communication control unit 165, an audio processing unit 187, and a display control unit 190.

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). The image processing unit 160 stores the converted digital image signal in the RAM of the control unit 140 for each frame as image data of the target image (Data in the figure). This image data is, for example, RGB data.
Note that the image processing unit 160 may perform a resolution conversion process for converting the resolution of the image data into a resolution suitable for the right display driving unit 22 and the left display driving unit 24 as necessary. The image processing unit 160 also performs image adjustment processing for adjusting the brightness and saturation of image data, 2D image data is created from 3D image data, or 2D / 3D conversion processing is performed to generate 3D image data from 2D image data. May be executed.

  The image processing unit 160 transmits the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data stored in the RAM via the transmission units 51 and 52, respectively. The transmission units 51 and 52 function as a transceiver and execute serial transmission between the control device 10 and the image display unit 20. The image data Data transmitted via the transmission unit 51 is referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is referred to as “left eye image data”.

  The display control unit 190 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24, and generates and generates image light by each of the right display drive unit 22 and the left display drive unit 24 based on the control signal. Control injection. Specifically, the right LCD 241 is controlled to be turned on / off by the right LCD control unit 211, and the right backlight 221 is turned on / off by the right backlight control unit 201. Further, the display control unit 190 controls the driving of the left LCD 242 by the left LCD control unit 212 and the driving of the left backlight 222 by the left backlight control unit 202.

  The audio processing unit 187 acquires an audio signal included in the content, amplifies the acquired audio signal, and outputs the amplified audio signal to the right earphone 32 and the left earphone 34. In addition, the sound processing unit 187 acquires sound collected by the microphone 63 and converts it into digital sound data. The voice processing unit 187 may perform a preset process on the digital voice data.

  The image display unit 20 includes the right camera 61 and the left camera 62 described above. The image display unit 20 includes an interface 25, a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, and a left guide as a left optical image display unit 28. An optical plate 262 and a nine-axis sensor 66 are provided.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). When the image display unit 20 is mounted on the user's head, the control unit 140 can detect the movement of the user's head based on the detection value of the 9-axis sensor 66. For example, the control unit 140 can estimate the magnitude of the tilt and the tilt direction of the image display unit 20 based on the detection value of the 9-axis sensor 66.

The interface 25 includes a connector to which the right cord 42 and the left cord 44 are connected. The interface 25 outputs the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data transmitted from the transmission unit 51 to the corresponding reception units (Rx) 53 and 54. Further, the interface 25 outputs a control signal transmitted from the display control unit 190 to the corresponding receiving units 53 and 54, the right backlight control unit 201, or the left backlight control unit 202.
The interface 25 is an interface for connecting the right camera 61, the left camera 62, and the 9-axis sensor 66. The imaging data of the right camera 61 and the left camera 62, the acceleration (three axes), angular velocity (three axes), and geomagnetism (three axes) detection results by the nine-axis sensor 66 are sent to the control unit 140 via the interface 25. It is done.

  The right display drive unit 22 includes the right backlight 221, the right LCD 241, and the right projection optical system 251 described above. The right display driving unit 22 includes a receiving unit 53, a right backlight (BL) control unit 201 that controls the right backlight (BL) 221, and a right LCD control unit 211 that drives the right LCD 241.

  The reception unit 53 operates as a receiver corresponding to the transmission unit 51 and executes serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data input via the reception unit 53. .

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes the left backlight 222, the left LCD 242 and the left projection optical system 252 described above. The left display driving unit 24 includes a receiving unit 54, a left backlight control unit 202 that drives the left backlight 222, and a left LCD control unit 212 that drives the left LCD 242.

The reception unit 54 operates as a receiver corresponding to the transmission unit 52 and executes serial transmission between the control device 10 and the image display unit 20. The left backlight control unit 202 drives the left backlight 222 based on the input control signal. The left LCD control unit 212 drives the left LCD 242 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data Data input via the reception unit 54. .
The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as a right “image light generation unit”. Similarly, the left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are collectively referred to as a left “image light generation unit”.

  As shown in FIG. 3A, the range in which the user wearing the HMD 100 visually recognizes the image light emitted from the half mirrors 261A and 262A is smaller than the real field of view. For example, when the HMD 100 displays an image that exhibits an AR (Augmented Reality) effect (hereinafter referred to as an AR image), the AR image is displayed so as to overlap the target OB that the user gazes at. In this case, the AR effect can be obtained by making the AR image appear to overlap the target OB, but the area where the AR image is visually recognized is restricted by the size of the half mirrors 261A and 262A. For this reason, even if the target OB is included in the real field of view, it may be difficult to display the AR image at a position overlapping the target OB. Also, when displaying an image other than the AR image, there is a possibility that the area where the HMD 100 can display the image is smaller than the actual visual field, which may be a functional limitation.

  Therefore, the HMD 100 uses an outside scene image wider than the actual field of view of the user, such as a captured image of the right camera 61 and the left camera 62 or an image prepared in advance. Display corresponding to the outside scene is performed, for example, AR content corresponding to the target OB is displayed.

In the HMD 100, the half mirror 261A and the half mirror 262A reflect the image light to the user's eyes to form a display area. The HMD 100 reflects the image light by the half mirrors 261A and 262A and makes the user's eyes recognize the virtual image. For this reason, in this embodiment, the display area is not the half mirrors 261A and 262A itself, but an area where the user perceives the image light reflected by the half mirrors 261A and 262A. When the image display unit 20 is configured to form an image on the half mirrors 261A and 262A, the half mirrors 261A and 262A correspond to the display area.
The display area corresponds to the right LCD 241 and the left LCD 242, and is an area where the user can visually recognize images displayed on the LCDs 241 and 242. For example, when an image is displayed in the entire displayable area (displayable area) of the right LCD 241 and the left LCD 242, the user can visually recognize an image having the entire size of the display area.
As will be described later, for example, the display area is located approximately in the center of the user's visual field, and is the same as or smaller than the visual field.

  The imaging control unit 161 controls the right camera 61 and the left camera 62 to execute imaging, and acquires captured image data. The imaging control unit 161 may cause only one of the right camera 61 and the left camera 62 to perform imaging, or may cause both to perform imaging.

The position detection unit 162 (input unit, target detection unit) executes a function for the control unit 140 to detect (recognize) a target (target object) from the outside scene image. In this function, the position detection unit 162 analyzes captured image data acquired by the imaging control unit 161, for example. Specifically, the position detection unit 162 detects the position of the AR display target and the operation position of the operation by the user based on the captured image data of the right camera 61 and / or the left camera 62. In the position detection unit 162, a target for AR display and a pointer used for an operation by the user are set as targets to be detected.
The target of AR display constitutes a part of the range (the user's field of view) that the user sees through the image display unit 20 such as an object in real space (including a fixed object such as a building) and a landscape. To do. The target of AR display is detected from the captured image data of the right camera 61 and the left camera 62, and the control unit 140 displays the AR content according to the detected target.

  The target can be set for each AR display content. For example, data used in the process of detecting the target may be included in the AR display content (hereinafter referred to as AR content) data. Further, data related to the target to be detected may be stored as setting data 121. The position detection unit 162 acquires data for detecting the target from the AR content or the setting data 121, and recognizes the target included in the outside scene image using this data. The data for detecting the object is data used for processing for detecting the image of the object from the captured image, and is a feature amount of the image of the object. For example, when the target object is an object, the setting data includes a feature amount indicating the color, shape, and other characteristics of the captured image when the object is imaged. In this case, the position detection unit 162 performs a process of extracting the object image from the image data of the outside scene image, calculates the feature amount of the extracted object image, and calculates the calculated feature amount and the feature included in the setting data 121. Compare with quantity. When the feature values are close or the same, the object of the image extracted from the outside scene image can be recognized as the target. Further, when the setting data 121 includes a plurality of feature amounts for the object, the position detection unit 162 can detect and recognize the object from the outside scene image based on the plurality of feature amounts. If the target cannot be recognized in the outside scene image, the position detection unit 162 waits until the imaging control unit 161 acquires a new outside scene image in step S12, and performs processing for recognizing the target for the new outside scene image.

The control unit 140 has a function of recognizing and detecting an input by the user, and this function can be realized as a function of the input unit. The input unit detects an image of an indicator such as a user's hand from the captured image of the right camera 61 and / or the left camera 62, and detects and recognizes the position, direction, or movement of the indicator as an input. For example, the input unit detects the position input by specifying the position of the indicator. The input unit may detect and recognize a gesture input, for example, when it is determined that the movement of the indicator corresponds to a preset gesture. The input unit detects an operation on the operation unit 135 based on a signal input from the input information acquisition unit 110. Further, the input unit may detect and recognize the input of the voice command when the voice processing unit 187 analyzes the voice collected by the microphone 63 and determines that the voice command is set in advance. Good. Further, the position detection unit 162 inputs a knock operation to the image display unit 20 when it is determined that the detection value pattern of the 9-axis sensor 66 included in the image display unit 20 corresponds to an operation of knocking the image display unit 20. May be detected and recognized. In addition, the position detection unit 162 detects and recognizes a marker reading input when detecting an image of an input marker such as a two-dimensional code or a barcode from a captured image of the right camera 61 and / or the left camera 62. Also good. The position detection unit 162 detects an indicator or marker from the captured image data of the right camera 61 and / or the left camera 62, and the indicator or marker corresponding to the user's line-of-sight direction detected by the line-of-sight sensor 68. May be detected and recognized.
Moreover, HMD100 is good also as a structure provided with the foot switch (not shown) which a user operates with a leg | foot. The foot switch may be wired to the control unit 140 or the operation unit 135, or may be connected to the communication unit 117 by wireless communication such as Bluetooth (registered trademark). In this case, the input unit may detect an operation of the foot switch and recognize it as an input. Further, vital sensors such as an electromyograph (not shown), a pulse measuring device (not shown), a blood pressure measuring device (not shown), a blood oxygen concentration measuring device (not shown) are provided in the HMD 100, and these vital sensors Based on the measured value and the detected value, the input unit may detect and recognize the input.

  In the present embodiment, the position detection unit 162 operates as an input unit. The position detection unit 162 detects the image of the indicator from the captured image data of the right camera 61 and the left camera 62, and detects the position indicated by the indicator. The indicator is an object used by the user for operation, and is a part of the body such as the user's hand or finger, or an object operated by the user. The shape of the object is not particularly limited, such as a rod-shaped object or a pen-shaped object. The indicator may be provided with a light emitting unit (not shown) that emits infrared light or visible light, or may be marked with a pattern or pattern that can be detected in the captured image data. In this case, the position detection unit 162 can detect the indicator from the captured image data at high speed. The position detection unit 162 detects the indicator from the captured image data using the feature amount data stored in advance in the storage unit 120, as in the process of detecting the target from the captured image data. The feature amount data of the indicator is included in the setting data 121, for example.

  The position detection unit 162 specifies the position of the target detected from the captured image data and the position of the indicator. Further, the position detection unit 162 performs a process of obtaining a relative position between the detected object and indicator and the display area visually recognized by the user. Thereby, it is possible to associate the positions of the images (including characters, images, symbols, and the like) displayed by the right display driving unit 22 and the left display driving unit 24 with the positions of the target and the indicator in the real space. . That is, the position where the user visually recognizes the image displayed by the right display driving unit 22 and the left display driving unit 24 is associated with the position where the user visually recognizes the target or the indicator through the image display unit 20. Can do. Therefore, it is possible to accurately detect the position indicated by the user using the indicator. In addition, an image can be displayed at a position corresponding to the position where the user visually recognizes the target.

Note that the method of recognizing the object or the indicator by the position detection unit 162 is not limited to the method of recognizing the image of the object based on the feature amount of the image as described above. For example, a target or indicator may be selected from an object included in the outside scene image according to a user instruction. In this case, the user's instruction may be an instruction by voice, and the voice processing unit 187 converts the voice collected by the microphone 63 into text, so that the position detection unit 162 recognizes and identifies the target. Get information for. For example, when the sound that specifies the feature of the object in the captured image, such as the color or the shape of the target, is converted into text, the position detection unit 162 detects an image corresponding to the specified feature from the captured image, recognize.
The method for inputting information related to the object may be an operation on the track pad 14 or the control unit 140 may detect a gesture operation with a finger or the like. In this case, the user may perform a gesture operation by moving a finger or the like within the imaging range of the right camera 61 and the left camera 62, and the control unit 140 may detect this gesture operation. Specifically, a method of designating the target object itself by pointing the target object at the user may be employed. In addition, a gesture may be used such that the user points the direction of the object with a hand or a finger, and the range in which the user visually recognizes the object is surrounded with the hand or the finger.

  The AR display control unit 163 reads content data (not shown) stored in the storage unit 120, controls the image processing unit 160 and the display control unit 170, and causes the image display unit 20 to display an image for AR display. Further, when the content data includes audio data, the AR display control unit 163 controls the audio processing unit 187 to output the audio of the content from the right earphone 32 and the left earphone 34.

  Here, the AR display control unit 163 controls the display position of the AR content based on the total position between the target and the display area detected by the position detection unit 162 and the relative position between the indicator and the display area. That is, the AR display control unit 163 performs AR display that displays an image, a character, or the like at a position corresponding to the target. The display position of the AR content may be a position overlapping the target or the periphery of the target. As a result, the HMD 100 provides information about the object or changes the appearance of the object seen through the image display unit 20. The AR content includes image and character data displayed at a position corresponding to the target. The AR display control unit 163 determines the display position of the AR content corresponding to the detected target, and displays the AR content.

  In this embodiment, the AR display control unit 163 displays item data included in the scenario data 123 based on the scenario data 123 stored in the storage unit 120. That is, an image that displays the scenario data 123 in a predetermined display format described later corresponds to the AR content.

  FIG. 5 is a schematic diagram illustrating a configuration example of data stored in the storage unit 120. (A) illustrates a configuration example of the scenario data 123 and the history data 124, and (B) illustrates another configuration example of the history data 124. Indicates.

The scenario data 123 includes items, and an order is associated with the items. The order is a relative order between items, and when the scenario data 123 includes one item, the order may not be associated.
Each item included in the scenario data 123 corresponds to an operation. The action refers to an action or an action performed by the user, and may be a movement of the user's body, or may include thought or judgment. One item of the scenario data 123 may correspond to one operation, and one item of the scenario data 123 may correspond to a sequence including a plurality of operations.

  Scenario data 123 illustrated in FIG. 5A is scenario data related to a work procedure, and includes a plurality of items 123a corresponding to the work. Each item 123a is given the names “Work 1”, “Work 2”, “Work 3”,. Further, the contents (work contents) of each item are associated with each item 123a and are included in the scenario data 123.

  The scenario data 123 in FIG. 5A has a form in which history data 124 is integrated. The history data 124 is data relating to the execution history of items included in the scenario data 123. When the scenario data 123 is a work scenario, the history data 124 is data related to a work progress history and an execution history corresponding to the items of the scenario data 123. In the example of FIG. 5A, the history data 124 includes execution history data 124a and execution detail data 124b. The execution history data 124a is data stored in association with each item 123a, and indicates an execution state (also referred to as an execution state, a progress state, or a progress state) of the work corresponding to the item 123a. In the example of FIG. 5A, when the work corresponding to the item 123a is completed, “completion” is set as the execution history data 124a. In addition, execution history data 124a such as non-executed and non-executed can be set.

  The execution detail data 124b includes more detailed data regarding the history of execution of the work corresponding to the item 123a. For example, it includes data such as the date and time when the work of each item 123a included in the scenario data 123 is executed, the degree of progress, and the date and time of completion. The execution detail data 124b may include captured image data captured by the right camera 61 and / or the left camera 62 when an operation corresponding to the item 123a is performed. The captured image data may be still image data or moving image data, and may include data on the imaging date and time.

  The scenario data 123 may include image data 123b in association with the item 123a. The image data 123b may be a still image or moving image data.

  FIG. 5B shows another configuration example of the scenario data 123 and the history data 124. FIG. 5B particularly shows an example in which the execution order of the items 123a and the work history corresponding to each item 123a are displayed as a time chart. The time chart of FIG. 5B is displayed on the image display unit 20 under the control of the AR display control unit 163, for example. This time chart includes the execution order of operations 1, 2, and 3 corresponding to the item 123a, and an arrow-shaped index (mark) indicating the current operation status. For example, when the AR display control unit 163 guides the work based on the scenario data 123 and the user executes the work, the time chart of FIG. 5B is displayed to inform the user of the execution state of the work. be able to.

In the HMD 100, when performing guidance related to work based on the scenario data 123, an execution state of the work can be input by a user input operation.
Here, an input method in the HMD 100 will be described.

6A and 6B are explanatory diagrams of an input method of the HMD 100. FIG. 6A shows an input example using a virtual keyboard, and FIG. 6B shows an example of position input.
6A and 6B, VR indicates a user's visual field, and V1 indicates an area in which the HMD 100 can display an image and allow the user to visually recognize it, that is, a display area of the image display unit 20. Show.
For example, the display area V1 is located approximately in the center of the user's visual field VR and is narrower than the visual field VR. The display area V1 may be the same size as the visual field VR, and the size and position are not limited to the examples in FIGS.

  In the input method of FIG. 6A, input is performed using the user's hand in a state where the virtual keyboard V11 is displayed in the display area V1. The display position of the virtual keyboard V11 can be a preset position. The AR display control unit 163 displays the virtual keyboard V11 based on the image data (not shown) of the virtual keyboard V11 stored in the storage unit 120. The position detection unit 162 detects the user's right hand HR and left hand HL in the visual field VR, and calculates the position of the fingertip. The region in which the position detection unit 162 can detect the position is the imaging range of the right camera 61 and the left camera 62, and may be outside the display region V1. The AR display control unit 163 detects an operation on the key of the virtual keyboard V11 when the position of the fingertip calculated by the position detection unit 162 overlaps the display position of the key of the virtual keyboard V11.

  FIG. 6B shows an input method in which a position instruction operation (position input operation) is performed by the user's hand in a state where the display area V1 is not displayed. This input method can be executed without displaying anything in the display area V1, and has an advantage that, for example, the visibility of an outside scene seen through the image display unit 20 is not affected. The position detection unit 162 detects the fingertip of the user's hand (right hand HR in the drawing) from the captured image data, and calculates the position of the fingertip P. The AR display control unit 163 acquires the position of the fingertip P calculated by the position detection unit 162 as the input position. For example, the position detection unit 162 may be configured to output coordinates indicating the relative position of the fingertip P with respect to the display area V1. Further, the position detection unit 162 may be configured to output coordinates indicating the relative position of the fingertip P with respect to the visual field VR.

FIGS. 7A, 7 </ b> B, and 7 </ b> C are diagrams illustrating display examples in the HMD 100, and illustrate examples of display modes in which items of the scenario data 123 are displayed under the control of the AR display control unit 163.
In the example of FIGS. 7A to 7C, the user's visual field VR includes the target OB, and the user views the target OB through the image display unit 20. The position detection unit 162 detects the position of the target OB that is the target for AR display from the captured image of the camera 61.

In the example of FIG. 7A, the AR display control unit 163 displays a list SR that displays a list of items of the scenario data 123. The list SR is a list in which the names of items included in the scenario data 123 are arranged according to the order of items set in the scenario data 123. In the list SR, the content of the item may be displayed together with the name (title) of the item.
A check box CH is arranged in the list SR in association with the display position of each item. The check box CH indicates that the operation corresponding to each item has been completed. When the state detection unit 164 detects the completion of the operation, the AR display control unit 163 changes the display state of the check box CH to a state indicating the operation completion.
When the scenario data 123 includes one item, the list SR includes a name and a check box CH for one item.

The list SR may be configured without the check box CH. In this case, the AR display control unit 163 may delete, from the list SR, an item whose operation has been detected by the state detection unit 164. Further, the display color and display brightness in the list SR may be changed.
Further, as shown in FIG. 7B, the marker M1 may be displayed so as to overlap the item for which the operation has been completed in the list SR. Furthermore, a marker M2 indicating that the operation has not been completed may be displayed for an item for which the operation has not been completed or an item for which the next operation has not been performed.

  The example of FIG. 7C is an example in which one item or a preset number of items of the scenario data 123 are displayed. In this example, the name of the item of the scenario data 123, or the name and contents thereof are displayed one by one on the display unit D in a balloon format in the display area V1. The end of the display unit D may be in the form of an arrow that indicates the position to be operated in the target OB. Further, the marker M3 may be displayed at the target position of the operation corresponding to the item displayed on the display unit D. In the example of FIG. 7C, when the state detection unit 164 detects that the operation corresponding to the item displayed on the display unit D is completed, the AR display control unit 163 includes the next included in the scenario data 123. Are displayed on the display section D. As described above, the items included in the scenario data 123 are sequentially displayed on the display unit D according to the execution order of the operation of the items.

  The display position and display mode of the list SR and the display unit D can be changed as appropriate. For example, the list SR and the display unit D are displayed at a position overlapping the operation target OB corresponding to the item of the scenario data 123. Also good. In addition, if it is displayed at a position that does not overlap the target OB, there is an advantage that the operation is not hindered and the operation is facilitated while viewing the item. The list SR and the display unit D may be displayed in 3D (stereoscopic) or as a 2D (planar) image. Furthermore, the list SR and the display unit D may be displayed only on one of the right display drive unit 22 and the left display drive unit 24.

  Further, when displaying the list SR and the display portion D, the time chart illustrated in FIG. 5B may be displayed. In this case, it is possible to display a plurality of items so that the execution order of the operations can be easily grasped. When a position indicating operation for specifying the position of the time chart is performed, an item corresponding to the position specified by the time chart may be displayed. Further, the image data included in the scenario data 123 may be displayed for the item corresponding to the designated position.

  Further, when items are displayed on the list SR or the display unit D, as illustrated in FIG. 5A, an image based on the image data included in the scenario data 123 may be displayed together. Further, the list SR or the display unit D and the image may be switched and displayed.

The state detection unit 164 (FIG. 4) detects the execution state of the operation (work) corresponding to the item 123 a of the scenario data 123 while the AR display control unit 163 performs display based on the scenario data 123.
The process in which the state detection unit 164 detects the execution state is performed by analyzing captured image data of the right camera 61 and / or the left camera 62, for example. In this case, the state detection unit 164 detects at least one of a work target, an instrument used for the work, a tool, a user's body, and the like from the captured image data. The state detection unit 164 generates data related to the execution state of the work based on the position, size, color, shape, and the like of the target of the work, the tool used for the work, the tool, the user's body, and the like. The data relating to the execution state includes, for example, not executed, being executed, and execution completed.

  Further, the state detection unit 164 may detect the execution state of the operation corresponding to the item based on the user's operation in a state where the AR display control unit 163 displays the item of the scenario data 123. In this case, the operation detected by the state detection unit 164 may be an operation on the operation unit such as the track pad 14 of the control device 10. Moreover, you may receive the input by the input method demonstrated with reference to FIG. 6 (A) and (B).

  For example, in the example shown in FIG. 7A, the state detection unit 164 performs the operation of designating the position overlapping the check box CH by the indicator as shown in FIG. 6B. , It may be detected that the operation has been completed. Further, it may be detected that the operation is completed when the user performs a preset gesture with the indicator. Further, by analyzing the sound collected by the microphone 63 with the sound processing unit 187, the user inputs an instruction that the operation is completed by voice, and it is detected that the operation is completed based on the instruction input. Good. The completion of the operation may be detected based on an operation on the direction key 16 or the track pad 14 provided in the control device 10.

In addition, after the AR display control unit 163 starts display based on the scenario data 123, the state detection unit 164 acquires the captured image data of the right camera 61 and the left camera 62, and performs the operation on the acquired captured image data. You may preserve | save as data regarding a state. In this case, moving image data during operation may be stored in the storage unit 120, for example, like a drive recorder.
In addition, the state detection unit 164 may generate a time chart (for example, FIG. 5B) reflecting the execution state of the detected operation. Further, this time chart may be a comparative time chart to which a comparison result indicating a difference from a preset standard time (standard time) is given.

  The communication control unit 165 controls the communication unit 117 and the short-range wireless communication unit 119 to execute wireless communication with an external device, IC tag, or the like (not shown), and acquires data.

FIG. 8 is a flowchart showing the operation of the HMD 100.
When the user wears the HMD 100 and the display of the scenario data 123 is instructed by operating the operation unit 135, the control unit 140 starts an operation related to the display (step S11).
The control unit 140 acquires the scenario data 123 from the storage unit 120 (step S12), and selects the display format of the scenario data 123 (step S13). In step S13, the control unit 140 may select a preset display format, or may select a display format designated by a user input. Examples of the display format include the formats shown in FIGS.

  The control unit 140 determines whether to use (reproduce) the history data 124 for displaying the scenario data 123 (step S14). When it is determined that the history data 124 is not to be reproduced due to the prior setting or the user's operation (step S14; NO), the control unit 140 displays an image of the item of the scenario data 123 in the display format selected in step S13. It is displayed on the unit 20 (step S15). In step S15, the display order of the items is controlled so as to match the execution order of the operations corresponding to the items of the scenario data 123.

Subsequently, the control unit 140 detects the operation state based on the captured image data of the right camera 61 and / or the left camera 62 and the user's input operation (step S16). Then, the control unit 140 creates a record of the detected operation state (step S17), and determines whether or not the operation corresponding to one item is completed (step S18). When the operation is not completed (step S18; NO), the control unit 140 returns to step S16.
Further, when the operation is completed (step S18; YES), the control unit 140 changes the display state of the item determined to have completed the operation among the items displayed on the image display unit 20 (step S19). The control unit 140 stores the history data 124 in the storage unit 120 or updates the history data 124 stored in the storage unit 120 based on the record created in step S17 (step S20). Here, if the display format selected in step S13 is a display format in which items are sequentially switched one by one or a preset number, and the operation is completed for all items being displayed, the control unit 140 displays a new item.

Thereafter, the control unit 140 determines whether or not the display is completed (step S21). When the corresponding operation is completed for all items included in the scenario data 123, or when the end of the display is instructed by the user's operation, the control unit 140 determines to end the display (step S21; YES) ), This process is terminated.
When it determines with not complete | finishing a display (step S21; NO), the control part 140 returns to step S16.

  When it is determined in step S14 that the history data 124 is to be reproduced (step S14; YES), the control unit 140 acquires the history data 124 (step S22). The control unit 140 displays the items included in the scenario data 123 from the item next to the item set as the operation completed in the history data 124 (step S23), and proceeds to step S16.

  Further, the state detection unit 164 may detect the execution state based on an input recognized and detected by the position detection unit 162. For example, voice, the position, direction, or movement of the indicator, the detected value of the 9-axis sensor 66, the image of the input marker detected from the captured image of the right camera 61 and / or the left camera 62, a foot switch (not shown) The execution state may be detected based on the operation.

As described above, the HMD 100 according to the first embodiment to which the present invention is applied includes the image display unit 20 that displays an image so as to be visible through the outside scene, and the range that is visible through the image display unit 20. And a position detection unit 162 that identifies the input position. Based on the scenario data 123 including the items, the control unit 140 displays the items on the image display unit 20 and changes the display on the image display unit 20 in accordance with the input position specified by the position detection unit 162.
Accordingly, a person who performs an operation (for example, work) on a place or an object can see the item on the image display unit 20, and the place or object that is the operation target is transmitted through the image display unit 20. Visible. For this reason, in order to view the displayed content, it is not necessary to perform a movement with a large load such as a large movement of the line of sight, and information can be easily viewed.

  The scenario data 123 includes a plurality of items corresponding to operations, and the order of items corresponding to the execution order of operations is set for each item. Thereby, the user of HMD100 can know the execution order of operation | movement based on the display mode of an item.

The AR display control unit 163 of the control unit 140 may arrange a plurality of items in a list format and display them on the image display unit 20. In this case, the items are arranged according to the order set for the items. Thereby, the user can know the execution order of the operation from the arrangement of the displayed items.
The control unit 140 may sequentially display the items in the order set for the items. Thereby, the user can know the execution order of the operation from the display order of the items.

  The scenario data 123 includes data including information related to the execution order of operations corresponding to the items included in the scenario data 123, and the control unit 140 displays images such as the list SR and the display unit D based on the execution order. 20 is displayed. Thereby, the execution order of operation | movement can be visually recognized with an image.

  Since the HMD 100 stores the scenario data 123 and the order data in association with each other in the storage unit 120, the correspondence between the data related to the operation and the data related to the execution order of the operation, and each data can be easily read and used. it can.

The control unit 140 may display a marker as an index in an overlapping manner on an item being displayed at a position corresponding to the input position specified by the position detection unit 162. In this case, the display mode of the displayed item can be changed by the user performing input.
Further, the state detection unit 164 of the control unit 140 detects the execution state of the operation corresponding to the item based on the captured image of the outside scene by the right camera 61 and the left camera 62 or the sound collected by the microphone 63 from the outside scene. Also good. The AR display control unit 163 may change the display state of the item being displayed by the image display unit 20 according to the detected execution state. In this case, the display state of the item can be changed corresponding to the execution state of the operation.

  The storage unit 120 stores history data 124 including information related to the execution state of the operation. The control unit 140 stores the operation history data 124 corresponding to the items included in the scenario data 123 in the storage unit 120 in association with the scenario data 123. When the history data 124 is read from the storage unit 120, the control unit 140 controls the display state of items displayed by the image display unit 20 based on the read history data 124. In this case, data relating to the execution state of the operation is stored, and by reading this data and reflecting it in the display, it is possible to record the execution state and call the recorded execution state.

  In addition, the control unit 140 uses the position detection unit 162 to detect the object in the outside scene. The AR display control unit 163 displays the items included in the scenario data 123 in association with the position where the object detected by the position detection unit 162 is viewed through the image display unit 20. Thereby, it is possible to easily visually recognize both the place and the object and the item to be operated.

  As described above, the HMD 100 can display AR by superimposing a check of work performed by a human on an object in the real world or an object in an electronic medium. In this case, by adjusting the display mode to the position of the object, a stabilized display that holds the relative position with respect to the object can be performed, and the item can be held at a position where it can be easily viewed.

  In the above embodiment, the method of changing the display state of the item displayed by the image display unit 20 according to the progress state of the work (operation) is to change the display state of the check box CH in the list SR to a state indicating the operation completion. It is not limited to the method of changing. For example, an item for which the operation has been detected may be deleted from the list SR, or the display color and display brightness in the list SR may be changed. For example, the display color of the item may be maintained, and a color such as a fluorescent color that enhances the visibility may be displayed in a position overlapping the item. Further, when changing the display color of the item, the transparency of the outside scene at the position overlapping the item may be changed by adjusting the display brightness. In addition, the display position of the item may be changed. For example, the item that has been completed may be moved to a different line. Further, the item is displayed on one of the left optical image display unit 28 and the right optical image display unit 26, and the display position of the item for which the operation is completed is changed from the left optical image display unit 28 to the right optical image display unit. 26 or in the opposite direction. In this way, although it can be confirmed later, if the display clearly shows the procedure that must be performed from now on, various “display change” forms can be realized.

Examples of application of the HMD 100 according to the first embodiment include the following examples.
When a work standard document for work including a plurality of processes is configured by the scenario data 123, the scenario data 123 can be used as a check list. In this case, a document for guiding the work, a still image, and a moving image can be included in the scenario data 123. Further, the moving image may include voice in the scenario data 123.
This work standard document can be displayed by the AR display control unit 163 as a target OB for a part, equipment, or part or equipment part to be worked for each work process.
Moreover, it is good also as a structure which can edit the scenario data 123 based on a user's operation in the display based on the scenario data 123. FIG. The input method in this case can employ the method described with reference to FIG.
In this example, a user can actually work with the HMD 100 attached, and changes in work, changes in key annotations, and the like can be reflected in the scenario data 123 in real time.
Moreover, it is good also as a structure which can include the moving image or still image imaged with the right camera 61 and the left camera 62 during work, and the audio | voice collected with the microphone 63 by including in the scenario data 123, and can be preserve | saved.

Examples of the work include a line work in a factory and a work in a cell production method. Further, by enabling editing of the scenario data 123, an image that can be a standard work while working can be registered and stored as a standard work.
In addition, the scenario data 123 can be used to manage the inventory of shelves such as small shops. In this case, the user wearing the HMD 100 circulates in the store and picks up the product with the right camera 61 and the left camera 62 so that the position detection unit 162 can detect the product. In this case, the number, location, time, etc. of the product can be recorded and saved as history data 124. Thus, it can be used as a work support system for inventory management work, picking work, and the like.

  Further, a diagnosis flow can be created as scenario data 123 and used in a medical examination operation at a hospital or clinic. For example, it is possible to make a diagnosis according to the scenario data 123 by including the confirmation item and the examination result item in the patient inquiry as items of the scenario data 123. In this case, data input by medical personnel such as a doctor who is a user for items included in the scenario data 123 and captured image data of the right camera 61 and the left camera 62 are stored in the storage unit 120 as history data 124. If saved, the history data 124 can be used as a part of the electronic medical record, that is, as a diagnostic record.

  Further, the present invention can also be applied to a case where the user wearing the HMD 100 counts the number of people or objects in real space that can be visually recognized as an outside scene. In this case, each time an input operation such as a gesture is performed, counting (counting) progresses, and it is effective to display an AR display in which images and markers indicating that the counting has been completed are overlapped for the counted person or object. is there. In this application example, effects such as prevention of double counting (double counting) and checking (counting) without leakage while being hands-free can be obtained. For example, the present invention can be applied to counting the number of people at an event venue, counting the number of parts during factory work, and the like.

  The present invention can also be applied to a case where the user wearing the HMD 100 is a ball game umpire or recording staff. In this case, when a game (game) is performed in a real space that can be visually recognized as an outside scene, the play during the game can be counted. Specifically, mistakes can be prevented by counting and displaying strikes, ball counts, inning numbers, back and front of innings, scores, etc. in a baseball game. Further, for example, mistakes can be prevented by using the HMD 100 for counting the number of rounds in golf competitions, calculating scores, and the like. The player may wear the HMD 100.

  In this way, by applying the present invention by the HMD 100, the accuracy of the operation of the user is guaranteed, the procedure is checked, the number is confirmed, and the traceability (quality assurance) for quality assurance at the manufacturing site is confirmed. Compensation can be realized. For example, errors and troubles caused by uncertainties in human behavior, ambiguity in memory, and the like can be reliably prevented by the function of the HMD 100.

  The scenario data 123 may be a presentation project file created in the format of presentation software (for example, PowerPoint (registered trademark) of Microsoft (registered trademark)), for example. In this case, the presentation project file is displayed as scenario data 123, and the user wearing the HMD 100 can make a presentation while viewing the display in the display area V1. The scenario data 123 can be used as a presentation manuscript by including a sentence uttered in the presentation as the content of the item. In this case, the voice uttered by the user is collected by the microphone 63, and the voice processing unit 187 converts the voice into text and collates it with the contents of the scenario data 123, thereby uttering the text (character, Character string) can be specified by the control unit 140. Here, if the AR display control unit 163 changes the display mode of the uttered text, the progress of the presentation can be shown to the user. For example, a mode in which the uttered text is erased from the display area V1 can be considered. In this case, it is preferable that the display position and display brightness of the image and text are set to a position and brightness that do not hinder the visibility of the outside scene viewed through the image display unit 20. Thereby, for example, it becomes easy for the user to visually recognize the slide image of the presentation displayed by the projector. Further, the text to be uttered from now on may be highlighted such as highlighted.

  Conventionally, when performing a work while looking at the manual of the work procedure, etc., it is dependent on the worker's memory to confirm how far the work has been performed and to confirm whether the work has been performed in the correct procedure. It was. Similarly, when a presentation is performed, it is not easy to record how far the presentation has progressed, and it is managed depending on the memory of the person who performs the presentation. If the HMD 100 of this embodiment is used, the user who wears the HMD 100 can confirm the work content by himself / herself using a check list. In addition, in the configuration in which the operation is detected based on the captured image data, only the user's memory is used to check whether the operation corresponding to the item included in the scenario data 123 has been performed, whether the progress state has been checked, or the like. This can be done easily and reliably compared to the case of relying on. In addition, it is possible to prevent omission of execution of operations, mistakes in the execution order of operations, and the like. Even if other operations are performed while the series of operations included in the scenario data 123 are sequentially performed, information about the suspended operation can be obtained from the list SR and the display unit D displayed by the AR display control unit 163. . Thereby, it is possible to improve accuracy in various aspects of speed, accuracy, and time management by improving the accuracy of work including multiple processes, presentations, and the like.

[Second Embodiment]
FIG. 9 is a flowchart showing the operation of the HMD 100 in the second embodiment.
Since the configuration of the HMD 100 of the second embodiment is the same as the configuration described with reference to FIGS. 1 to 4 in the first embodiment, illustration and description thereof are omitted.
In the second embodiment, the HMD 100 acquires the scenario data 123 by the short-range wireless communication unit 119. Further, the history data 124 may be acquired together with the scenario data 123.

In the flowchart of FIG. 9, the same step numbers are assigned to the processes common to the operations described in FIG.
In the operation of FIG. 9, the short-range wireless communication unit 119 waits for communication under the control of the communication control unit 165 (step S31). In step S31, the short-range wireless communication unit 119 tries communication with an external device. Specifically, transmission of a wireless signal for searching for an external device, reception of a wireless signal transmitted from the external device, and the like are tried. The external device is, for example, a device that communicates with Bluetooth or NFC, an IC tag, or the like. The communication control unit 165 continues the communication attempt until the short-range wireless communication unit 119 becomes ready to start wireless communication with an external device (step S31; NO). When communication by the short-range wireless communication unit 119 becomes possible (step S31; YES), the communication control unit 165 executes wireless communication with an external device and acquires the scenario data 123 (step S32). . In step S32, the history data 124 may be acquired. The scenario data 123 acquired by the communication control unit 165 or the scenario data 123 and the history data 124 is stored in the storage unit 120.

  The AR display control unit 163 reads the scenario data 123 acquired by the communication control unit 165 and stored in the storage unit 120, and displays items included in the scenario data 123 (step S33). For the display of items, the same processing as in steps S14, S15, and S23 of FIG. 8 is performed. The control unit 140 executes the operations of steps S16 to S21.

Since the short-range wireless communication unit 119 performs short-range wireless communication as described above, the scenario data 123 is not acquired when there is no external device in a range in which the short-range wireless communication unit 119 can communicate. When an external device appears in the communicable range of the short-range wireless communication unit 119, the scenario data 123 is acquired from this device and displayed.
For example, when the control device 10 of the HMD 100 is brought close to a device that holds the scenario data 123, wireless communication occurs in step S31 (step S31; YES), and the items included in the scenario data 123 acquired from the device are images. Displayed by the display unit 20.

  When the HMD 100 according to the second embodiment approaches a device that holds the scenario data 123, the HMD 100 performs display based on the scenario data 123 held by the approached device. And display based on the scenario data 123 held by the adjacent devices. In this case, the HMD 100 can acquire and display the scenario data 123 corresponding to the place.

  Specific examples include furniture such as desks in a building, furniture and equipment, household electrical appliances such as refrigerators, machines in factories, production lines composed of multiple machines, and places such as automobiles and wireless communication with the HMD 100. A device that is capable and has scenario data 123 is arranged. In this case, when the user wearing the HMD 100 approaches a device having the scenario data 123, display based on the scenario data 123 is performed. Therefore, the scenario data 123 corresponding to the place where the user is present can be provided to the user. Further, if the scenario data 123 has a configuration that can be modified by a user operation, the scenario data 123 can be modified at any time. The modified scenario data 123 may be transmitted to an external device.

  As described above, the HMD 100 may include the communication unit 117 and the short-range wireless communication unit 119 that communicate with an external device, and may receive and acquire the scenario data 123 from the external device. In this case, the data regarding the item can be acquired from the outside. The short-range wireless communication unit 119 may acquire the scenario data 123 by executing short-range wireless communication. In this case, since the distance at which short-range wireless communication can be performed is limited, data associated with the place can be acquired.

[Modification]
It is also possible to configure as a system including a plurality of HMDs 100 described in the first and second embodiments. In this case, a mode in which an administrator who manages the operation and an executor who executes the operation wear the HMD 100, respectively.
In this configuration, the scenario data 123 and the history data 124 can be shared by transmitting and receiving the scenario data 123 and the history data 124 between the administrator's HMD 100 and the executor's HMD 100. That is, the scenario data 123 and the history data 124 are transmitted from the executing person's HMD 100 to the administrator's HMD 100, and display is performed based on the scenario data 123 and the history data 124 received by the administrator's HMD 100. Thereby, the administrator can know the execution state of the performer's operation.

  Alternatively, the scenario data 123 may be edited by the administrator's HMD 100, the edited scenario data 123 may be transmitted to the executor's HMD 100, and the executor's HMD 100 may perform display based on the edited scenario data 123. In this case, the administrator can assist the executor by editing the scenario data 123 so as to assist the execution of the executor's action. In this configuration, the administrator can be referred to as the information provider, that is, the information provider, and the performer can be referred to as the information receiver.

  A plurality of different scenario data 123 may be associated with each other. For example, the scenario data 123 for the manager and the scenario data 123 for the performer can be associated with each other. The scenario data 123 for the administrator and the scenario data 123 for the executor may be different data. When the items confirmed by the administrator and the items confirmed by the executor are different, the scenario data 123 for the administrator and the scenario data 123 for the executor include different items, and history data corresponding to each item 124 may be configured to be held by the HMD 100.

  In this case, for the work performed by the performer, the execution state of the operation can be confirmed and the history data 124 can be generated by the HMD 100 worn by the administrator and the performer. Thereby, it is possible to perform a double check regarding the execution of the operation, and to improve the execution accuracy of the operation.

  In addition, this invention is not restricted to the structure of said each embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.

  In the above embodiments, the configuration in which the user visually recognizes the outside scene through the display unit is not limited to the configuration in which the right optical image display unit 26 and the left optical image display unit 28 transmit external light. For example, the present invention can be applied to a display device that displays an image in a state where an outside scene cannot be visually recognized. Specifically, a captured image of the right camera 61 and / or the left camera 62, an image or CG generated based on the captured image, a video based on video data stored in advance or video data input from the outside, and the like are displayed. The present invention can be applied to a display device. This type of display device can include a so-called closed display device in which an outside scene cannot be visually recognized. Of course, a display device that displays video data or an analog video signal input from the outside without performing processing such as AR display, MR display, or VR display is also included as an application target of the present invention.

  Further, for example, instead of the image display unit 20, another type of image display unit such as an image display unit worn like a hat may be adopted, and an image is displayed corresponding to the left eye of the user. And a display unit that displays an image corresponding to the right eye of the user. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet. In this case, the part for positioning the position with respect to the user's body and the part positioned with respect to the part can be used as the mounting portion.

Furthermore, in the above embodiment, the image display unit 20 and the control device 10 are separated and connected by way of the connection unit 40, but the control device 10 and the image display unit 20 are integrated. It is also possible to be configured to be mounted on the user's head.
As the control device 10, a notebook computer, a tablet computer, or a desktop computer may be used. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like. Further, the control device 10 may be configured separately from the image display unit 20, and various signals may be transmitted and received between the control device 10 and the image display unit 20 by wireless communication.

  For example, as a configuration for generating image light in the image display unit 20, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used. Further, as a configuration for generating image light, LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like can be used.

The “display unit” referred to in the present invention corresponds to a configuration that emits image light, and in the following description, the fact that the HMD 100 emits image light is referred to as “display”.
In each of the above embodiments, image light is generated by the left and right image light generation units described with reference to FIG. 4, and the right optical image display unit 26 and the left optical image display unit 28 shown in FIG. A configuration in which image light is irradiated toward each of the eyes and the left eye and the image light is incident on each of the right eye and the left eye of the user is illustrated. The configuration of the “display unit” is not limited to this. That is, as long as it emits image light, it is not limited to the structure of FIG.2 and FIG.4. For example, in the configuration of the present embodiment, image light is emitted toward the user's eyes by the “right light guide” and “left light guide” having the half mirrors 261A and 262A. Further, as a configuration for generating image light, a right backlight 221 and a left backlight 222, and a right LCD 241 and a left LCD 242 are provided. The “display unit” does not require these configurations.

  For example, image light generated by a mechanism built in either or both of the right display driving unit 22 and the left display driving unit 24 of the image display unit 20 is transmitted to the user side of the image display unit 20, that is, the user's eyes. The light may be reflected by a reflection mechanism provided on the facing side and emitted to the user's eyes. Here, as the reflection mechanism, for example, a scanning optical system using a MEMS (Micro Electro Mechanical Systems) mirror can be adopted. In other words, a scanning optical system having a MEMS mirror that scans the light emitted from the image light generation unit may be provided, and the light scanned by the scanning optical system may be directly incident on the user's eyes. Further, an optical member on which a virtual image is formed by light scanned by the scanning optical system may be provided in the image display unit 20. This optical member forms a virtual image by scanning light scanned by the MEMS mirror. In this case, when the MEMS mirror scans light, a virtual image is formed on the virtual image forming surface, and the image is visually recognized (recognized) when the user catches the virtual image with eyes. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

Further, the scanning optical system is not limited to the configuration including the MEMS mirror. The mechanism that generates the image light may also be a laser light source that emits laser light. For example, the present invention can also be applied to a laser retinal projection type head mounted display. That is, the light emitting unit includes a laser light source and an optical system that guides the laser light source to the user's eyes, and the laser light is incident on the user's eyes to scan the retina and form an image on the retina. A configuration that allows the user to visually recognize an image may be employed.
Moreover, it may replace with the virtual image formation surface which receives the light scanned, and the structure which guides image light to a user's eyes using a diffraction grating may be sufficient. That is, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.

  In the configuration including the scanning optical system having MEMS or the like, the position where the user visually recognizes the image, that is, the display position of the image can be changed by changing the mounting angle of the scanning optical system in the image display unit 20. Therefore, in the process of changing the display position in each of the above embodiments, an operation of changing the angle of the scanning optical system may be performed instead of the operation of changing the display position of the image on the right LCD 241 and the left LCD 242.

  The optical system that guides the image light to the user's eyes may include an optical member that transmits external light that is incident on the apparatus from the outside and that is incident on the user's eyes together with the image light. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field.

  Further, in each of the above embodiments, a configuration in which virtual images are formed by the half mirrors 261A and 262A on a part of the right optical image display unit 26 and the left optical image display unit 28 positioned in front of the user's eyes is illustrated. The present invention is not limited to this, and an image may be displayed on a display area having an area that occupies the entire surface or most of the right optical image display unit 26 and the left optical image display unit 28. In this case, a process of reducing the image may be included in the operation of changing the display position of the image.

  Furthermore, the optical element of the present invention is not limited to the right light guide plate 261 and the left light guide plate 262 having the half mirrors 261A and 262A, and may be any optical component that allows image light to enter the user's eyes. A diffraction grating, a prism, or a holographic display unit may be used.

  Also, at least a part of each functional block shown in FIG. 4 may be realized by hardware, or may be realized by cooperation of hardware and software, as shown in FIG. It is not limited to a configuration in which independent hardware resources are arranged. The program executed by the control unit 140 may be stored in the storage unit 120 or the storage device in the control device 10, or the program stored in the external device may be stored in the communication unit 117, the short-range wireless communication unit 119, or The configuration may be such that it is acquired through the interface 125 and executed. Of the components formed in the control device 10, only the operation unit 135 may be formed as a single user interface (UI). Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, the control unit 140 illustrated in FIG. 4 may be formed in both the control device 10 and the image display unit 20, or the control unit 140 formed in the control device 10 and the CPU formed in the image display unit 20. The functions performed by and may be configured separately.

  DESCRIPTION OF SYMBOLS 10 ... Control apparatus, 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display drive part, 23 ... Left holding part, 24 ... Left display drive part, 25 ... Interface, 26 ... Right optical image Display unit 28 ... Left optical image display unit 61 ... Right camera 62 ... Left camera 100 ... Head-mounted display device (display device) 117 ... Communication unit 119 ... Short-range wireless communication unit (communication unit) , 120 ... storage unit (order data storage unit, execution data storage unit), 125 ... interface, 140 ... control unit, 150 ... operating system, 160 ... image processing unit, 161 ... imaging control unit, 162 ... position detection unit (input) 163 ... AR display control unit, 164 ... state detection unit, 165 ... communication control unit, 187 ... audio processing unit, 190 ... display control unit, 201 ... right backlight control unit, 202 ... left bar Crite control unit 211 ... Right LCD control unit 212 ... Left LCD control unit 221 ... Right backlight 222 ... Left backlight 241 ... Right LCD 242 ... Left LCD 251 ... Right projection optical system 252 ... Left Projection optical system, 261, right light guide plate, 262, left light guide plate, 261A, 262A, half mirror.

Claims (16)

A display unit that displays an image that is visible through the outside scene;
An input unit for recognizing input;
A control unit that displays items on the display unit and changes the display on the display unit in accordance with the input recognized by the input unit;
A display device comprising:   The display device according to claim 1, wherein the control unit causes the display unit to display the item based on item data including the item. The item data includes a plurality of the items corresponding to operations, and the order of the items corresponding to the execution order of the operations is set for each of the items.
The display device according to claim 2. The control unit arranges a plurality of the items in a list format to be displayed on the display unit, and the items are arranged according to an order set in the items.
The display device according to claim 3. The controller sequentially displays the items in the order set in the items;
The display device according to claim 3. The control unit causes the display unit to display an image based on order data including information related to an execution order of operations corresponding to the items included in the item data;
The display device according to claim 3, wherein: An order data storage unit for storing the item data and the order data in association with each other;
The display device according to claim 6. The input unit detects an input in a range that is visible through the display unit and identifies an input position;
The control unit changes the display on the display unit corresponding to the input position specified by the input unit,
The display device according to claim 1, wherein: The control unit displays an index on the item being displayed at a position corresponding to the input position specified by the input unit,
The display device according to claim 8. A state detection unit that detects an execution state of an operation corresponding to the item based on a captured image of the outside scene or a sound collected from the outside scene, wherein the control unit is configured according to the execution state detected by the state detection unit; Changing the display state of the item being displayed by the display unit,
The display device according to claim 1, wherein: An execution data storage unit for storing execution data including information relating to the execution state of the operation;
The control unit stores execution data of an operation corresponding to the item included in the item data in the execution data storage unit in association with the item data, and reads the execution data from the execution data storage unit In this case, based on the read execution data, controlling the display state of the item displayed by the display unit,
The display device according to claim 10. An object detection unit for detecting an object in the outside scene;
The control unit displays an item included in the item data in association with a position where the target detected by the target detection unit is viewed through the display unit;
The display device according to claim 1, wherein: A communication unit that communicates with external devices
Receiving and acquiring the item data from the external device by the communication unit;
The display device according to claim 1, wherein: The communication unit performs short-range wireless communication;
The display device according to claim 13. A control method for a display device having a display unit that displays an image so as to be visible through an outside scene,
Detecting the input of a range that is visible through the display unit to identify the input position,
Displaying the item on the display unit based on item data including the item, and changing the display state of the item being displayed to a position corresponding to the specified input position;
A control method of a display device characterized by the above. A computer-executable program for controlling a display device having a display unit that displays an image so as to be visible through the outside scene,
By the computer
Detecting the input of a range that is visible through the display unit to identify the input position,
A program for displaying the item on the display unit based on item data including the item and changing a display state of the item being displayed at a position corresponding to the specified input position.

Images