JP2016116066A - Display device and control method of display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-wear type display device capable of swiftly executing processing with respect to a movement of a user head corresponding to motion-centered movement.SOLUTION: The head-wear type display device 100 includes: an image display unit 20 that irradiates a beam of image light to eyes of the user; and a first sensor 66 and a second sensor 68 as plural movement sensors. In a worn state, either one of the first sensor 66 and the second sensor 68 is positioned at one side with respect to the center of the head; and the other movement sensor is positioned at the other side with respect the center of the head.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a display device and a display device control method.

Conventionally, a display device mounted on the head is known (for example, see Patent Document 1). The display described in Patent Document 1 is a display in which a see-through light guide is provided on a spectacle-shaped frame, and an imaging device is attached to the frame. The display generates correction data by matching an image captured by the imaging device with an image of an object visually recognized by the user through the light guide, and based on the correction data, the imaging data is used as the user's visual field. Match. As a result, AR (Augmented Reality) technology for displaying various data related to the object superimposed on the image of the object is realized.
In addition, an example in which a device mounted on the head detects the movement of the head is known (see, for example, Patent Document 2). Patent Document 2 describes a spectacle-type operating device that has a spectacle-type frame and is provided with a sensor in the temple portion of the spectacles. This device detects an operation such as a user moving his / her head or moving his / her eyes based on a detection signal of the sensor.

JP2011-27543A JP 2014-137522 A

As described in Patent Document 1, in a display device mounted on the head, it is conceivable to control display in accordance with the movement of the head. In this case, for example, as described in Patent Document 2, it is conceivable to detect movement using a sensor. In many cases, when a human moves his / her head, the center of the head or the center of the face becomes the center of motion or the reference. If the movement at the center of the movement can be directly detected, the movement amount and the direction of the movement can be obtained easily and accurately, but it is difficult to provide a sensor at the center of the head or the center of the face.
The present invention has been made in view of the above circumstances, and in a head-mounted display device, it is possible to quickly execute a process corresponding to a movement centered on a movement of a user's head. Objective.

In order to solve the above problems, the present invention is a head-mounted display device mounted on a user's head, the display unit for irradiating the user's eyes with image light, a plurality of motion sensors, Any one of the motion sensors is located on one side from the center of the head in the mounted state, and the other motion sensor is located on the other side from the center of the head. .
According to the present invention, when the user's head moves using the detection results of the motion sensors arranged on one side and the other side from the center of the head, the amount of movement and the direction of movement at the center of movement. Etc. can be promptly requested.
Here, the center of the head refers to the center of the head in a horizontal plane perpendicular to the height of the user. For example, the center of the head may refer to the center in the horizontal plane at the height position of both eyes. It may be the three-dimensional center of the head.

In the display device according to the aspect of the invention, any one of the motion sensors may be located on the left side of the center of the head in the mounted state, and the other motion sensors may be located on the right side of the center of the head. It is characterized by.
ADVANTAGE OF THE INVENTION According to this invention, when a user's head moves, the movement amount in the movement center of a head, the direction of a movement, etc. can be calculated | required rapidly.

In the display device according to the aspect of the invention, the display unit includes an optical element having a display area that emits image light toward the user's eyes, and includes a plurality of the motion sensors and the display of the optical elements. The center position of the region is arranged in a straight line.
According to the present invention, it is possible to quickly obtain the amount of movement, the direction of movement, and the like at the center of motion in response to the motion centered on the position of the optical element that constitutes the display area.

Moreover, the present invention is characterized in that, in the above display device, the optical element has a half mirror as a display area that emits image light toward the eyes of the user.
According to the present invention, it is possible to quickly determine the amount of movement, the direction of movement, and the like at the center of motion in response to the motion centered on the center of the image that can be visually recognized by the user.

In the display device according to the present invention, the display device further includes a main body that is mounted in front of the left and right eyes of the user, and at least two of the motion sensors are symmetrical with respect to a center of the main body. It is in position.
According to the present invention, there is a high possibility that a plurality of motion sensors detect both sides of the motion center of the user's motion, and it is possible to quickly determine the amount of motion, the direction of motion, and the like.

In the display device according to the present invention, the main body includes a right part positioned in front of the user's right eye, a left part positioned in front of the user's left eye, and a center of the main body. And a connecting part that connects the right part and the left part, and at least two of the motion sensors are in symmetrical positions with respect to the connecting part.
According to the present invention, it is possible to quickly determine the amount of movement, the direction of movement, and the like at the center of motion in response to the motion centered on the center of the eyeglass-type display device.

In the display device according to the present invention, the plurality of motion sensors include at least an optical sensor and an inertial sensor.
According to the present invention, by using the detection results of different types of sensors, it is possible to obtain information on the movement of the user's head, and the detection values of the sensors can be used for other purposes.

Moreover, the present invention is characterized in that, in the display device, the plurality of motion sensors are configured by any one of an optical sensor and an inertial sensor.
ADVANTAGE OF THE INVENTION According to this invention, the information regarding a user's head movement can be obtained more easily using the detection result of the same kind of sensor.

Further, the present invention is characterized in that the display device includes a motion detection unit that obtains a motion at the center of the motion of the user based on detection values of the plurality of motion sensors.
According to the present invention, a user's movement can be accurately obtained by using a plurality of movement sensors, so that processing corresponding to the movement can be quickly executed.

Further, the present invention is characterized in that, in the display device, the motion detection unit obtains a motion at the center of the motion based on each detection value of the plurality of motion sensors and the position of the motion sensor. To do.
According to the present invention, since a user's movement can be obtained more accurately using a plurality of movement sensors, processing corresponding to the movement can be quickly executed.

In order to solve the above problems, the present invention provides a plurality of display devices including a display unit that is mounted on a user's head and irradiates image light to the user's eyes, and a plurality of motion sensors. The movement at the movement center of the head is obtained based on the detection value of the movement sensor.
According to the present invention, when the user's head moves, the amount of movement and the direction of movement at the center of movement can be quickly obtained, and processing corresponding to this movement can be executed.

According to the present invention, in the control method for the display device, the movement at the center of the movement is obtained based on the detection values of the plurality of movement sensors and the position of the movement sensor.
According to the present invention, a user's movement can be accurately obtained by using a plurality of movement sensors, so that processing corresponding to the movement can be quickly executed.

Explanatory drawing which shows the external appearance structure of a head mounting | wearing type display apparatus. The figure which shows the structure of the optical system of an image display part. The top view which shows typically the state which mounted | wore the user's head with the image display part. The functional block diagram of each part which comprises the head mounted display apparatus of 1st Embodiment. The flowchart which shows operation | movement of the head mounted display apparatus of 1st Embodiment. The functional block diagram of each part which comprises the head mounted display apparatus of 2nd Embodiment. The flowchart which shows operation | movement of the head mounted display apparatus of 2nd Embodiment. Explanatory drawing which shows the modification of embodiment of this invention.

[First Embodiment]
FIG. 1 is an explanatory diagram showing an external configuration of a head-mounted display device 100 (display device) according to an embodiment to which the present invention is applied.
The head-mounted display device 100 includes an image display unit 20 (display unit) that allows the user to visually recognize a virtual image while being mounted on the user's head, and a control device 10 that controls the image display unit 20. I have. The control device 10 also functions as a controller for the user to operate the head mounted display device 100.

  The image display unit 20 is a mounting body to be mounted on the user's head, and has a glasses-shaped frame 2 (main body) in the present embodiment. The frame 2 has a right holding part 21 and a left holding part 23. The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 contacts the right ear or the vicinity thereof on the user's head, and the left holding unit 23 contacts the user's left ear or the vicinity thereof and holds the image display unit 20 on the user's head. To do.

The frame 2 is provided with a right display drive unit 22, a left display drive unit 24, a right optical image display unit 26, a left optical image display unit 28, and a microphone 63.
In this embodiment, a spectacle-type frame 2 is illustrated as an example of the main body. The shape of the main body is not limited to the spectacles type, as long as it is attached and fixed to the user's head, and if it is a shape that is worn across the user's left and right eyes, preferable. For example, in addition to the eyeglass shape described here, it may have a snow goggles-like shape that covers the upper part of the user's face, and is arranged in front of the left and right eyes of the user like binoculars. It may be a shape.
The glasses-type frame 2 has a right part 2A located in front of the user's right eye and a left part 2B located in front of the left eye. The right part 2A and the left part 2B are bridge parts 2C ( It is the shape connected by the connection part). The bridge part 2C connects the right part 2A and the left part 2B to each other at a position corresponding to the eyebrow of the user when the user wears the image display part 20.
The right part 2A and the left part 2B are connected to the temple parts 2D and 2E, respectively. The temple portions 2D and 2E support the frame 2 on the user's head like temples of glasses. The temple portion 2D of the present embodiment is constituted by the right holding portion 21 and the temple portion 2E is constituted by the left holding portion 23.
The right optical image display unit 26 is arranged in the right part 2A, and the left optical image display unit 28 is arranged in the left part 2B. When the user wears the image display unit 20, the right and left of the user Located in front of the eyes.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. The right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as “display drive unit”, and the right optical image display unit 26 and the left optical image display unit 28 are simply referred to as “optical image display unit”. Also called.

The display drive units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 described later with reference to FIG.
The right optical image display unit 26 and the left optical image display unit 28 include light guide plates 261 and 262 (FIG. 2) and a light control plate 20A. The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. The light control plate 20A is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. As the light control plate 20A, various materials such as a material that has almost no light transmission, a material that is nearly transparent, a material that attenuates the amount of light and transmits light, and a material that attenuates or reflects light of a specific wavelength may be used. it can. By appropriately selecting the optical characteristics (light transmittance, etc.) of the light control plate 20A, the external light amount incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to visually recognize the virtual image. You can adjust the ease. In the present embodiment, a case will be described in which at least a light control plate 20A having such a light transmittance that a user wearing the image display unit 20 can visually recognize an outside scene is used. The light control plate 20A protects the right light guide plate 261 and the left light guide plate 262, which are optical elements, and suppresses damage to the right light guide plate 261 and the left light guide plate 262, adhesion of dirt, and the like.
The dimming plate 20A may be detachable from the right optical image display unit 26 and the left optical image display unit 28, or may be mounted by replacing a plurality of types of dimming plates 20A, or may be omitted. .

  The frame 2 is provided with a camera unit 3. The camera unit 3 includes a camera pedestal 3C on which the upper camera 61 is disposed, and arms 3A and 3B that support the camera pedestal 3C. The arm portion 3A is rotatably connected to the right holding portion 21 by a hinge 21A provided at the distal end portion AP of the right holding portion 21. The arm 3 </ b> B is rotatably connected to the left holding part 23 by a hinge 23 </ b> A provided at the distal end part AP of the left holding part 23. Therefore, the camera unit 3 as a whole can be rotated up and down in the direction indicated by the arrow K in the drawing, that is, in the mounted state. The camera unit 3 contacts the frame 2 at the lower end of the rotation range. The upper end of the rotation range of the camera unit 3 is determined by the specifications of the hinges 21A and 23A.

The camera pedestal portion 3C is a plate-like or bar-like member that extends over the right part 2A, the left part 2B, and the bridge part 2C, and the upper camera 61 is embedded in a position corresponding to the bridge part 2C. It is installed complicatedly. The upper camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like, and may be a monocular camera or a stereo camera.
The upper camera 61 images at least a part of the outside scene in the front side direction of the head-mounted display device 100, in other words, the user's visual field direction when the image display unit 20 is mounted. The width of the angle of view of the upper camera 61 can be set as appropriate. For example, at the lower end of the rotation range of the camera unit 3, the imaging range of the upper camera 61 is the right optical image display unit 26, the left optical image. It is preferable that the outside world visually recognized through the display part 28 is included. Furthermore, it is more preferable that the imaging range of the upper camera 61 is set so that the entire field of view of the user through the light control plate 20A can be imaged.
The upper camera 61 executes imaging in accordance with the control of the imaging processing unit 181 (FIG. 4) included in the control unit 140, and outputs captured image data to the imaging processing unit 181.

  The image display unit 20 is connected to the control device 10 via the connection unit 40. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control device 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the housing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and is connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collecting unit of the microphone 63 faces the user's line of sight, collects sound, and sends the sound signal to the sound processing unit 187 (FIG. 4). Output. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

  The right cord 42, the left cord 44, and the main body cord 48 only need to be capable of transmitting digital data, and can be constituted by, for example, a metal cable or an optical fiber. Further, the right cord 42 and the left cord 44 may be combined into a single cord.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. The end of the main body cord 48 opposite to the connecting member 46 and the control device 10 are provided with connectors (not shown) that fit together. By fitting the connector of the main body cord 48 and the connector of the control device 10 or removing this fitting, the control device 10 and the image display unit 20 can be contacted and separated.

  The control device 10 controls the head-mounted display device 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 that is operated by a user with fingers.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 includes a light source such as an LED (Light Emitting Diode), and notifies the operating state (for example, power ON / OFF) of the head-mounted display device 100 according to the lighting state of the light source. The display switching key 13 outputs, for example, a signal instructing switching of the image display mode in response to the pressing operation.

  The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or the like can be adopted. The luminance switching key 15 outputs a signal instructing increase / decrease of the luminance of the image display unit 20 in response to the pressing operation. The direction key 16 outputs an operation signal in response to a pressing operation on a key corresponding to the up / down / left / right direction. The power switch 18 is a switch for switching power on / off of the head-mounted display device 100.

FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. FIG. 2 shows the user's left eye LE and right eye RE for explanation.
The left display drive unit 24 includes a left backlight 222, a left LCD 242, and a left projection optical system 252. The left backlight 222 has a light source such as an LED and a diffusion plate. The left LCD 242 is a transmissive liquid crystal panel that is arranged on the optical path of light emitted from the diffusion plate of the left backlight 222 and has a plurality of pixels arranged in a matrix. The left projection optical system 252 includes a lens group that guides the image light L transmitted through the left LCD 242.

The left projection optical system 252 includes a collimator lens that converts the image light L emitted from the left LCD 242 into a parallel light beam. The image light L converted into a parallel light beam by the collimating lens is incident on the left light guide plate 262 (optical element). The left light guide plate 262 is a prism formed with a plurality of reflecting surfaces that reflect the image light L, and the image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 262. The left light guide plate 262 is formed with a half mirror 262A (reflection surface) positioned in front of the left eye LE.
The image light L reflected by the half mirror 262A is emitted from the left optical image display unit 28 toward the left eye LE, and this image light L forms an image on the retina of the left eye LE so that the user can visually recognize the image.

  The right display driving unit 22 is configured to be symmetrical with the left display driving unit 24. The right display drive unit 22 includes a right backlight 221, a right LCD 241, and a right projection optical system 251. The right backlight 221 includes a light source such as an LED and a diffusion plate. The right LCD 241 is a transmissive liquid crystal panel that is arranged on the optical path of light emitted from the diffusion plate of the right backlight 221 and has a plurality of pixels arranged in a matrix. The right projection optical system 251 includes a lens group that guides the image light L transmitted through the right LCD 241.

The right projection optical system 251 has a collimator lens that converts the image light L emitted from the right LCD 241 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimating lens is incident on the right light guide plate 261 (optical element). The right light guide plate 261 is a prism formed with a plurality of reflecting surfaces that reflect the image light L, and the image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 261. The right light guide plate 261 is formed with a half mirror 261A (reflection surface) located in front of the right eye RE.
The image light L reflected by the half mirror 261A is emitted from the right optical image display unit 26 toward the right eye RE, and this image light L forms an image on the retina of the right eye RE to make the user visually recognize the image.

  The image light L reflected by the half mirror 261A and the external light OL transmitted through the light control plate 20A are incident on the right eye RE of the user. The image light L reflected by the half mirror 262A and the external light OL transmitted through the light control plate 20A are incident on the left eye LE. As described above, the head-mounted display device 100 causes the image light L and the external light OL of the internally processed image to overlap each other and enter the user's eyes, and for the user, seeing through the light control plate 20A. An outside scene can be seen, and an image by the image light L is visually recognized over the outside scene. Thus, the head-mounted display device 100 functions as a see-through display device.

  The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “left light guide unit”, and the right projection optical system 251 and the right light guide plate 261 are collectively referred to as “right light guide unit”. The configuration of the right light guide and the left light guide is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light. It may be used, or a semi-transmissive reflective film may be used.

  As shown in FIGS. 1 and 2, two motion sensors are attached to the frame 2. The motion sensor of the first embodiment is an inertial sensor, specifically, a first sensor 66 and a second sensor 68. The first sensor 66 is disposed at the end on the temple portion 2D side in the right portion 2A, and the second sensor 68 is disposed at the end on the temple portion 2E side in the left portion 2B. The first sensor 66 and the second sensor 68 are inertial sensors such as an acceleration sensor and an angular velocity sensor (gyro sensor). In the present embodiment, the first sensor 66 and the second sensor 68 are triaxial gyro sensors. The first sensor 66 and the second sensor 68 are respectively rotated around the X axis (pitch), rotated around the Y axis (yaw), and rotated around the Z axis (described later) at the measurement reference point of the built-in detection mechanism. Roll). In the following description, the positions of the first sensor 66 and the second sensor 68 are the positions of measurement reference points.

FIG. 3 is a plan view schematically showing a state in which the image display unit 20 is mounted on the user's head.
One of the first sensor 66 and the second sensor 68 is on one side of the center of the user's head, and the other sensor is on the other side of the center of the user's head. Specifically, the first sensor 66 is on the right side of the user's head and the second sensor 68 is on the left side.
In the present embodiment, the center of the head refers to the center of the head in a horizontal plane perpendicular to the height of the user. The positions of the first sensor 66 and the second sensor 68 in the horizontal plane are on the right side and the left side of the center of the head in the horizontal plane.
The head-mounted display device 100 may specify the center of the head in the height direction (height direction). In this case, for example, the center in the horizontal plane at the height position of both eyes It may be the “center of the part”. Further, for example, the three-dimensional center of the user's head may be obtained using the head from the top of the head to the upper end of the cervical vertebra, and this center may be used as the “head center”.
The center position of the user's head may be obtained by actual measurement, or may be estimated from the height of the user and the size of the head circumference. For example, when a parameter or calculation formula that estimates the center position of the head from height or head circumference is obtained in advance by a statistical method, the center position of the head is calculated using this parameter or calculation formula. May be.

More preferably, as shown in the plan views of FIGS. 2 and 3, in the present embodiment, the first sensor 66 is disposed on the side of the right light guide plate 261, and the second sensor 68 is on the side of the left light guide plate 262. Placed in. Assuming that the center position in the left-right direction of the frame 2 is C1, the center C2 of the half mirror 261A, which is a display area for allowing the user to visually recognize the image light, and the center C3 of the half mirror 262A are symmetrical with respect to the center position C1. In position. In other words, the center position C1 is at the midpoint between the center C2 of the half mirror 261A and the center C3 of the half mirror 262A.
Thus, it is preferable that the first sensor 66 and the second sensor 68 are provided at positions symmetrical with respect to the center position C1. A straight line connecting the position of the first sensor 66 and the position of the second sensor 68 passes through the half mirrors 261A and 262A. That is, the position of the first sensor 66, the position of the second sensor 68, the center C2 of the half mirror 261A, and the center C3 of the half mirror 262A are aligned on the same straight line.

  Although the positional relationship in the horizontal plane including the centers of the half mirrors 261A and 262A has been described above, the positional relationship in the vertical direction (height direction) perpendicular to the horizontal plane is not particularly limited. It is preferable that the positions of the first sensor 66 and the second sensor 68 and the centers of the half mirrors 261A and 262A are close to each other in the vertical direction. For example, the positions of the first sensor 66 and the second sensor 68 may be on the side of the half mirrors 261A and 262A. Furthermore, it is more preferable that the positions of the first sensor 66 and the second sensor 68 and the centers of the half mirrors 261A and 262A are at the same position in the height direction.

As shown in FIG. 1, the first sensor 66 detects the angular velocity with respect to the head of the user wearing the image display unit 20 in the left-right direction as the X axis, the front-rear direction as the Y axis, and the vertical direction as the Z axis. And More specifically, when the head-mounted display device 100 is worn, the image display unit 20 is in a horizontal position that is perceived by the user with respect to the left and right eyes. In this wearing state, the detection axes (X axis, Y axis, Z axis) of the first sensor 66 and the second sensor 68 are configured to coincide with the left, right, front, back, and top, as perceived by the user. If the mounting position of the image display unit 20 tilts or deviates with respect to the user's head, the detection axes of the first sensor 66 and the second sensor 68 deviate from the left, right, front, back, and top. The problem is easily solved by the user adjusting the inclination and displacement of the image display unit 20.
The head-mounted display device 100 detects the movement of the user's head while wearing the image display unit 20 by the first sensor 66 and the second sensor 68. The movements detected by these motion sensors are movements at the measurement reference point (P3 in FIG. 3) of the first sensor 66 and the measurement reference point (P4 in FIG. 3) of the second sensor 68.

The head-mounted display device 100 detects the movement of the user's head (head tracking) and displays the detected user's head when the AR content is displayed by the function of the AR display control unit 186 described later. The display mode is changed corresponding to the movement of the.
In head tracking, it is desirable to assume a reference position of head movement (movement) and obtain the movement at this reference position. As described above, when a human moves the head, the center of the head (corresponding to position P1 in FIG. 3) or the front of the head (corresponding to position P2 in FIG. 3) Often becomes. Here, the center of motion or the reference position is called the motion center. When the head-mounted display device 100 detects a motion, the motion is detected not only when the user's head moves, but also when the user's head and a body other than the head move together. . Therefore, the movement detected by the head-mounted display device 100 is the movement of the head, but the movement center of this movement may be outside the user's head.

  In the head tracking process, the motion at the motion center is obtained. In order to directly detect the movement of the movement center, it is necessary to arrange a movement sensor at a position where the movement center is likely to be, for example, the position P1 or the position P2, which is not realistic. Therefore, the head-mounted display device 100 calculates a motion at the center of motion through arithmetic processing based on the detection results of the first sensor 66 and the second sensor 68. Arithmetic formulas, tables, parameters, and the like used for this arithmetic processing are stored in advance as sensor position data 122 described later.

When the positions of the first sensor 66 and the second sensor 68 satisfy a certain condition, the arithmetic processing for calculating the motion at the motion center becomes easy, can be executed with a light load, and more accurately the motion center. The motion at (for example, positions P1, P2) can be calculated.
As described above, this condition is that one of the first sensor 66 and the second sensor 68 is on one side of the center of the user's head, and the other sensor is on the center of the user's head. It is on the other side. In this case, when the motion centered on the center P1 of the head is performed, the motion at the motion center can be easily calculated.

More preferably, the first sensor 66 is disposed on the side of the right light guide plate 261, and the second sensor 68 is disposed on the side of the left light guide plate 262. Furthermore, it is more preferable that the first sensor 66 and the second sensor 68 are provided at positions symmetrical with respect to the central position C1. More preferably, the position of the first sensor 66, the position of the second sensor 68, the center C2 of the half mirror 261A, and the center C3 of the half mirror 262A are aligned on the same straight line. In this case, when the motion centered on the head center P1 or the front center P2 is performed, the motion at the motion center can be easily calculated.
The positional relationship in the vertical direction (height direction) is preferably such that the positions of the first sensor 66 and the second sensor 68 and the centers of the half mirrors 261A and 262A are close to each other in the vertical direction. Furthermore, it is more preferable that the positions of the first sensor 66 and the second sensor 68 and the centers of the half mirrors 261A and 262A are at the same position in the height direction.

FIG. 4 is a functional block diagram of each part constituting the head-mounted display device 100.
The head-mounted display device 100 includes an interface 125 for connecting various external devices OA that are content supply sources. As the interface 125, for example, an interface corresponding to wired connection such as a USB interface, a micro USB interface, a memory card interface, or the like may be used, and the interface 125 may be configured by a wireless communication interface. The external device OA is an image supply device that supplies an image to the head-mounted display device 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used.

The control device 10 includes a control unit 140, an input information acquisition unit 110, a storage unit 120, a transmission unit (Tx) 51, and a transmission unit (Tx) 52.
The input information acquisition unit 110 is connected to the operation unit 135. The operation unit 135 includes the track pad 14, the direction key 16, the power switch 18, and the like. The input information acquisition unit 110 acquires input contents based on a signal input from the operation unit 135. The control device 10 includes a power supply unit (not shown), and supplies power to each unit of the control device 10 and the image display unit 20.

The storage unit 120 is a nonvolatile storage device, and stores various computer programs and data related to these programs. The storage unit 120 may store still image data or moving image data to be displayed on the image display unit 20.
The storage unit 120 stores setting data 121. The setting data 121 includes setting values relating to various processes executed by the control unit 140. For example, it includes setting values such as resolution when the image processing unit 160 and the display control unit 170 process image signals. The setting value included in the setting data 121 may be a value input in advance by operating the operation unit 135, or a setting value from an external device OA or another device (not shown) via the communication unit 117 or the interface 125. May be received and stored.

  The storage unit 120 also stores sensor position data 122 and content data 123. The sensor position data 122 includes arithmetic expressions, parameters, and the like used by the motion detection unit 185 described later for arithmetic processing. The content data 123 includes image (still image or moving image) data and / or audio data of content displayed by the AR display control unit 186 as AR.

Sensor 113, GPS 115, and communication unit 117 are connected to control unit 140. The sensor 113 includes an inertial sensor such as an acceleration sensor or an angular velocity sensor, and the control unit 140 acquires a detection value of the sensor 113. The sensor 113 may be configured as a nine-axis sensor having, for example, a three-axis acceleration sensor, a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis magnetic sensor.
The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and calculates the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. The GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140.

The communication unit 117 executes wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), or Bluetooth (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires the content data 123 from the communication unit 117 and causes the image display unit 20 to display an image. On the other hand, when the external device OA is wired to the interface 125, the control unit 140 acquires the content data 123 from the interface 125 and causes the image display unit 20 to display an image. The communication unit 117 and the interface 125 function as a data acquisition unit DA that acquires the content data 123 from the external device OA.

  The control unit 140 stores a CPU (not shown) for executing a program, a RAM (not shown) for temporarily storing a program and data executed by the CPU, and a basic control program and data executed by the CPU in a nonvolatile manner. ROM (not shown). The control unit 140 controls each unit of the head-mounted display device 100 by executing a control program by the CPU. In addition, the control unit 140 reads out and executes the computer program stored in the storage unit 120 and realizes various functions of the control unit 140. That is, it functions as an operating system (OS) 150, an image processing unit 160, a display control unit 170, an imaging processing unit 181, a detection control unit 183, a motion detection unit 185, an AR display control unit 186, and an audio processing unit 187.

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). The image processing unit 160 stores the converted digital image signal in the RAM of the control unit 140 for each frame as image data of the target image (Data in the figure). This image data is, for example, RGB data.
Note that the image processing unit 160 may perform a resolution conversion process for converting the resolution of the image data into a resolution suitable for the right display driving unit 22 and the left display driving unit 24 as necessary. The image processing unit 160 also performs image adjustment processing for adjusting the brightness and saturation of image data, 2D image data is created from 3D image data, or 2D / 3D conversion processing is performed to generate 3D image data from 2D image data. May be executed.

  The image processing unit 160 transmits the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data stored in the RAM via the transmission units 51 and 52, respectively. The transmission units 51 and 52 function as a transceiver and execute serial transmission between the control device 10 and the image display unit 20. The image data Data transmitted via the transmission unit 51 is referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is referred to as “left eye image data”.

  The display control unit 170 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24, and generates and generates image light by each of the right display drive unit 22 and the left display drive unit 24 based on the control signal. Control injection. Specifically, the right LCD 241 is controlled to be turned on / off by the right LCD control unit 211, and the right backlight 221 is turned on / off by the right backlight control unit 201. In addition, the display control unit 170 controls the driving of the left LCD 242 by the left LCD control unit 212 and the driving of the left backlight 222 by the left backlight control unit 202.

The imaging processing unit 181 controls the upper camera 61 to execute imaging, and acquires captured image data.
The detection control unit 183 drives each of the first sensor 66 and the second sensor 68 to acquire a detection value. For example, the first sensor 66 and the second sensor 68 are initialized when the head-mounted display device 100 is turned on, for example. In addition, the detection control unit 183 acquires the detection values of the first sensor 66 and the second sensor 68 at a preset sampling cycle. In the present embodiment, the first sensor 66 and the second sensor 68 are detected values of angular velocity of rotation around the X axis (pitch), rotation around the Y axis (yaw), and rotation around the Z axis (roll), respectively. Is output. This detection value may include the magnitude of the angular velocity and the rotation direction.

  The motion detection unit 185 calculates a motion amount at the center of motion based on the detection values of the first sensor 66 and the second sensor 68 acquired by the detection control unit 183. The motion center is the center of the user's head movement or the reference position obtained from the movement detected by the first sensor 66 and the second sensor 68. For example, when the user's head rotates, the center of rotation corresponds to the center of motion. For example, when the user's head goes straight, the position of the movement center is not fixed at a specific point. In this case, the center of the head (P1 in FIG. 3) is used as the movement center for convenience.

  The motion detection unit 185 performs head tracking calculation using the sensor position data 122 stored in the storage unit 120. The sensor position data 122 includes data indicating a relative positional relationship or an absolute position between the position P3 of the first sensor 66 and the position P4 of the second sensor 68. The sensor position data 122 may include data indicating a positional relationship between each of the positions P3 and P4 and the position P1 and / or the position P2. The sensor position data 122 includes arithmetic expressions, tables, parameters, and the like used to calculate the amount of motion and the direction of motion at the motion center from the detection values of the first sensor 66 and the second sensor 68. Here, the motion detection unit 185 may calculate the position of the motion center based on the detection results of the first sensor 66 and the second sensor 68. Alternatively, the motion amount at the motion center may be calculated without calculating the position of the motion center. The movement amount may be a speed, may include a speed and time, or may be a movement amount.

  For example, the motion detection unit 185 obtains a component around the X axis, a component around the Y axis, and a component around the Z axis from the detected values of the angular velocities of the first sensor 66 and the second sensor 68. Then, the motion detection unit 185 is based on the detection values of the first sensor 66 and the second sensor 68 and the positions of the first sensor 66 and the second sensor 68 of the sensor position data 122, and the center of rotation around the X axis, The center of rotation about the Y axis and the center of rotation about the Z axis are obtained. The motion detection unit obtains the center of the three-dimensional motion, that is, the position of the motion center, the direction of motion at the motion center, and the magnitude (or strength) of the motion center.

  Further, the motion detection unit 185 may obtain not only the direction and magnitude of movement at the center of movement, but also the direction and / or magnitude of movement at a point designated in advance. For example, the direction and / or magnitude of the movement at the position P1 may be obtained. When the motion detection unit 185 is set to obtain the direction and / or magnitude of the movement of the position P1, if the movement center does not overlap the position P1, the movement detection unit 185 determines the movement direction and / or magnitude from the movement center. The direction and / or magnitude of the movement of the position P1 is calculated and output. The position P1 shown in FIG. 3 indicates the position in the horizontal plane, but the direction and / or magnitude of the movement may be obtained at a three-dimensional position that specifies the height direction (Z-axis direction) of the user. .

  The AR display control unit 186 reads the content data 123 stored in the storage unit 120, controls the image processing unit 160 and the display control unit 170, and causes the image display unit 20 to display an image for AR display. In addition, when the content data 123 includes audio data, the AR display control unit 186 controls the audio processing unit 187 to output the audio of the content from the right earphone 32 and the left earphone 34.

The AR display control unit 186 controls the display of the AR content based on the data indicating the movement direction and the movement amount of the movement center output from the movement detection unit 185.
The AR display control unit 186 displays the AR content while the user is looking at the object through the image display unit 20. The AR display control unit 186 provides information about the object by performing AR display that displays an image, characters, or the like at a position corresponding to the object, or the shape of the object seen through the image display unit 20 Change how it looks. The AR content includes image and character data displayed at a position corresponding to the object. In addition, the AR content may include data for specifying an object and data regarding the display position of an image or a character. The display position of the AR content may be a position overlapping the target object or may be around the target object. The object may be an object, may be a real estate such as a building, may be a moving body such as an automobile or a train, or may be a living organism such as a human being or an animal. The AR display control unit 186 detects an object located in the user's field of view from the captured image data acquired by the imaging processing unit 181. Then, the AR display control unit 186 determines the display position of the AR content corresponding to the detected object, and displays the AR content.

  The AR display control unit 186 performs processing for changing the display position of the AR content and processing for switching the AR content in accordance with the movement of the head of the user wearing the image display unit 20. For example, when the user's head rotates around the Z axis at an angular velocity that exceeds a set value, the AR display control unit 186 deletes the display of the AR content. This is because, when there is such a head movement, the user's field of view moves greatly, and there is a high possibility that the object of AR display will be out of the field of view. For example, when the user's head moves in the X-axis direction, the AR display control unit 186 enlarges or reduces the display size of the AR content. When there is such a head movement, the user can see the AR display object larger. For this reason, the AR display control unit 186 may enlarge the AR content to make the AR content easier to see, and may reduce the AR content so that the object can be easily seen.

A mode in which the AR display control unit 186 changes the display mode of the AR content in accordance with the movement of the user's head is set in advance as a setting value included in the setting data 121. Further, it may be set to a program executed by the control unit 140.
Here, the display control of the AR display control unit 186 is designed as a process corresponding to the movement at the center of the user's head (P1 in FIG. 3) or the center of the front of the head (P2 in FIG. 3). The This is because if the movement of the center of the head or the center of the front of the head is matched, it is easy to imagine a display change when designing it, and it is simple, error-free, and an effective display change can be designed. For this reason, in the setting data 121, a set value of the movement at the center of the user's head or the center of the front of the head is set. The AR display control unit 186 processes the movement at the center of the user's head or the front of the head calculated by the motion detection unit 185 in comparison with the setting data 121.

As a method different from the above, it is also possible to design the display control of the AR display control unit 186 corresponding to the movement at the position of the first sensor 66 and the second sensor 68 that actually detect the movement. For example, setting values related to the detection values of the first sensor 66 and the second sensor 68 are set in the setting data 121, and the AR display control unit 186 performs processing based on the setting values. Also in this case, the display can be controlled in accordance with the movement of the user's head. However, this set value is a value fixed to the positions of the first sensor 66 and the second sensor 68, and is newly set when the position or number of sensors changes due to the specification change of the head-mounted display device 100. A value is required.
On the other hand, in the method of calculating the motion reference position from the detection values of the first sensor 66 and the second sensor 68 and the motion at the reference position, the set value is not a device-specific value but has versatility. . Therefore, there is an advantage that control of the display mode of the AR content is facilitated and effective AR display is possible.

  The audio processing unit 187 acquires the audio signal of the content under the control of the AR display control unit 186, amplifies the acquired audio signal, and outputs the amplified audio signal to the right earphone 32 and the left earphone 34. In addition, the sound processing unit 187 acquires sound collected by the microphone 63 and converts it into digital sound data. The voice processing unit 187 may perform a preset process on the digital voice data.

  The image display unit 20 includes an interface 25, a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, and a left light guide plate 262 as a left optical image display unit 28. With. The image display unit 20 includes the upper camera 61, the first sensor 66, and the second sensor 68 described above. The upper camera 61 is provided in the camera unit 3 separate from the frame 2 (FIG. 1), but is connected to the interface 25.

The interface 25 includes a connector to which the right cord 42 and the left cord 44 are connected. The interface 25 outputs the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data transmitted from the transmission unit 51 to the corresponding reception units (Rx) 53 and 54. Further, the interface 25 outputs a control signal transmitted from the display control unit 170 to the corresponding reception units 53 and 54, the right backlight control unit 201, or the left backlight control unit 202.
The interface 25 is an interface for connecting the upper camera 61, the first sensor 66, and the second sensor 68. Imaged image data and imaging signals of the upper camera 61, detection results of the first sensor 66 and the second sensor 68, and the like are sent to the control unit 140 via the interface 25.

  The right display drive unit 22 includes the right backlight 221, the right LCD 241, and the right projection optical system 251 described above. The right display driving unit 22 includes a receiving unit 53, a right backlight (BL) control unit 201 that controls the right backlight (BL) 221, and a right LCD control unit 211 that drives the right LCD 241.

  The reception unit 53 operates as a receiver corresponding to the transmission unit 51 and executes serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data input via the reception unit 53. .

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes the left backlight 222, the left LCD 242 and the left projection optical system 252 described above. The left display driving unit 24 includes a receiving unit 54, a left backlight control unit 202 that drives the left backlight 222, and a left LCD control unit 212 that drives the left LCD 242.

The reception unit 54 operates as a receiver corresponding to the transmission unit 52 and executes serial transmission between the control device 10 and the image display unit 20. The left backlight control unit 202 drives the left backlight 222 based on the input control signal. The left LCD control unit 212 drives the left LCD 242 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data Data input via the reception unit 54. .
The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as a right “image light generation unit”. Similarly, the left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are collectively referred to as a left “image light generation unit”.

FIG. 5 is a flowchart showing the operation of the head-mounted display device 100, and particularly shows the operation in which the control unit 140 obtains the movement of the user's head.
The control unit 140 starts detection of the sensor when the head-mounted display device 100 is turned on, when displaying the AR content, or when the start is instructed by the operation of the operation unit 135 (Step S140). S11). The motion detection unit 185 of the control unit 140 acquires the sensor position data 122 from the storage unit 120 (step S12), and the detection control unit 183 acquires the detection values of the first sensor 66 and the second sensor 68 (step S13). .

  The motion detection unit 185 associates the detection value of the first sensor 66 acquired by the detection control unit 183 with the data related to the position of the first sensor 66 included in the sensor position data 122 (step S14). In step S <b> 14, the motion detection unit 185 associates the detection value of the second sensor 68 acquired by the detection control unit 183 with the data related to the position of the second sensor 68 included in the sensor position data 122.

  The detection control unit 183 acquires detection values of the first sensor 66 and the second sensor 68 at a preset sampling frequency. When the control unit 140 repeatedly executes steps S11 to S17 in FIG. 5 and the execution cycle is longer than the sampling cycle of the detection control unit 183, the detection control unit 183 includes the total value, average value, A value such as a median value may be output in step S13. In this case, in step S14, the motion detection unit 185 associates values such as a total value, an average value, and a median value obtained from a plurality of detection values with data relating to the position of each sensor.

  Based on the sensor position data 122, the motion detection unit 185 calculates the direction of motion and the amount of motion at the motion center from the detection values of the first sensor 66 and the second sensor 68 (step S15). The motion detection unit 185 outputs data relating to the calculated direction of motion and the amount of motion to the AR display control unit 186 (step S16), and proceeds to step S17.

  In step S17, the control unit 140 determines whether or not a motion detection end condition is satisfied (step S17). If the end condition is satisfied (step S17; YES), the process ends. If the end condition is not satisfied (step S17; NO), the process returns to step S12.

  In the process of FIG. 5, the detection values of the first sensor 66 and the second sensor 68 are acquired via the interface 25 by the operation of the detection control unit 183 and processed by the motion detection unit 185. Here, when the detection value acquired by the detection control unit 183 and processed by the motion detection unit 185 is added with information specifying the detected sensor and information indicating the timing at which the detection or detection value is acquired. Good. In this case, when the first sensor 66 and / or the second sensor 68 outputs the detection value to the interface 25, data indicating whether the detected sensor is the first sensor 66 or the second sensor 68 and the timing are indicated. Data may be added. Alternatively, when the detection control unit 183 obtains the detection value of the first sensor 66 and / or the second sensor 68, the detection sensor 183 determines whether the detected sensor is the first sensor 66 or the second sensor 68 and the timing. You may add the data shown. Further, when the motion detection unit 185 acquires the detection value from the detection control unit 183, data indicating whether the detected sensor is the first sensor 66 or the second sensor 68 and data indicating the timing are added. Also good.

  As described above, the head-mounted head mounted display device 100 mounted on the user's head includes the image display unit 20 that irradiates the user's eyes with image light, and a plurality of motion sensors. The first sensor 66 and the second sensor 68 are provided. One of the first sensor 66 and the second sensor 68 is located on one side from the center of the head in the mounted state, and the other motion sensor is located on the other side from the center of the head. For this reason, when the user's head moves, the amount of movement and the direction of movement at the center of movement can be quickly obtained.

  In the head-mounted display device 100, one of the first sensor 66 and the second sensor 68 is located on the left side from the center of the head in the mounted state, and the other motion sensors are located on the right side from the center of the head. To position. For this reason, based on the detection result of the motion sensor, the motion at the motion center of the head can be quickly obtained.

  The image display unit 20 includes a right light guide plate 261 and a left light guide plate 262 having half mirrors 261A and 262A that emit image light toward the user's eyes. The first sensor 66 and the second sensor 68 and the center positions of the half mirrors 261A and 262A are arranged in a straight line. Therefore, based on the detection results of the first sensor 66 and the second sensor 68, the motion at the motion center corresponds to the motion with the positions of the right light guide plate 261 and the left light guide plate 262 constituting the display area as the motion center. The amount and direction of movement can be quickly obtained.

  The right light guide plate 261 and the left light guide plate 262 include half mirrors 261A and 262A as display areas that emit image light toward the user's eyes. With a configuration including an optical element that emits image light using a half mirror, a plurality of motion sensors are arranged in a straight line with the center position of the half mirror. For this reason, it is possible to promptly determine the amount of movement, the direction of movement, and the like at the center of motion corresponding to the motion centered on the center of the image visually recognized by the user.

Further, the image display unit 20 includes a glasses-type frame 2, and the first sensor 66 and the second sensor 68 are in a symmetrical position with respect to the center of the frame 2. Corresponding to the motion centered on the center of the eyeglass-shaped frame 2, the amount of motion and the direction of motion at the motion center can be quickly obtained.
The frame 2 includes a right part 2A located in front of the user's right eye, a left part 2B located in front of the user's left eye, and a bridge part 2C that connects the right part 2A and the left part 2B. And have. The first sensor 66 and the second sensor 68 are located symmetrically with respect to the bridge portion 2C. For this reason, it is possible to promptly determine the amount of movement, the direction of movement, and the like at the center of motion corresponding to the motion centered on the frame 2.

  In addition, when the head moves in a complicated manner, such as when the user moves the head and the body together, the movement center of the head may be plural. For example, if a rotational movement around the position P1 and a rotational movement around the user's waist occur at the same time, the position P1 and the waist position are the centers of motion, respectively. In this case, the motion detection unit 185 may determine the amount of motion and the direction of motion at each of the plurality of motion centers, or may determine the position of each motion center. Further, there may be a case where the motions at all the motion centers cannot be obtained from the detection results of the first sensor 66 and the second sensor 68. This is because when there are a plurality of motion centers, there are more variables in the arithmetic expression for obtaining the motion at the motion center. In such a case, the motion detection unit 185 may cause the detection control unit 183 to acquire the detection value of each sensor a plurality of times with a time interval, and obtain a motion at the center of motion based on the detection values of the plurality of times.

In the first embodiment, as an example of the motion sensor, the configuration in which the first sensor 66 and the second sensor 68 that are two inertial sensors are provided is described as an example. The present invention is not limited to this, and an optical sensor can be used as the motion sensor.
For example, the motion at the motion center may be obtained using a plurality of motion sensors including an inertial sensor and an optical sensor. Further, for example, the motion at the center of motion may be obtained using a plurality of optical sensors.
These cases will be described as a second embodiment.

[Second Embodiment]
FIG. 6 is a functional block diagram of each part constituting the head-mounted display device 100A according to the second embodiment.
The head-mounted display device 100A according to the second embodiment includes an image display unit 200 instead of the image display unit 20. The image display unit 200 has a configuration in which the first camera 64 and the second camera 65 are provided in the image display unit 20, and the other units have a common configuration. Accordingly, in the head-mounted display device 100A, components that are the same as those in the head-mounted display device 100 of the first embodiment are denoted by the same reference numerals, and illustration and description thereof are omitted.

  The first camera 64 and the second camera 65 are digital cameras each including an imaging element such as a CCD or CMOS, an imaging lens, and the like, and may be a monocular camera or a stereo camera. The first camera 64 and the second camera 65 are not limited to cameras that receive visible light, and may be cameras that capture images with light outside the visible region, such as ultraviolet cameras and infrared cameras. The first camera 64 and the second camera 65 are examples of optical sensors and function as motion sensors.

  The first camera 64 is disposed in front of the first sensor 66 in the right part 2A. The second camera 65 is installed in front of the second sensor 68 in the left part 2B. The angle of view of the first camera 64 includes a range that is visible to the right eye of the user through the right optical image display unit 26, and the angle of view of the second camera 65 is transmitted through the left optical image display unit 28. It is preferable that the range visually recognized by the user's left eye is included. The angle of view of the first camera 64 and the second camera 65 may include a range that overlaps the angle of view of the upper camera 61 or may include a range that does not overlap.

  The imaging processing unit 181 controls the first camera 64 and the second camera 65 to execute imaging, and acquires captured image data. The sensor position data 122 stored in the storage unit 120 includes data related to the positions of the first camera 64 and the second camera 65. The motion detection unit 185 obtains the direction and amount of motion at the position of the first camera 64 or the second camera 65 from the captured image data acquired by the imaging processing unit 181.

In the head-mounted display device 100A, the movement at the center of motion can be obtained by combining the first camera 64, the second camera 65, the first sensor 66, and the second sensor 68, which are four motion sensors.
There are various combinations of motion sensors, but it is desirable to combine at least a motion sensor on the right side of the user's head and a motion sensor on the left side of the user's head. In the head-mounted display device 100A, the first camera 64 and the second camera 65, the first camera 64 and the second sensor 68, the first sensor 66 and the second camera 65, and the first sensor 66 and the second sensor 68. There are four combinations. Among these, the example using the first sensor 66 and the second sensor 68 has been described in the first embodiment.

In the above combination, the motion sensor on the right side of the selected user's head is the right sensor, and the motion sensor on the left side of the user's head is the left sensor.
The position of the right sensor is preferably the position of the first sensor 66 described in the first embodiment, and the position of the left sensor is preferably the position of the second sensor 68 described in the first embodiment. That is, when the center position in the left-right direction of the frame 2 is C1, it is preferable that the right sensor and the left sensor are provided at positions symmetrical with respect to the center position C1. Further, it is more preferable that a straight line connecting the position of the right sensor and the position of the left sensor passes through the half mirrors 261A and 262A. More preferably, the position of the right sensor, the position of the left sensor, the center C2 of the half mirror 261A, and the center C3 of the half mirror 262A are arranged on the same straight line.

  The positional relationship in the horizontal plane including the centers of the half mirrors 261A and 262A is as described above. In the vertical direction (height direction) perpendicular to the horizontal plane, it is preferable that the positions of the right sensor and the left sensor are close to the centers of the half mirrors 261A and 262A in the vertical direction. For example, the right sensor and the left sensor may be on the side of the half mirrors 261A and 262A. Furthermore, it is more preferable that the positions of the right sensor and the left sensor and the centers of the half mirrors 261A and 262A are in the same position in the height direction.

In the present embodiment, a configuration including all of the first camera 64, the second camera 65, the first sensor 66, and the second sensor 68 will be described as an example. This is merely an example, and the present invention can be applied to any configuration provided with one motion sensor corresponding to the right sensor and one motion sensor corresponding to the left sensor. Accordingly, the configuration may be such that one or two of the first camera 64, the second camera 65, the first sensor 66, and the second sensor 68 are omitted.
The sensor position data 122 includes data indicating a relative positional relationship or an absolute position of a motion sensor used in an operation of detecting a motion at the motion center among the motion sensors included in the head-mounted display device 100A.

FIG. 7 is a flowchart showing the operation of the head-mounted display device 100A of the second embodiment, and particularly shows the operation in which the control unit 140 obtains the movement of the user's head.
In the operation of FIG. 7, an example is described in which a motion sensor used in a process for obtaining head movement is selected from the first camera 64, the second camera 65, the first sensor 66, and the second sensor 68 and can be set. To do. Also, in the flowchart of FIG. 7, the same steps as those in FIG.

  When the control unit 140 starts sensor detection (step S11), the motion detection unit 185 acquires sensor position data 122 (step S12). The motion detection unit 185 selects a motion sensor to be used for head tracking from the motion sensors included in the head-mounted display device 100A according to the settings of the sensor position data 122 and the input operation of the operation unit 135. (Step S23).

  The motion detection unit 185 determines whether or not the selected sensor includes a camera (step S21). When one or more of the first camera 64 and the second camera 65 is selected (step S21; YES), the imaging processing unit 181 executes imaging with the selected camera (step S22), and acquires captured image data. . The motion detection unit 185 analyzes the captured image data and detects the direction and amount of motion at the selected camera position (step S23). The motion detection unit 185 performs processing for associating the detected direction and amount of motion with the position of the camera (step S24).

  Here, for example, the motion detection unit 185 may obtain a difference between a plurality of captured images captured by the first camera 64 with a time interval, and obtain a motion direction and a motion amount from the obtained difference. Further, the imaging processing unit 181 may set the exposure time of the first camera 64 to a long time for motion detection. In this case, the captured image of the first camera 64 is blurred by reflecting the movement of the first camera 64. The motion detection unit 185 can detect the direction and amount of motion at the position of the first camera 64 based on the blur direction, blur amount, and exposure time of the captured image. The same applies to the image captured by the second camera 65.

The motion detection unit 185 determines whether or not the motion sensor selected in step S23 is only a camera (step S25). When any motion sensor other than the camera, that is, neither the first sensor 66 nor the second sensor 68 is selected (step S25; YES), the motion detection unit 185 proceeds to step S26.
On the other hand, when at least one of the first sensor 66 and the second sensor 68 is selected in step S23 (step S25; NO), the motion detection unit 185 proceeds to step S13. Moreover, the motion detection part 185 transfers to step S13, when neither the 1st camera 64 nor the 2nd camera 65 is selected by step S23 (step S21; NO).

  In step S13, the detection control unit 183 acquires the detection value of the first sensor 66 and / or the second sensor 68 (step S13). The motion detection unit 185 associates the detection value acquired by the detection control unit 183 with the data regarding the sensor position included in the sensor position data 122 (step S14). Thereafter, the control unit 140 proceeds to step S26.

  In step S26, the motion detection unit 185 calculates the direction and amount of motion at the center of motion based on the sensor position data 122 (step S26). In step S26, the motion amount and position associated in step S24 and / or the motion amount and position associated in step S14 are used for the calculation process. Thereafter, the motion detection unit 185 outputs data regarding the calculated direction of motion and the amount of motion to the AR display control unit 186 (step S16), and proceeds to step S17. In step S17, the control unit 140 determines whether or not an end condition is satisfied (step S17). If the end condition is satisfied (step S17; YES), the process ends. If the end condition is not satisfied (step S17; NO), the process returns to step S12.

Thus, the head-mounted display device 100A is not limited to the first sensor 66 and the second sensor 68 that are inertial sensors, and uses the first camera 64 and / or the second camera 65 that are optical sensors. Find the movement in the movement center. That is, the head tracking can be executed with respect to the movement of the user's head by the combination of the optical sensor and the motion sensor including the inertial sensor. In this case, for example, the detection result of the sensor can be used for other purposes, for example, the captured image data is used for detection of an AR display target.
Further, when any one of an optical sensor and an inertial sensor is used as the plurality of motion sensors, the head tracking process can be executed promptly with a lighter load by using the detection result of the same type of sensor.

  In the first and second embodiments, the movement is detected based on the detection results of the two motion sensors and the movement of the user's head is obtained. However, the detection results of three or more motion sensors are used. Of course it is possible.

  In the head-mounted display devices 100 and 100A, a plurality of detection results detected by one motion sensor can be processed as detection values of different motion sensors. About these examples, it shows in FIG. 8 as a modification.

FIG. 8 is an explanatory diagram showing a modification of the embodiment of the present invention, (A) is an explanatory diagram of a process of the first modification, and (B) is an explanatory diagram of a process of the second modification. .
In the example of FIG. 8A, the first sensor 66 included in the image display unit 20 is used. The head of the user wearing the image display unit 20 rotates as indicated by the arrow M around the position P1. During this rotation, the detection control unit 183 acquires the detection value of the first sensor 66 three times with a time difference. The movement of the arrow M continues at least during the period from the first detection to the third detection.
Position of the first sensor 66 at the time of the first detection (1) Position of the first sensor 66 at the time of the second detection (2) Position of the first sensor 66 at the time of the third detection (3). Position (1) can also be said to be the initial position. The positions (1), (2), and (3) are positions different from each other if they are positions relative to a reference that does not move, such as the ground, floor, or wall surface of the installation environment of the head-mounted display device 100.

In this case, the motion detection unit 185 associates the first detection value with the position (1), associates the second detection value with the position (2), and associates the third detection value with the position (3). By associating these, detection values of three sensors arranged at different positions are obtained in a pseudo manner. The position (2) can be calculated from, for example, the first and second detection time intervals, the first detection value, and the position (1). Similarly, the position (3) can also be calculated from the second and third detection time intervals, the second detection value, and the position (2).
Using the detection values of these three sensors, the direction and amount of movement at the movement center P1 can be obtained. The calculation result obtained in this example is the direction and amount of motion when the motion detected from the first time to the third time is regarded as one motion.

In the example of FIG. 8B, the head-mounted display device 100 uses the first sensor 66. Further, a virtual sensor 69 is set at a position where the second sensor 68 is provided in the first embodiment. Data indicating the relative position between the position of the first sensor 66 and the position of the virtual sensor 69 is included in the setting data 121.
The head of the user wearing the image display unit 20 rotates as indicated by the arrow M around the position P1. During this rotation, the detection control unit 183 acquires the detection value of the first sensor 66 twice with a time difference. The movement of the arrow M continues at least during the time including the first and second detection times. The position of the first sensor 66 at the first detection is defined as position (4), and the position of the first sensor 66 at the second detection is defined as position (5). The positions (4) and (5) are positions different from each other if the positions (4) and (5) are positions with respect to a reference that does not move, such as the ground, floor, or wall surface of the installation environment of the head-mounted display device 100. Position (4) can also be said to be the initial position.

The motion detection unit 185 associates the detection value of the first time with the position (4) and associates the detection value of the second time with the position (5), so that two sensors arranged in different positions are simulated. The detected value is obtained. The position (5) can be calculated from, for example, the first and second detection time intervals, the first detection value, and the position (4). Similarly, the position (3) can also be calculated from the second and third detection time intervals and the second detection value.
Furthermore, the motion detection unit 185 calculates the detection value of the virtual sensor 69 based on the position of the virtual sensor 69 at the time of the first detection (initial position of the virtual sensor 69). The virtual sensor 69 is a three-axis angular velocity sensor configured similarly to the first sensor 66, and the detection value of the virtual sensor 69 is calculated from the position of the virtual sensor 69 and the first detection value of the first sensor 66. it can.
Accordingly, the motion detection unit 185 detects the initial position and detection value of the virtual sensor 69, the position (4) of the first sensor 66 and the first detection value, and the position (5) of the first sensor 66 and the second time. Based on the detection value, a combination of the position of the three sensors and the detection value is obtained. Using the detection values of these three sensors, the direction and amount of movement at the movement center P1 can be obtained. The calculation result obtained in this example is the direction and amount of motion when the motion detected from the first time to the second time is regarded as one motion.

  Thus, in the head-mounted display devices 100 and 100A, not only a plurality of motion sensors but also one motion sensor can be used to obtain the direction and amount of motion at the head motion center. it can. Therefore, the present invention is applicable not only to the configuration using two or more motion sensors as in the first and second embodiments. That is, some of the plurality of motion sensors of the present invention may be virtual sensors, or may be sensors that are realized by shifting the time axis using other sensors.

The present invention is not limited to the configuration of each of the embodiments described above, and can be implemented in various modes without departing from the scope of the invention.
In the above embodiment, the image display units 20 and 200 including the frame 2 having a spectacle-shaped outer shape have been described. However, the frame 2 may be configured to be incorporated in a body protective device such as a hat or a helmet. . That is, a portion that supports the optical system including the right display drive unit 22 and the left display drive unit 24 and motion sensors such as the first camera 64, the second camera 65, the first sensor 66, and the second sensor 68 is provided. If it is spectacles type, an external shape will not be restrict | limited. Further, the camera unit 3 may be provided integrally with the frame 2. The camera unit 3 may be configured to be movable with a higher degree of freedom by being connected to the frame 2 by a member other than the hinges 21A and 23A.

  Furthermore, in the above-described embodiment, the head-mounted display devices 100 and 100A have been described as a configuration in which the image display units 20 and 200 and the control device 10 are separated and connected via the connection unit 40. The present invention is not limited to this, and the control device 10 may be configured integrally with the image display units 20 and 200. The head-mounted display devices 100 and 100A may be devices that can be mounted on the head of the user (operator or commander) at least. The mounting state of the control device 10 is not limited. For this reason, a notebook computer, a tablet computer, or a desktop computer may be used as the control device 10. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like. The control device 10 may be configured separately from the image display units 20 and 200, and various signals may be transmitted and received between the control device 10 and the image display units 20 and 200 by wireless communication.

For example, as a configuration in which the image display unit 20 generates image light, an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be provided. Further, as a configuration for generating image light, LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like can be used.
The optical elements that guide the image light to the user's eyes are not limited to the right light guide plate 261 and the left light guide plate 262. That is, any optical element that transmits external light incident on the apparatus from the outside and enters the user's eyes together with image light may be used. For example, an optical element that is positioned in front of the user's eyes and that overlaps part or all of the user's field of view may be used. Further, a scanning optical element that scans a laser beam or the like to obtain image light may be employed. Further, the optical element is not limited to guiding the image light inside the optical element, but may be an optical element having only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes. Furthermore, the optical element of the present invention may use a diffraction grating, a prism, and a holographic display unit.

For example, the present invention can also be applied to a laser retinal projection type head mounted display. That is, the light emitting unit includes a laser light source and an optical system that guides the laser light source to the user's eyes, and the laser light is incident on the user's eyes to scan the retina and form an image on the retina. A configuration that allows the user to visually recognize an image may be employed.
Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, a light emitting unit includes a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system. You may prepare. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, thereby recognizing the image. The optical component in this case may be a component that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

  4 and 6, at least a part of each functional block may be realized by hardware, or may be realized by cooperation of hardware and software. It is not limited to the structure which arrange | positions an independent hardware resource as follows. The program executed by the control unit 140 may be stored in the storage unit 120 or the storage device in the control device 10. Alternatively, a program stored in an external device may be acquired and executed via the communication unit 117 or the interface 125. Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, the control unit 140 illustrated in FIGS. 4 and 6 may be provided in the image display units 20 and 200. In this case, the control unit 140 and the control units of the image display units 20 and 200 share the functions. May be.

  2 ... Frame (main body), 2A ... right part, 2B ... left part, 2C ... bridge part (connection part), 2D ... temple part, 2E ... temple part, 3 ... camera unit, 3A ... arm part, 3B ... arm part DESCRIPTION OF SYMBOLS 3C ... Camera base part, 10 ... Control apparatus, 20 ... Image display part (display part), 21 ... Right holding part, 21A, 23A ... Hinge, 23 ... Left holding part, 40 ... Connection part, 61 ... Upper camera, 63 ... Microphone, 64 ... First camera (motion sensor, optical sensor), 65 ... Second camera (motion sensor, optical sensor), 66 ... First sensor (motion sensor, inertial sensor), 68 ... Second sensor (motion) Sensor, inertial sensor), 69 ... virtual sensor (motion sensor, inertial sensor), 100, 100A ... head-mounted display device (display device), 120 ... storage unit, 121 ... setting data, 122 Sensor position data, 123 ... content data, 135 ... operation unit, 140 ... control unit, 160 ... image processing unit, 170 ... display control unit, 181 ... imaging processing unit, 183 ... detection control unit, 185 ... detection unit, 186 ... AR display control unit, 187 ... sound processing unit, 261 ... right light guide plate (optical element), 261A ... half mirror (display area), 262 ... left light guide plate (optical element), 262A ... half mirror (display area).

Claims (12)

A head-mounted display device mounted on a user's head,
A display unit that emits image light to the user's eyes;
A plurality of motion sensors;
Any one of the motion sensors is located on one side from the center of the head in the mounted state, and the other motion sensor is located on the other side from the center of the head.
A display device. Any one of the motion sensors is located on the left side from the center of the head in the mounted state, and the other motion sensors are located on the right side of the center of the head.
The display device according to claim 1. The display unit includes an optical element having a display region that emits image light toward the user's eyes,
A plurality of the motion sensors and the center position of the display area of the optical element are arranged in a straight line;
The display device according to claim 1 or 2. The optical element has a half mirror as a display area that emits image light toward the user's eyes;
The display device according to claim 3. Having a main body worn across the user's left and right eyes,
At least two of the motion sensors are symmetric with respect to the center of the body;
The display device according to claim 1, wherein: The main body includes a right part located in front of the user's right eye, a left part located in front of the user's left eye, and the right part and the left part located in the center of the main body. And a connecting part for connecting
At least two of the motion sensors are in a symmetrical position with respect to the coupling part;
The display device according to claim 5. The plurality of motion sensors include at least an optical sensor and an inertial sensor;
The display device according to claim 1, wherein: The plurality of motion sensors are configured by any one of an optical sensor and an inertial sensor;
The display device according to claim 1, wherein: A motion detection unit that obtains a motion at the center of the motion of the user based on detection values of the plurality of motion sensors;
The display device according to claim 1, wherein: The motion detection unit obtains a motion at the center of the motion based on a detection value of each of the plurality of motion sensors and a position of the motion sensor;
The display device according to claim 9. By a display device that is mounted on the user's head and irradiates the user's eyes with image light, and a plurality of motion sensors,
Obtaining a motion at the motion center of the head based on detection values of the plurality of motion sensors;
A control method of a display device characterized by the above. Obtaining a motion at the center of the motion based on a detection value of each of the plurality of motion sensors and a position of the motion sensor;
The method for controlling a display device according to claim 11.

Images