JP2016033763A - Display device, method for controlling display device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device that can effectively combine an environment visible for a user with display contents, a method for controlling display device, and a program.SOLUTION: A head-mounted display device 100 is to be used by mounting on a user's body; and includes an image display unit 20 that transmits outside scenery and displays an image to be visible together with the scenery, and a camera 61 that captures scenery in a visual line of the user. A control unit 140 of the head-mounted display device 100 includes: an image generation unit 177 that generates an image for display from the image captured by the camera 61; and an image display control unit 176 that controls the image display unit 20 to display the image for display generated by the image generation unit 177.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a display device, a display device control method, and a program.

  2. Description of the Related Art Conventionally, wearable display devices are known that perform display while being superimposed on an external field of view (see, for example, Patent Document 1). The device described in Patent Document 1 has a configuration in which characters are superimposed on an external field of view, and the display attributes such as font size and character color are changed for some characters or words to be displayed. Makes it easy to identify some characters.

JP 2014-56217 A

Patent Document 1 describes a display method when a user who uses a wearable display device sees either a character displayed on the display device or an external field of view. On the other hand, there has been no proposal for a display method that considers that the user visually recognizes with the outside world.
The present invention has been made in view of the above-described circumstances, and provides a display device, a display device control method, and a program capable of effectively combining an external world visible to a user and display contents. With the goal.

In order to achieve the above object, a display device according to the present invention is a display device that is used by being worn on a user's body, and displays an image so that the image can be seen through the outside scene and visible together with the outside scene. An imaging unit that captures the direction of the user's line of sight, an image generation unit that generates a display image from a captured image of the imaging unit, and a display image generated by the image generation unit is displayed on the display unit And an image display control unit.
ADVANTAGE OF THE INVENTION According to this invention, an image for a display based on the picked-up image which imaged the user's gaze direction can be displayed so that it may overlap with the target object seen as an outside scene. For this reason, the external scene outside the display device and the display contents can be effectively improved by supplementing the appearance of the external scene viewed by the user based on a captured image obtained by capturing the external scene, or presenting another view. In combination, it is possible to provide a new method for utilizing display by a display device.

In the display device according to the present invention, the image generation unit generates the display image by extracting and processing a part of the captured image of the imaging unit.
According to the present invention, since a captured image obtained by capturing the user's line-of-sight direction is used for display, the user can be shown an image obtained by processing a part of the outside scene together with the outside scene.

In the display device according to the present invention, the image generation unit generates the display image by cutting out and enlarging a part of a captured image of the imaging unit.
According to the present invention, since a part of the captured image obtained by capturing the user's line-of-sight direction is enlarged and displayed, the user can view the line-of-sight direction while viewing the outside scene.

Further, the present invention is characterized in that, in the display device, the image generation unit generates the display image by cutting out a region corresponding to a preset condition in a captured image of the imaging unit. To do.
According to the present invention, a captured image corresponding to a preset condition can be displayed. For this reason, for example, when a captured image corresponding to the user's preference or interest is captured, the captured image can be displayed on the display unit and shown to the user.

In the display device, the present invention further includes an object detection unit that detects an object in the user's line-of-sight direction from a captured image of the imaging unit, and the image generation unit A region including an image of the object detected by the object detection unit is extracted and processed to generate the display image.
According to the present invention, when there is an object in the user's line-of-sight direction, an image of the object can be shown to the user.

The display device may further include a storage unit that stores the feature amount of the image of the object, and the target detection unit may convert the captured image of the imaging unit into a feature amount stored by the storage unit. The object is detected by detecting a suitable image.
According to the present invention, an object can be easily detected based on a captured image.

Further, according to the present invention, in the display device, the feature quantity stored in the storage unit is a feature quantity of an image of a specific object and a general-purpose feature quantity corresponding to a plurality of similar objects. Any one of these is included.
According to the present invention, a process for detecting a specific object or a process for detecting any one of a plurality of similar objects can be executed based on the stored feature amount.

The display device may further include a distance detection unit that detects a distance between the object detected by the target detection unit and the user, and the image display control unit includes the distance detection unit. The display image is generated by enlarging a part of the captured image of the imaging unit with an enlargement ratio corresponding to the detected distance.
ADVANTAGE OF THE INVENTION According to this invention, the image of a target object can be shown to a user with the expansion ratio matched with the distance with the target object of a user's gaze direction.

According to the present invention, in the display device, the image generation unit acquires a plurality of captured images captured when different by the imaging unit, and an area corresponding to a preset condition in the plurality of captured images. The display image is generated by specifying.
According to the present invention, more complicated conditions can be set for a plurality of captured images, and images can be shown to the user based on the captured images corresponding to the conditions.

Further, according to the present invention, in the display device, the image generation unit acquires a plurality of captured images captured at different times by the imaging unit, and a difference between the images of the plurality of captured images is preset. When the condition is met, the acquired captured image is cut out to generate the display image.
According to the present invention, since a display image can be generated and displayed based on a difference between a plurality of captured images, for example, a display image is generated based on a change in the image for a moving image including a plurality of continuous images. Can be displayed.

In order to achieve the above object, the present invention includes a display unit that transmits an outside scene and displays an image so as to be visible together with the outside scene, and controls a display device that is used by being worn on a user's body. A method of capturing a user's line-of-sight direction, cutting out a part of the captured image and enlarging the generated image for display, and displaying the generated display image on the display unit. And
ADVANTAGE OF THE INVENTION According to this invention, an image for a display based on the picked-up image which imaged the user's gaze direction can be displayed so that it may overlap with the target object seen as an outside scene. For this reason, the external scene outside the display device and the display contents can be effectively improved by supplementing the appearance of the external scene viewed by the user based on a captured image obtained by capturing the external scene, or presenting another view. In combination, it is possible to provide a new method for utilizing display by a display device.

In order to achieve the above object, the present invention includes a display unit that transmits an outside scene and displays an image so that the outside scene can be viewed with the outside scene, and controls a display device that is used by being worn on the user's body. An image generation unit that generates a display image by cutting out and enlarging a part of a captured image obtained by capturing the user's line-of-sight direction. It is a program for functioning as an image display control unit that displays an image generated by a generation unit on the display unit.
ADVANTAGE OF THE INVENTION According to this invention, an image for a display based on the picked-up image which imaged the user's gaze direction can be displayed so that it may overlap with the target object seen as an outside scene. For this reason, the external scene outside the display device and the display contents can be effectively improved by supplementing the appearance of the external scene viewed by the user based on a captured image obtained by capturing the external scene, or presenting another view. In combination, it is possible to provide a new method for utilizing display by a display device.

It is explanatory drawing which shows the external appearance structure of a head mounted type display apparatus. It is a block diagram which shows the functional structure of a head mounted display apparatus. It is explanatory drawing which shows the example of the data memorize | stored in a memory | storage part. It is a flowchart which shows operation | movement of a head mounting type display apparatus. It is explanatory drawing which shows the typical example of application of a head-mounted display apparatus, (A) shows the example of a user's visual field, (B) and (C) show the example of the image cut out from a captured image , (D) shows an example of the visual field of the user when the display image is displayed.

  FIG. 1 is an explanatory diagram showing an external configuration of the head-mounted display device 100. The head-mounted display device 100 is a display device that is mounted on the head, and is also referred to as a head mounted display (HMD). The head-mounted display device 100 of the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. In this specification, a virtual image visually recognized by the user with the head-mounted display device 100 is also referred to as a “display image” for convenience. Moreover, emitting image light generated based on image data is also referred to as “displaying an image”.

  The head-mounted display device 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when mounted on the user's head, and a control device 10 that controls the image display unit 20. The control device 10 also functions as a controller for the user to operate the head-mounted display device 100. The image display unit 20 is also simply referred to as a “display unit”.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. (Imaging unit) and a microphone 63. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

The display drive units 22 and 24 include liquid crystal displays 241 and 242 (Liquid Crystal Display, hereinafter also referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 2). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2) and a light control plate 20A. The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. The dimming plate 20A is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. As the light control plate 20A, various ones such as one having almost no light transmission, one having near transparency, one having a light amount attenuated to transmit light, one having a specific wavelength attenuated or reflected, and the like are used. Can do. By appropriately selecting the optical characteristics (light transmittance, etc.) of the light control plate 20A, the external light amount incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to visually recognize the virtual image. You can adjust the ease. In the present embodiment, a case will be described in which at least a light control plate 20A having such a light transmittance that a user wearing the head-mounted display device 100 can visually recognize an outside scene is used. The light control plate 20A protects the right light guide plate 261 and the left light guide plate 262, and suppresses damage to the right light guide plate 261 and the left light guide plate 262, adhesion of dirt, and the like.
The light control plate 20A may be detachable with respect to the right optical image display unit 26 and the left optical image display unit 28, and a plurality of types of light control plates 20A may be exchanged and may be omitted.

The camera 61 is disposed at the end ER that is the other end of the right optical image display unit 26. The camera 61 captures an outside scene that is an external scenery in a direction opposite to the user's eye side, and acquires an outside scene image. The camera 61 of this embodiment shown in FIG. 1 is a monocular camera, but may be a stereo camera.
The shooting direction of the camera 61, that is, the angle of view, is taken from the front side direction of the head-mounted display device 100, in other words, at least a part of the outside view in the user's viewing direction when the head-mounted display device 100 is worn Direction. In addition, although the angle of view of the camera 61 can be set as appropriate, the imaging range of the camera 61 includes the outside world (outside scene) that the user visually recognizes through the right optical image display unit 26 and the left optical image display unit 28. A range is preferable. Furthermore, it is more preferable that the imaging range of the camera 61 is set so that the entire field of view of the user through the light control plate 20A can be captured.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control device 10. The connection unit 40 includes a main body cord 48 connected to the control device 10, a right cord 42, a left cord 44, and a connecting member 46. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  The specific specifications of the microphone 63 are arbitrary, and may be a directional microphone or an omnidirectional microphone. Examples of directional microphones include unidirectionality (Cardioid), narrow directivity (Super cardioid), sharp directivity (Hypercardioid), and superdirectivity (Ultra Cardioid). In the case where the microphone 63 has directivity, the sound from the line of sight of the user wearing the head-mounted display device 100 may be particularly well collected and detected. In this case, in order to ensure the directivity of the microphone 63, the microphone 63 or a part that accommodates the microphone 63 may have a structural feature. For example, in the example of FIG. 1, the microphone 63 and the connecting member 46 are designed so that the sound collection unit of the microphone 63 faces the user's line-of-sight direction with the user wearing the right earphone 32 and the left earphone 34. It only has to be. Alternatively, the microphone 63 may be embedded in the right holding unit 21 or the left holding unit 23. In this case, if a sound collecting hole is formed in the front side of the right holding unit 21 or the left holding unit 23, that is, the surface aligned with the right optical image display unit 26 and the left optical image display unit 28, the direction of the user's line of sight The directivity corresponding to can be given. The line-of-sight direction of the user is, for example, the direction that the right optical image display unit 26 and the left optical image display unit 28 face, and the center of the field of view that the user sees through the right optical image display unit 26 and the left optical image display unit 28. In other words, the shooting direction of the camera 61, and the like. Note that the direction of directivity of the microphone 63 may be variable. In this case, it may be configured to detect the user's line-of-sight direction and adjust the line-of-sight direction of the microphone 63 so as to face the direction.

  It is also possible to combine the right code 42 and the left code 44 into one code. Specifically, the lead wire inside the right cord 42 is drawn into the left holding portion 23 side through the inside of the main body of the image display unit 20, and is covered with a resin together with the lead wire inside the left cord 44 to be combined into one cord. Also good.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control device 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control device 10, the control device 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control device 10 is a device for controlling the head-mounted display device 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 that is touched by a user with a finger.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 notifies the operation state of the head-mounted display device 100 by its light emission state. Examples of the operating state of the head-mounted display device 100 include power ON / OFF. As the lighting unit 12, for example, an LED (Light Emitting Diode) is used. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content moving image between 3D and 2D, for example.

  The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the head-mounted display device 100 by detecting a slide operation of the switch.

FIG. 2 is a functional block diagram of each unit constituting the display system 1 according to the embodiment.
As shown in FIG. 2, the display system 1 includes an external device OA and a head-mounted display device 100. Examples of the external device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. The external device OA is used as an image supply device that supplies an image to the head-mounted display device 100.

The control device 10 of the head-mounted display device 100 includes a control unit 140, an operation unit 135, an input information acquisition unit 110, a storage unit 120, a power supply 130, an interface 180, a transmission unit (Tx) 51, and And a transmission unit (Tx) 52.
The operation unit 135 detects an operation by the user. The operation unit 135 includes the determination key 11, the display switching key 13, the track pad 14, the luminance switching key 15, the direction key 16, the menu key 17, and the power switch 18 shown in FIG.

The input information acquisition unit 110 acquires a signal corresponding to an operation input by the user. As a signal corresponding to the operation input, for example, there is an operation input to the track pad 14, the direction key 16, and the power switch 18.
The power supply 130 supplies power to each part of the head-mounted display device 100. As the power supply 130, for example, a secondary battery can be used.

The storage unit 120 stores various computer programs. The storage unit 120 is configured by a ROM, a RAM, or the like. The storage unit 120 may store image data to be displayed on the image display unit 20 of the head-mounted display device 100.
The storage unit 120 stores detection feature data 124 referred to by a target detection unit 171 described later, and replacement image data 125 processed by the image display control unit 176.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control device 10. As the interface 180, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface can be used.

The control unit 140 reads out and executes the computer program stored in the storage unit 120, thereby realizing the function of each unit. That is, the control unit 140 includes an operating system (OS) 150, an image processing unit 160, an audio processing unit 170, an object detection unit 171, a distance detection unit 173, an image display control unit 176, an image generation unit 177, and a display control unit. It functions as 190.
A three-axis sensor 113, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 can acquire the detection value of the triaxial sensor 113. The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. Further, the GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140 of the control device 10.

The communication unit 117 executes wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), or Bluetooth (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and performs control for displaying an image on the image display unit 20. On the other hand, when the external device OA is wired to the interface 180, the control unit 140 acquires content data from the interface 180 and performs control for displaying an image on the image display unit 20. Therefore, the communication unit 117 and the interface 180 are hereinafter collectively referred to as a data acquisition unit DA.
The data acquisition unit DA acquires content data to be displayed by the head-mounted display device 100 from the external device OA. The content data includes image data to be described later.

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal in the DRAM in the storage unit 120 frame by frame as image data (Data in the figure) of the target image. This image data is, for example, RGB data.
Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control device 10 and the image display unit 20.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. And a vibration sensor 65 and a nine-axis sensor 66.

  The vibration sensor 65 is configured using an acceleration sensor, and is arranged inside the image display unit 20 as shown in FIG. In the example of FIG. 1, the right holding unit 21 is built in the vicinity of the end ER of the right optical image display unit 26. When the user performs an operation of hitting the end ER (knock operation), the vibration sensor 65 detects vibration caused by this operation and outputs the detection result to the control unit 140. Based on the detection result of the vibration sensor 65, the control unit 140 detects a knocking operation by the user.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of the user's head is detected. Since the orientation of the image display unit 20 is known from the detected movement of the user's head, the control unit 140 can estimate the user's line-of-sight direction.

  The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are provided. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1 input through the reception unit 53. . The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, a left LCD control unit 212 and a left that function as a display element. An LCD 242 and a left projection optical system 252 are provided. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

The head-mounted display device 100 displays an image based on image data so as to overlap the outside scene when the user views the outside scene through the right optical image display unit 26 and the left optical image display unit 28. To do.
The target detection unit 171 performs control for causing the camera 61 to perform imaging, and acquires a captured image. The captured image is output from the camera 61 as color image data or monochrome image data. The camera 61 outputs an image signal, and the target detection unit 171 generates image data that conforms to a predetermined file format from the image signal. May be.
The object detection unit 171 analyzes the acquired captured image data and detects an object reflected in the captured image data. The target object is an object or a person that exists in the imaging direction of the camera 61, that is, the user's line-of-sight direction.

The target detection unit 171 searches the captured image for an image that matches the detection feature data 124 and detects the matching image as an image of the target object.
FIG. 3 is a diagram illustrating a configuration example of the detection feature data 124 and the replacement image data 125 stored in the storage unit 120.
The detected feature data 124 is data of the feature amount of the image detected from the captured image. In the present embodiment, the detected feature data 124 includes single image feature data 124a and moving image feature data 124b. The single image feature data 124a is feature amount data applied to one captured image. When the single image feature data 124a is used, the target detection unit 171 performs matching processing for detecting a portion that matches the single image feature data 124a for one captured image.

  The moving image feature data 124b is feature amount data applied to a plurality of captured images. When the moving image feature data 124b is used, the target detection unit 171 takes a captured image of the camera 61 a plurality of times, and obtains a difference between the captured plurality of captured images. The camera 61 can capture images at predetermined time intervals, and can output captured image data every time it captures images. That is, the camera 61 functions as a moving image camera that captures images at a frame rate corresponding to the time interval. In this case, the target detection unit 171 can obtain a change in the moving image by obtaining a difference between a plurality of captured images captured by the camera 61 at different timings. The moving image feature data 124b can be referred to as feature amount data applied to changes in moving images. The target detection unit 171 performs matching processing for detecting a portion that matches the moving image feature data 124b in the difference between the captured images.

  In the present embodiment, whether the detection feature data 124 used by the target detection unit 171 is the single image feature data 124a or the moving image feature data 124b is set in advance. When the target detection unit 171 is set to use the single image feature data 124a, the target detection unit 171 acquires one captured image of the camera 61 and performs matching processing. On the other hand, when the moving image feature data 124b is set to be used, the target detection unit 171 temporarily stores the captured image of the camera 61 in the storage unit 120 every time it acquires the captured image. Then, the target detection unit 171 obtains a plurality of captured images stored in the storage unit 120, obtains a difference, and performs a matching process based on the moving image feature data 124b for the difference.

  The object detection unit 171 identifies an object that appears in the captured image captured by the camera 61 through matching processing. That is, when the target detection unit 171 detects a region that matches the single image feature data 124a in the captured image or a region that matches the moving image feature data 124b in the difference between the captured images, the target detection unit 171 reflects the detected region. The subject is the target.

The distance detection unit 173 obtains the distance to the object detected by the object detection unit 171. For example, the distance detection unit 173 obtains the distance to the object based on the size of the image of the object detected by the object detection unit 171 in the captured image of the camera 61.
The head-mounted display device 100 may include a distance meter that detects the distance to the object using laser light or ultrasonic waves. For example, the distance meter includes a light source of laser light and a light receiving unit that receives reflected light of the laser light emitted from the light source, and detects a distance to an object based on a light receiving state of the laser light. The distance meter may be an ultrasonic distance meter, for example. That is, a distance meter that includes a sound source that emits ultrasonic waves and a detection unit that detects ultrasonic waves reflected on the object, and detects the distance to the object based on the reflected ultrasonic waves may be used. Further, the distance meter can be configured by combining a distance meter using laser light and a distance meter using ultrasonic waves. Such a distance meter is preferably provided in the right holding unit 21 or the right display driving unit 22 of the image display unit 20, and may be installed, for example, facing the front side on the surface aligned with the light control plate 20A. The direction in which the distance meter measures the distance is preferably the user's line-of-sight direction, similar to the imaging direction of the camera 61.
The distance detection unit 173 detects the distance from the camera 61 or the distance meter to the object, and this distance can be regarded as the distance from the user of the head-mounted display device 100 to the object.
When the target detected by the target detection unit 171 is not a specific object, the distance detection unit 173 detects the distance to this target, for example, using the center of the captured image as a virtual target.

The image generation unit 177 generates a display image by extracting and processing a part of the captured image of the camera 61. In the present embodiment, the image generation unit 177 generates a display image by cutting out an area of a predetermined size including the feature detected by the target detection unit 171 from the captured image. When the object detection unit 171 detects an object from a plurality of captured images using the moving image feature data 124b, the image generation unit 177 includes the object detected by the object detection unit 171 in the latest captured image. Cut out the area.
The display image generated by the image generation unit 177 is displayed in the displayable area of the image display unit 20 under the control of the image display control unit 176. The display size of the display image is set in advance, and is, for example, a level that does not hinder the user when viewing the outside scene. Therefore, the image generation unit 177 cuts out the captured image with a size that matches the display size of the display image.

The image generation unit 177 generates a display image so that a part of the captured image including the feature detected by the target detection unit 171 is displayed more easily. Therefore, as described above, the process executed by the image generation unit 177 is not limited to the process of enlarging the area extracted from the captured image. For example, a display image may be obtained by surrounding the object in a region extracted from the captured image with a frame such as a circle or a rectangle. In addition, for example, a decoration image such as a balloon or an arrow may be given to the object to generate a display image that can easily recognize the object. The decoration in this case is an image stored in advance for decoration, and can be called an auxiliary image or an enhancement image. Further, the luminance and color tone of the region including the object may be changed. For example, the object and its surroundings may be a high-luminance image.
Further, when a decoration is given to an object detected based on the single image feature data 124a or the moving image feature data 124b, the data related to the decoration is used as the single image feature data 124a and the moving image feature data 124b. May be included.

  Further, the image generation unit 177 may cut out the target object and its surrounding images when cutting out an image including the target object detected by the target detection unit 171. In this case, it can be clearly seen that the image is an enlarged image of a part of the captured image, so that the user can enlarge and view a portion that is difficult to see, for example. In addition, the image generation unit 177 may cut out the image along the contour of the image of the object. In this case, the image to be cut out is combined with another background image and displayed, or the cut out image is displayed in 3D ( It is possible to display the image on the image display unit 20 as a (stereo) image.

  Furthermore, the image generation unit 177 determines the enlargement ratio based on the distance detected by the distance detection unit 173. The image generation unit 177 makes the user see an enlarged image when a part of the image cut out from the captured image is displayed under the control of the image display control unit 176. Therefore, the image generation unit 177 determines the size of the image to be cut out from the captured image based on the size (resolution or the number of image dots) for displaying the display image in the displayable area of the image display unit 20 and the enlargement ratio. To do. Then, the image generation unit 177 executes enlargement processing (or resolution conversion processing) for converting the size of the cut-out image according to the display size. Thereby, since the image cut out from the captured image is in an enlarged state, the user can see the enlarged image.

  The size at which the display image enlarged by the image generation unit 177 is visually recognized by the user is affected by the specifications of the image display unit 20. In order to appropriately manage and adjust the enlargement ratio at which the image generation unit 177 enlarges the image cut out from the captured image, for example, when the image cut out from the captured image has the same size as the outside scene in the user's field of view (that is, It is preferable to set the size of an image with an enlargement ratio of 1) in advance. That is, the size of the image cut out by the image generation unit 177 when the user visually recognizes the display image with a size that is neither enlarged nor reduced is set in advance. With this size as a reference, the image generation unit 177 cuts out and enlarges the image in accordance with the target enlargement ratio.

The enlargement ratio in the process of enlarging the image by the image generation unit 177 may be determined according to the distance detected by the distance detection unit 173, for example.
FIG. 3 shows an example in which the display setting data 126 stored in the storage unit 120 includes enlargement ratio setting data 126 a that associates the distance to the object detected by the distance detection unit 173 and the enlargement ratio. The enlargement ratio setting data 126a is data for determining the distance and the enlargement ratio detected by the distance detection unit 173. For example, a table for obtaining the enlargement ratio step by step, or an arithmetic expression or parameter for calculating the enlargement ratio from the distance. Including. Based on the enlargement factor setting data 126a, the image generation unit 177 can obtain the enlargement factor from the distance detected by the distance detector 173. Further, the case where the image generation unit 177 does not enlarge may be set in the enlargement ratio setting data 126a, or the case where the image generation unit 177 does not generate a display image may be set. For example, when the distance to the object detected by the distance detection unit 173 is closer than the set distance, it can be set not to generate and display the display image. When the image generation unit 177 does not generate and output display image data, the image display control unit 176 does not display the display image.
The display setting data 126 includes, in addition to the enlargement ratio setting data 126a, data set in advance with respect to image display, such as setting value data relating to the default display size in the image display unit 20, and setting value data relating to display color adjustment. May be included.

  The image display control unit 176 controls the display control unit 190 to cause the image display unit 20 to display an image based on the display image data generated by the image generation unit 177.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker in the left earphone 34 (not shown) connected to the connecting member 46. (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing is performed on the audio signal, and different sounds with different frequencies or the like are output from each of the right earphone 32 and the left earphone 34, for example.

  In addition, the sound processing unit 170 acquires sound collected by the microphone 63, converts it into digital sound data, and performs processing related to sound. For example, the speech processing unit 170 performs speaker recognition that identifies a person who speaks for each voice by recognizing voices of a plurality of persons separately by extracting and modeling features from the acquired speech. May be.

A three-axis sensor 113, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 can detect the detection value of the triaxial sensor 113 and detect the movement and the direction of movement of the control device 10.
The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. Further, the GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140 of the control device 10.
The communication unit 117 performs wireless data communication conforming to a wireless LAN (WiFi (registered trademark)) or Bluetooth (registered trademark) standard.

The interface 180 is an interface for connecting various image supply apparatuses OA that are content supply sources to the control apparatus 10. The content supplied by the image supply device OA includes a still image or a moving image, and may include sound. Examples of the image supply device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.
Here, it is also possible to connect the image supply device OA to the control device 10 by a wireless communication line. In this case, the image supply device OA performs wireless communication with the communication unit 117 and transmits content data using a wireless communication technique such as Miracast (registered trademark).

FIG. 4 is a flowchart showing the operation of the head-mounted display device 100. In this operation, when the user views the outside scene through the right optical image display unit 26 and the left optical image display unit 28, a part of the captured image obtained by capturing the outside scene is displayed by the image display unit 20.
FIG. 5 is an explanatory diagram showing how the head-mounted display device 100 displays an image by the operation shown in FIG. 5A shows an example of the user's visual field VR, FIGS. 5B and 5C show examples of images cut out from the captured image, and FIG. 5D shows a case where a display image is displayed. An example of the visual field VR of the user is shown.

  In the control unit 140 of the head-mounted display device 100, the target detection unit 171 causes the camera 61 to perform imaging (step S11) and obtains a captured image (step S12). The target detection unit 171 acquires the single image feature data 124a or the moving image feature data 124b from the storage unit 120 (step S13), and the acquired detection feature data target is a single captured image or a plurality of captured images. Is determined (step S14). This determination can be made based on whether the single image feature data 124a or the moving image feature data 124b is designated as the detection feature data to be used.

When processing a single captured image (step S14; YES), the target detection unit 171 performs matching processing on the captured image acquired in step S12 using the single image feature data 124a (step S15). ), The process proceeds to step S18.
When processing a plurality of captured images (step S14; NO), the target detection unit 171 includes the captured image acquired in step S12 and the captured image acquired in the past and temporarily stored in the storage unit 120. The difference is obtained (step S16). The target detection unit 171 executes matching processing based on the moving image feature data 124b for the obtained difference (step S17), and proceeds to step S18.

  In step S18, the target detection unit 171 determines an area to be cut out from the captured image. The region determined here is a region including the object detected by the object detection unit 171, but the detailed cutout range is determined by the image generation unit 177 in step S <b> 21 described later. Therefore, the region determined in step S18 is a temporary region, and may be, for example, the center position of the region to be cut out or the contour of the object.

  The example shown in FIG. 5A shows an example where the user is watching a soccer game from the audience seat. In this example, a soccer field F and a plurality of players FP playing in the field F are visible. It can be seen that one player FP has the ball B. In this example, since the user is far away from the field F, the player P and the ball B in the field of view VR appear small.

The imaging range (view angle) of the camera 61 is, for example, the same range as the visual field VR, and the field F, the player FP, and the ball B are captured in the captured image.
The object detection unit 171 performs matching processing on the captured image based on the single image feature data 124a, and detects the image of the ball B as the object. In this case, the target detection unit 171 detects the ball B in step S15, and in step S18, an area including the ball B is determined as a cut-out area.

Subsequently, the distance detection unit 173 detects the distance to the target object included in the region determined by the target detection unit 171 in step S18 (step S19).
Then, the image generation unit 177 determines the enlargement rate based on the distance detected by the distance detection unit 173 and the enlargement rate setting data 126a stored in the storage unit 120 (step S20). Based on the determined enlargement factor, the image generation unit 177 determines the size and position of the image to be cut out from the captured image, cuts out the image, and outputs the cut out image as a display image (step S21). In step S21, the image generation unit 177 may perform the above-described size conversion or resolution conversion processing. The image generation unit 177 does not output a display image when the setting of the enlargement ratio setting data 126a corresponding to the distance detected by the distance detection unit 173 is a setting for not displaying.

  5B and 5C show examples of display images output from the image generation unit 177. FIG. The image generation unit 177 cuts out an area including the image of the ball B from the captured image in accordance with contents set in advance with respect to the cutout of the image. The contents set in advance for image cropping are the aspect ratio of the clipped image and the size of the image when the enlargement ratio is 1. In the example of FIG. 5B, an area in which the whole body of the ball B and the player FP having the ball B is shown is cut out and output as a display image. In the example of FIG. 5C, the ball B and a region of a predetermined size including the ball B are cut out and output as a display image. In the example of FIG. 5C, only part of the body of the player FP having the ball B is included. When the single image feature data 124a is feature amount data for detecting the ball B, whether the whole body of the player FP or a part of the body is included in the image cut out by the image generation unit 177 is: It depends on the setting such as magnification.

  The image display control unit 176 stores the display image output from the image generation unit 177 in the storage unit 120 (step S22). The storage unit 120 stores display images in time series. Therefore, when the process shown in steps S11 to S24 in FIG. 4 is repeatedly executed, a plurality of display images are stored in the storage unit 120 in chronological order. The plurality of display images can be reproduced as moving images by displaying them continuously. For example, in accordance with an instruction operation on the operation unit 135, the control unit 140 can reproduce and display the display image stored in the storage unit 120.

The image display control unit 176 outputs the display image output from the image generation unit 177 in step S21 to the display control unit 190 and displays it on the image display unit 20 (step S23).
Thereafter, the control unit 140 determines whether or not to end the display (step S24). When the display is continued (step S24; NO), the control unit 140 returns to step S1. When the display is terminated according to the operation detected by the operation unit 135 (step S24; YES), the control unit 140 stops the display by the display control unit 190 and ends the present process.

  FIG. 5D shows a state in which a display image is displayed by the image display unit 20. In the field of view VR, as in FIG. 5A, the field F is seen as an outside scene, but the image IV is displayed so as to be superimposed on a part of the field of view VR. This image IV is a display image displayed by the image display unit 20. Since the image IV is an image obtained by enlarging a part of the outside scene, the user can view the outside scene viewed through the image display unit 20 and a part of the outside scene at the same time.

  In addition, the object detection part 171 may perform the process which tracks the target once detected, when performing the process of FIG.4 S11-S17 repeatedly. That is, the target detection unit 171 detects a target object based on the single image feature data 124a or the moving image feature data 124b, and then monitors the change in the captured image of the camera 61 to detect the movement of the detected target object. Chase. Thereby, the target detection unit 171 can detect the target object from the captured image of the detection target without performing the matching processing in steps S15 and S17. Such a method is, for example, processing load when a moving image is captured by the camera 61 at a predetermined frame rate (for example, 30 frames / second), and the object detection unit 171 detects an object for each frame. Can be mitigated and effective.

  Further, the single image feature data 124a and the moving image feature data 124b are not limited to feature amounts for detecting a specific object. For example, the single image feature data 124a can be a feature amount for detecting a captured image including a specific feature, and the moving image feature data 124b is a feature amount for detecting a moving image having a specific change. can do. In this case, when a feature that matches the single image feature data 124a or the moving image feature data 124b is detected in the captured image, the target detection unit 171 sets a portion having the feature as a target. In this case, the target is not a specific object but a part of the captured image. The image generation unit 177 may cut out a part of the captured image detected as the target by the target detection unit 171 and use it as a display image.

The single image feature data 124a and the moving image feature data 124b may include feature amount data for a specific object for detecting one specific object or a small number of objects, or a plurality of similar types of data. General-purpose feature amount data that can detect the target object may be included.
For example, in the example of FIGS. 5B and 5C, the target detection unit 171 detects the ball based on the single image feature data 124a. The single image feature data 124a may be feature amount data for a specific object for detecting a ball of a specific design (including a color and a shape of the design), or a general-purpose data for detecting a sphere. It may be feature amount data. In the case of general-purpose feature amount data, for example, a feature amount that identifies only a shape, a feature amount that identifies a shape and a color, or a feature amount that identifies one of a plurality of colors It may be. In addition to the feature amount data, the single image feature data 124a may include data defining a range of difference from the allowable feature amount. The same applies to the moving image feature data 124b.
Specifically, if the data of general-purpose features that allow for differences in a specific person's face, hairstyle, clothes (uniforms, shoes, hats, bibs, etc.), not only can multiple objects be detected, It is possible to cope with changes in color in captured images due to sunlight, lighting conditions, environmental brightness, and sunlight. In addition, the target detection unit 171 may track the target in the captured image data that is continuously captured after the target is detected. In this case, when the object image in the captured image data no longer corresponds to the feature amount while tracking the target object, the feature amount may be calibrated (calibrated). As a result, when using feature amount data for a specific object and when using general-purpose feature amount data, the color and shape of the object in the captured image can be determined from the feature amount due to factors such as sunlight. Even if the vehicle deviates, the object can be detected and the object can be detected continuously.

  As described above, the head-mounted display device 100 according to the embodiment to which the present invention is applied is a display device that is used while being worn on a user's body, and can be seen through the outside scene and together with the outside scene. Thus, the image display unit 20 that displays an image and the camera 61 that captures the user's line-of-sight direction are provided. The control unit 140 of the head-mounted display device 100 generates an image for display from an image captured by the camera 61 and an image for causing the image display unit 20 to display the display image generated by the image generation unit 177. A display control unit 176. As a result, the display image based on the captured image obtained by capturing the user's line-of-sight direction can be displayed so as to overlap the object that appears as an outside scene. For this reason, the external scene outside the display device and the display contents can be effectively improved by supplementing the appearance of the external scene viewed by the user based on a captured image obtained by capturing the external scene, or presenting another view. In combination, it is possible to provide a new method for utilizing the display by the display device.

Moreover, since the image generation unit 177 generates a display image by cutting out and enlarging a part of the captured image of the camera 61, the captured image obtained by capturing the user's line-of-sight direction can be enlarged and displayed. For this reason, the user can enlarge the line of sight while viewing the outside scene.
In addition, the image generation unit 177 generates a display image by cutting out a region corresponding to a preset condition in the captured image of the camera 61, and thus can display a captured image corresponding to the preset condition. For example, when a captured image corresponding to the user's preference or interest is captured, the captured image can be displayed by the image display unit 20 and shown to the user.

The head-mounted display device 100 includes a target detection unit 171 that detects an object in the direction of the user's line of sight from an image captured by the camera 61. The image generation unit 177 generates a display image by cutting out a region including the image of the target detected by the target detection unit 171 from the captured image of the camera 61. Thereby, when there is an object in the direction of the user's line of sight, the image of the object can be shown to the user.
The head-mounted display device 100 includes a distance detection unit 173 that detects the distance between the object detected by the object detection unit 171 and the user. The image generation unit 177 enlarges a part of the captured image of the camera 61 at an enlargement rate corresponding to the distance detected by the distance detection unit 173, and generates a display image. Thereby, the image of the object can be shown to the user at an enlargement rate that matches the distance from the object in the direction of the user's line of sight.
In addition, the image generation unit 177 acquires a plurality of captured images captured at different times by the camera 61, and specifies a region corresponding to a preset condition in the plurality of captured images, thereby generating a display image. Generate. Thereby, more complicated conditions can be set for a plurality of captured images, and images can be shown to the user based on the captured images corresponding to the conditions.

  In addition, the image generation unit 177 acquires a plurality of captured images captured at different times by the camera 61, and when the difference between the images of the plurality of captured images satisfies a preset condition, the acquired imaging A display image is generated by cutting out the image. Accordingly, since a display image can be generated and displayed based on a difference between a plurality of captured images, for example, a display image can be generated and displayed based on a change in image for a moving image including a plurality of continuous images. .

In addition, the image display control unit 176 stores the display image generated by the image generation unit 177 in the storage unit 120 in time series, and when the operation unit 135 detects an operation instructing reproduction, the image display control unit 176 stores the display image in the storage unit 120. The stored display image is reproduced and displayed. Here, the image display control unit 176 can display a moving image by sequentially reproducing a plurality of display images stored in time series at a predetermined frame rate. In this case, the user can see a moving image in which an image with an enlarged line-of-sight direction is recorded. Here, when the display image stored in the storage unit 120 is reproduced and displayed, the image display control unit 176 may change the magnification of the display image. For example, when an operation for designating an enlargement ratio is detected by the operation unit 135, the enlargement ratio may be changed according to this operation. Further, in FIG. 5D, the display image is displayed so as to overlap a part of the visual field VR. However, when the display image stored in the storage unit 120 is reproduced and displayed, the entire displayable area of the image display unit 20 is displayed. A display image may be displayed on the screen.
Further, the image generation unit 177 may adjust the transparency of the image IV. In the example of FIG. 5D, the transparency is zero, and the outside scene is not visible in the portion overlapping the image IV, but the image generation unit 177 sets the transparency based on the brightness of the outside scene overlapping the image IV, the brightness of the image IV, and the like. You may adjust.

  In the above embodiment, the case has been described in which the object detection unit 171 recognizes an object using the single image feature data 124a and the moving image feature data 124b stored in the storage unit 120. The single image feature data 124a is not only stored in the storage unit 120 in advance, but may be configured to be generated or changed by a user operation while the head-mounted display device 100 is in use. For example, a configuration in which the head-mounted display device 100 can perform an operation of selecting a part of the display image by operating the track pad 14 while displaying the captured image of the camera 61 on the image display unit 20 is conceivable. . With this configuration, when the user selects a part of the display image, the target detection unit 171 detects an image of a person or an object from the selected part, and the single image feature data 124a or the like is detected from the detected image. The moving image feature data 124b may be generated. Alternatively, the head-mounted display device 100 may be configured to be operable by a voice command. In this case, the control unit 140 detects the user's voice with the microphone 63, analyzes the detected voice with the voice processing unit 170, and recognizes the voice command. When the head-mounted display device 100 displays a captured image of the camera 61 on the image display unit 20, when the user instructs to select a part of the display image with a voice command, the target detection unit 171 Alternatively, an image of a person or an object may be detected from the selected portion, and single image feature data 124a or moving image feature data 124b may be generated from the detected image. Specifically, when a soccer game is shot and displayed as shown in FIG. 5A, the object is detected when the user instructs “Track No. 11” with a voice command. The unit 171 detects the number “11” from the captured image of the camera 61, extracts a portion including the detected number, obtains a feature amount, and generates single image feature data 124 a and moving image feature data 124 b. May be.

In addition, this invention is not restricted to the structure of the said embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.
In the above-described embodiment, as an example in which the image generation unit 177 cuts out a region corresponding to a preset condition from the captured image of the camera 61, the target detection unit 171 cuts out a region where the target is detected by performing a matching process. An example was explained. The present invention is not limited to this. For example, the image generation unit 177 may detect and cut out a region corresponding to a preset condition in the captured image to generate a display image. For example, when the position and size of the region are set as the conditions for the region to be cut out from the captured image with respect to the image generation unit 177, the image generation unit 177 does not depend on the detection result of the target detection unit 171. Cut out the area of the set position and size. In addition, the condition of the region to be cut out may be a condition for determining a color or luminance in the captured image, and does not necessarily require processing by the target detection unit 171.
Further, the process performed when the image generation unit 177 generates a display image from the captured image is not limited to the process of cutting out and enlarging a part of the captured image. For example, the entire captured image may be used as it is or as a display image after resolution conversion. Further, for example, the image generation unit 177 may cut out a plurality of regions from the captured image to generate a plurality of display images, and simultaneously display the plurality of images IV in the visual field VR. Also, the position where the display image is displayed may be changed or adjusted as appropriate by the image generation unit 177 or the image display control unit 176.

  Further, the image display control unit 176 may control the movement of the image displayed by the image display unit 20. For example, the image display control unit 176 may control the image displayed on the image display unit 20 based on the movement of the image display unit 20 that can be detected from the detection values of the 9-axis sensor 66 and / or the GPS 115. The image display control unit 176 displays an image to be displayed on the image display unit 20 when the acceleration or speed of the movement of the image display unit 20 exceeds a threshold value while displaying an image generated from the captured image of the camera 61. The image displayed in the direction opposite to the movement of the image display unit 20 may be shifted. In this case, the image display control unit 176 may change the display image generated by the image generation unit 177 or may display a part of the display image. According to this configuration, when the movement of the head of the user wearing the head-mounted display device 100 is fast and the change in the display image becomes severe, the movement of the image displayed by the image display unit 20 is suppressed, User discomfort can be reduced. This function can also be applied when the object detected by the object detection unit 171 moves at a high speed in the captured image. For example, when a ball is detected as an object as illustrated in FIGS. 5A to 5C, a moving speed is obtained from a change in the position of the ball in the captured image, and the acceleration or speed is equal to or greater than a threshold value. The change of the image to be displayed or the movement of the ball at the displayable position of the image display unit 20 may be suppressed. Alternatively, even when the moving speed or acceleration obtained from the change in the position of the ball in the captured image is greater than or equal to the threshold value, the image generation unit 177 may temporarily stop the process of extracting the target image and generating the display image. Good. Further, even if the image generation unit 177 decreases the enlargement ratio of the extracted image, changes the angle of view of the camera 61, or reduces the zoom magnification when the camera 61 has a zoom mechanism or a digital zoom function. Good. Thus, by reducing the display size of the image of the object displayed by the image display unit 20, an effect that the object can be easily seen even when the moving speed of the object is fast can be expected.

  Further, for example, in the above-described embodiment, the example in which the image generation unit 177 determines the enlargement ratio and cuts out the captured image according to the enlargement ratio has been described. For example, the camera 61 may have a zoom function, and the image generation unit 177 may control the zoom magnification of the camera 61. In this case, the image generation unit 177 may acquire a captured image captured at a zoom magnification specified by the image generation unit 177, and the image may be cut out to be a display image.

  Further, as the image display unit, instead of the image display unit 20, another type of image display unit such as an image display unit worn like a hat may be adopted, corresponding to the left eye of the user. What is necessary is just to provide the display part which displays an image, and the display part which displays an image corresponding to a user's right eye. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Moreover, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet, or may be a head-up display (HUD) used for a windshield of an automobile. Furthermore, the user's eyeball forms an image on the retina such as a so-called contact lens type display that is worn on the user's eyes (for example, on the cornea) and an implant type display that is embedded in the eyeball. The display may be the image display unit 20.

In addition, the present invention may be applied to the user's body, and can be applied regardless of whether support by another method is required. For example, a binocular-type hand held display (Hand Held Display) that the user uses with both hands may be adopted as the image display unit 20 of the present application. This type of display needs to be held by the user in order to keep it attached to the user, but should be attached to the user's head or face when the user looks at the display. Therefore, it is included in the display device of the present invention. Furthermore, even if the display device is fixed to the floor or wall surface by a support leg or the like, as long as it can be attached to the user's head or face when the user views the display on the display, the display of the present invention Included in the device.
Further, only the image display unit 20 or a display unit including a configuration related to image display in the image display unit 20 is mounted on the user's body, and the control device 10 other than the display unit, or the control device 10 and the control unit. The control system including 140 may be physically configured as a separate body. For example, an image display unit 20 or a display unit that is a part of the image display unit 20 may be wirelessly connected to a device having another control system, and the display device may be similar to the head-mounted display device 100. As an apparatus provided with this control system, a conventional electronic device such as a smartphone, a mobile phone, a tablet computer, or a personal computer of another shape can be used. Of course, this type of display device can also be applied.

Furthermore, in the above embodiment, the image display unit 20 and the control device 10 are separated and connected by way of the connection unit 40, but the control device 10 and the image display unit 20 are integrated. It is also possible to be configured to be mounted on the user's head.
In addition, the control device 10 and the image display unit 20 are connected by a longer cable or a wireless communication line. As the control device 10, a notebook computer, a tablet computer or a desktop computer, a game machine, a portable phone, a smartphone, A portable electronic device including a portable media player, other dedicated devices, or the like may be used.

  Further, for example, as a configuration for generating image light in the image display unit 20, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used, or an LCOS (Liquid crystal) may be provided. On silicon and LCoS are registered trademarks), and digital micromirror devices can also be used. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. That is, the image generation unit includes a laser light source and an optical system that guides the laser light source to the user's eye, makes the laser light enter the user's eye, scans the retina, and forms an image on the retina. A configuration that allows the user to visually recognize an image may be employed. When a laser retina projection type head-mounted display is employed, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes.

  As an optical system that guides image light to the user's eyes, an optical member that transmits external light that is incident from the outside toward the apparatus and that enters the user's eyes together with the image light can be employed. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.

  Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, as an image display element, a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system May be provided. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, thereby recognizing the image. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

  The display device of the present invention is not limited to a head-mounted display device, and can be applied to various display devices such as a flat panel display and a projector. The display device of the present invention may be any device that allows an image to be visually recognized by external light and image light. For example, a configuration in which an image by image light is visually recognized by an optical member that transmits external light is exemplified. Specifically, the above-described head-mounted display includes an optical member that transmits external light, and a light-transmitting plane or curved surface (glass) that is fixedly or movably installed at a position away from the user. It is also applicable to a display device that projects image light onto a transparent plastic or the like. As an example, a configuration of a display device that projects image light onto a window glass of a vehicle and allows a user who is on board or a user outside the vehicle to visually recognize the scenery inside and outside the vehicle together with an image by the image light. It is done. In addition, for example, image light is projected onto a transparent, semi-transparent, or colored transparent display surface that is fixedly installed such as a window glass of a building, and is displayed together with an image by the image light to a user around the display surface. A configuration of a display device that visually recognizes a scene through the surface can be given.

  In addition, at least a part of each functional block shown in FIG. 2 may be realized by hardware, or may be realized by cooperation of hardware and software, as shown in FIG. It is not limited to a configuration in which independent hardware resources are arranged. The program executed by the control unit 140 may be stored in the storage unit 120 or the storage device in the control device 10, or the program stored in an external device is acquired via the communication unit 117 or the interface 180. It is good also as a structure to execute. Of the configurations formed in the control device 10, only the operation unit 135 may be formed as a single user interface (UI), or the power source 130 in the above embodiment is formed independently and is replaceable. It may be a configuration. Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, the control unit 140 illustrated in FIG. 2 may be formed in both the control device 10 and the image display unit 20, or the control unit 140 formed in the control device 10 and the CPU formed in the image display unit 20. It is good also as a structure by which the function to perform is divided | segmented separately.

  DESCRIPTION OF SYMBOLS 10 ... Control apparatus, 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display drive part, 23 ... Left holding part, 24 ... Left display drive part, 26 ... Right optical image display part, 28 ... Left optical image display unit, 61 ... Camera (imaging unit), 63 ... Microphone, 100 ... Head-mounted display device (display device), 117 ... Communication unit, 120 ... Storage unit, 140 ... Control unit, 150 ... Operating System ... 160 ... Image processing unit 170 ... Audio processing unit 171 ... Target detection unit 173 ... Distance detection unit 176 ... Image display control unit 177 ... Image generation unit 180 ... Interface 190 ... Display control unit 201 ... right backlight control unit, 202 ... left backlight control unit, 211 ... right LCD control unit, 212 ... left LCD control unit, 221 ... right backlight, 222 ... left backlight, 241 ... right LCD, 2 2 ... Left LCD, 251 ... right projection optical system, 252 ... left projection optical system, 261 ... Migishirubekoban, 262 ... Hidarishirubekoban, DA ... data acquisition unit.

Claims (12)

A display device used by being worn on a user's body,
A display unit that displays an image so that it can be seen through the outside scene together with the outside scene;
An imaging unit that images the user's line-of-sight direction;
An image generation unit that generates a display image from a captured image of the imaging unit;
An image display control unit that causes the display unit to display the display image generated by the image generation unit;
A display device comprising: The image generation unit generates the display image by extracting and processing a part of the captured image of the imaging unit;
The display device according to claim 1. The image generation unit generates the display image by cutting out and enlarging a part of the captured image of the imaging unit;
The display device according to claim 2. The image generation unit generates the display image by cutting out a region corresponding to a preset condition in the captured image of the imaging unit;
The display device according to claim 2, wherein: An object detection unit for detecting an object in the direction of the user's line of sight from a captured image of the imaging unit;
The image generation unit extracts and processes a region including an image of the object detected by the target detection unit from a captured image of the imaging unit, and generates the display image;
The display device according to claim 2, wherein: A storage unit for storing the feature amount of the image of the object;
The target detection unit detects the target object by detecting an image suitable for a feature amount stored in the storage unit from a captured image of the imaging unit;
The display device according to claim 5. The feature quantity stored in the storage unit includes any one of a feature quantity of an image of the specific object and general-purpose feature quantities corresponding to a plurality of similar objects.
The display device according to claim 6. A distance detection unit for detecting a distance between the object detected by the object detection unit and the user;
The image display control unit generates a display image by enlarging a part of the captured image of the imaging unit at an enlargement rate according to the distance detected by the distance detection unit;
The display device according to claim 5, wherein: The image generation unit acquires a plurality of captured images captured at different times by the imaging unit, and identifies the region corresponding to a preset condition in the plurality of captured images, thereby obtaining the display image. Generating,
The display device according to claim 1, wherein: The image generation unit acquires a plurality of captured images captured at different times by the imaging unit, and when the difference between the images of the plurality of captured images satisfies a preset condition, the acquired captured image And generating the display image,
The display device according to claim 9. A control method for a display device that includes a display unit that transmits an outside scene and displays an image so that the outside scene can be visually recognized, and is used by being worn on the user's body,
Image the user's line of sight,
Generate a display image by cutting out and enlarging part of the captured image,
Displaying the generated display image on the display unit;
A control method of a display device characterized by the above. A program that can be executed by a computer that controls a display device that is mounted on a user's body and includes a display unit that transmits an outside scene and displays an image so as to be visible together with the outside scene.
The computer,
An image generation unit that generates a display image by cutting out and enlarging a part of the captured image obtained by capturing the user's line-of-sight direction;
An image display control unit that causes the display unit to display an image generated by the image generation unit;
Program to make it function.

Images