JP6229381B2 - Head-mounted display device, method for controlling head-mounted display device, image display system, and information processing device - Google Patents

Description

  The present invention relates to a head-mounted display device.

  A head-mounted display device (Head Mounted Display (HMD)), which is a display device mounted on the head, is known. The head-mounted display device, for example, generates image light representing an image using a liquid crystal display and a light source, and uses the generated image light to a user's eye using a projection optical system, a light guide plate, or the like. By guiding, the user is made to recognize the virtual image. Such a head-mounted display device is connected to an external device such as a smartphone by a wired interface such as MicroUSB (Universal Serial Bus), and is connected to the external device in accordance with a standard such as Mobile High Definition Link (MHL). The video signal is received. Similarly, the head-mounted display device is connected to an external device through a wireless interface such as a wireless LAN, for example, and receives a video signal from the external device in accordance with a standard such as Miracast. The head-mounted display device allows the user of the head-mounted display device to visually recognize the same virtual image as the image (display image) displayed on the display screen of the external device based on the received video signal. be able to.

  In Patent Literature 1, when a predetermined function installed in a portable information terminal is displayed on one of the display screen of the portable information terminal and the display screen of the head-mounted display device, the open / close state of the portable information terminal The structure which determines a display destination according to is described. In Patent Document 2, in order to enable text input with one hand regardless of time and place on the head-mounted display device, means for detecting the motion of each finger and analyzing the detected motion are analyzed. A text input system including means for generating a code such as a character code is described. Patent Document 3 describes a virtual mouse for providing an input unit with excellent intuition and visibility for a head-mounted display device. Patent Document 4 describes a simple operation type wireless data transmission / reception system that transmits and receives electronic data between a plurality of electronic devices by a wireless communication method in a Wi-Fi infrastructure mode.

JP 2013-92781 A JP 2000-284886 A JP 2000-29619 A Japanese Patent No. 5037718

  In the technique described in Patent Document 1, only a predetermined function mounted on the portable information terminal is considered, and there is a problem that the display of the function mounted on the head-mounted display device is not considered. It was. In addition, in the techniques described in Patent Documents 2 to 4, there is a problem that the head-mounted display device does not consider displaying the display image of another device.

  Therefore, a head-mounted display device that allows a user to visually recognize both the display image of the head-mounted display device and the display image of an external device connected to the head-mounted display device is desired. It was rare. In addition, the head-mounted display device has various requests such as usability improvement, versatility improvement, convenience improvement, reliability improvement, and manufacturing cost reduction.

SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.
According to an aspect of the present invention, a head-mounted display device that allows a user to visually recognize a virtual image and an outside scene is provided. The head-mounted display device; an acquisition unit that acquires a first image that is an image displayed on a display screen of the external device from an external device connected to the head-mounted display device ; said first image, said head mounted display device can be closed, generator and for generating a list image and a second image that is an image relating to the operation of the head-mounted display device; wherein An image display unit that forms the virtual image representing the generated list image. The generating unit arranges the first image with respect to the first area according to the definition of area information including a first area and a second area different from the first area, and the second area The list image in which the second image is arranged in the area is generated.
According to the head-mounted display device of this aspect, the image display unit is a first image that is an image displayed on the display screen of the external device from the external device connected to the head-mounted display device. acquires, forming a first image acquired, the head-mounted display device is closed, a virtual image representing the list image contains a second image that is an image relating to the operation of the head-mounted display device And can be visually recognized by the user. Therefore, it is possible to provide a head-mounted display device that allows the user to visually recognize both the display image of the head-mounted display device and the image displayed on the display screen of the external device.
In addition, the generation unit arranges the first image acquired from the external device in the first area in accordance with the definition of the area information including the first area and the second area different from the first area. By arranging the second image of the head-mounted display device in the second region, it is possible to easily generate a list image.
According to one aspect of the present invention, a method for controlling a head mounted display device is provided. The method includes: (a) obtaining a first image which is an image displayed on a display screen of the external device from an external device connected to the head-mounted display device ; and (b) said first image, said head mounted display device is Yes, a step of generating list image and a second image that is an image relating to the operation of the head-mounted display device; (c And forming a virtual image representing the generated list image. In the step (b), the first image is arranged with respect to the first region in accordance with the definition of region information including a first region and a second region different from the first region, The list image in which the second image is arranged with respect to the second region is generated. According to this method, an effect similar to that of the above embodiment can be obtained.
According to an aspect of the present invention, an image display system is provided. The image display system includes: a head-mounted display device that allows a user to visually recognize a virtual image and an outside scene; and an external device connected to the head-mounted display device; the external device; A head unit that acquires a first image that is an image displayed on the display screen of the device and transmits the acquired first image to the head-mounted display device; the display device; and said first image, generating said head mounted display device is closed, generates a list image and a second image that is an image relating to the operation of the head-mounted display device And an image display unit for forming the virtual image representing the generated list image. The generating unit arranges the first image with respect to the first area according to the definition of area information including a first area and a second area different from the first area, and the second area The list image in which the second image is arranged in the area is generated. According to this image display system, it is possible to obtain the same effect as that of the above embodiment.
According to an aspect of the present invention, an information processing apparatus that is connected to a head-mounted display device and generates an image to be displayed on the head-mounted display device is provided. The information processing apparatus includes: a first image that is an image acquired by an external device connected to the head-mounted display device and is displayed on a display screen of the external device; head-mounted display device can be closed, and the second image is an image relating to the operation of the head-mounted display device, an acquisition unit that acquires, a first image in which the acquired, is the acquisition A list image generation unit that generates a list image including the second image; and a list image transmission unit that transmits the generated list image to the head-mounted display device. The list image generation unit is configured to arrange the first image with respect to the first region in accordance with a rule of region information including a first region and a second region different from the first region, The list image in which the second image is arranged with respect to the second region is generated. According to the information processing apparatus of this aspect, the same effect as that of the above aspect can be obtained by using the information processing apparatus connected to the head-mounted display device.
In addition, the present invention can be realized as the following forms.

(1) According to an aspect of the present invention, a head-mounted display device that allows a user to visually recognize a virtual image and an outside scene is provided. The head-mounted display device includes: a first image that is a display image of an external device connected to the head-mounted display device; and a second image that the head-mounted display device has. A generating unit that generates a list image including; and an image display unit that forms the virtual image representing the generated list image. According to the head-mounted display device of this aspect, the image display unit includes the first image that is a display image of an external device connected to the head-mounted display device, and the head-mounted display device. A virtual image representing a list image including the second image that is an image can be formed and made visible to the user. Therefore, it is possible to provide a head-mounted display device that allows the user to visually recognize both the display image of the head-mounted display device and the display image of the external device.

(2) The head-mounted display device according to the above aspect further includes: an acquisition unit that acquires the first image from the external device; and the generation unit is configured to acquire the acquired first image with respect to a first region. The list image may be generated in which one image is arranged and the second image is arranged in a second area different from the first area. According to the head-mounted display device of this aspect, the generation unit arranges the first image acquired from the external device with respect to the first region, and the head-mounted display with respect to the second region. By arranging the second image of the device, it is possible to easily generate a list image.

(3) In the head-mounted display device according to the above aspect; the generation unit may use a current display image on the head-mounted display device as the second image. According to the head-mounted display device of this aspect, the generation unit can generate a list image using the display image of the head-mounted display device as it is as the second image. For this reason, the processing content in a production | generation part can be simplified.

(4) In the head-mounted display device according to the above aspect; the generation unit may generate the second image by changing an arrangement of icon images included in the head-mounted display device. According to the head-mounted display device of this aspect, the generation unit generates the second image in which the aspect ratio is freely changed by changing the arrangement of the icon images included in the head-mounted display device. The list image can be generated using the generated second image.

(5) In the head-mounted display device of the above aspect; when the generation unit generates the second image, a change in shape and a change in transmittance with respect to the icon image; You may implement at least any one of a change of a color, a change of a magnitude | size, and the provision of a decoration. According to the head-mounted display device of this aspect, when the generation unit generates the second image, the generation unit changes the shape and changes the transmittance of the icon image of the head-mounted display device. Then, at least one of color change, size change, and decoration is performed. As a result, the user can easily identify the icon image of the head-mounted display device in the list image.

(6) In the head-mounted display device of the above aspect; the generation unit further changes the size of at least one of the first image and the second image, and after the change The list image may be generated using the image. According to the head-mounted display device of this aspect, the generation unit can change the size of the first image in accordance with the size of the first region. In addition, the generation unit can change the size of the second image in accordance with the size of the second region.

(7) In the head-mounted display device according to the above aspect; the generation unit may further change a shape and transmit at least one of the first image and the second image. At least one of rate change, color change, size change, and decoration may be implemented, and the list image may be generated using the image after the implementation. According to the head-mounted display device of this aspect, when generating the list image, the generation unit changes the shape and transmits the transmittance for at least one of the first image and the second image. At least one of a change, a color change, a size change, and a decoration can be implemented. As a result, the user can easily identify the first image and the second image in the list image.

(8) In the head-mounted display device according to the above aspect, the operation acquisition unit that acquires the user's operation on the list image; and the acquired operation is an operation on the first image In this case, a notification unit that notifies the external device of the operation may be provided. According to the head-mounted display device of this form, the notification unit notifies the external device of the user's operation on the first image of the list image. For this reason, since the user can operate the external device via the input interface of the head-mounted display device, the usability of the head-mounted display device can be improved.

(9) In the head-mounted display device according to the above aspect; the image display unit forms the virtual image in which a pointer image is further superimposed on the list image; and the generation unit includes the first image The pointer image superimposed on the image and the pointer image superimposed on the second image may be different from each other. According to the head-mounted display device of this aspect, the image display unit uses the pointer image superimposed on the first image and the pointer image superimposed on the second image as different images (virtual images). Can be visually recognized. As a result, the user can easily identify whether the user is operating in the list image is the external device represented by the first image or the head-mounted display device represented by the second image. Become.

(10) According to an aspect of the present invention, an image display system is provided. The image display system includes: a head-mounted display device that allows a user to visually recognize a virtual image and an outside scene; and an external device connected to the head-mounted display device; the external device; A transmission unit configured to acquire a first image that is a display image of the device and to transmit the acquired first image to the head-mounted display device; the head-mounted display device; A generating unit that generates a list image including one image and a second image included in the head-mounted display device; and an image display unit that forms the virtual image representing the generated list image; Prepare.

(11) According to one aspect of the present invention, there is provided an information processing apparatus that is connected to a head-mounted display device and generates an image to be displayed on the head-mounted display device. This information processing apparatus acquires a first image that is a display image of an external device connected to the head-mounted display device, and a second image that the head-mounted display device has A list image generation unit that generates a list image including the acquired first image and the acquired second image; and the generated list image as the head-mounted display device. A list image transmitting unit for transmitting to According to the information processing apparatus of this aspect, the same effect as that of the above aspect can be obtained by using the information processing apparatus connected to the head-mounted display device.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with a new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including a part or all of the two elements of the generation unit and the image display unit. That is, this apparatus may or may not have the generation unit. Moreover, this apparatus may or may not have an image display unit. Such a device can be realized, for example, as a head-mounted display device, but can also be realized as a device other than the head-mounted display device. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can be realized in various modes. For example, a head-mounted display device, a method for controlling the head-mounted display device, an image display system using the head-mounted display device, The present invention can be realized in the form of a computer program for realizing the functions of these methods, apparatuses or systems, a recording medium on which the computer program is recorded, and the like.

It is explanatory drawing which shows schematic structure of the image display system 1000 in one Embodiment of this invention. 2 is a block diagram functionally showing a configuration of a head mounted display 100. FIG. 7 is an explanatory diagram illustrating an example of region information stored in a region information storage unit 122. FIG. It is explanatory drawing which shows an example of the virtual image visually recognized by the user. It is a sequence diagram which shows the procedure of an arrangement | positioning process. It is explanatory drawing which shows a mode that the list image is displayed on the head mounted display. It is explanatory drawing for demonstrating the change of the pointer image in a list image. It is a sequence diagram which shows the procedure of a notification process. It is explanatory drawing for demonstrating step S202 of a notification process. It is a block diagram which shows functionally the structure of the head mounted display 100a in 2nd Embodiment. It is explanatory drawing which shows an example of the flame | frame memorize | stored in the frame information. It is a sequence diagram which shows the procedure of the arrangement | positioning process in 2nd Embodiment. It is explanatory drawing which shows a mode that the list image is displayed in the head mounted display 100a. It is explanatory drawing which shows the structure of the external appearance of the head mounted display in a modification.

A. First embodiment:
A-1. Image display system configuration:
FIG. 1 is an explanatory diagram showing a schematic configuration of an image display system 1000 according to an embodiment of the present invention. The image display system 1000 includes a head-mounted display device 100 and a portable information terminal 300 as an external device. The image display system 1000 is a system that causes the head-mounted display device 100 to display a list image including the display image of the portable information terminal 300 and the image that the head-mounted display device 100 has. Here, the “display image of the smartphone 300” means an image displayed on the display screen of the smartphone 300. Note that the display image of the smartphone 300 includes an image that should be displayed on the display screen but is not displayed as a result of external output. Further, “an image that the head-mounted display device 100 has” means an image that is currently displayed on the display screen of the head-mounted display device 100. The display image of the head-mounted display device 100 includes an image that should be displayed on the display screen but is not displayed as a result of external output.

The head-mounted display device 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The head mounted display 100 according to the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. The portable information terminal 300 is a portable information communication terminal. In the present embodiment, a smartphone is exemplified as the portable information terminal. The head mounted display 100 and the smartphone 300 are connected so that wireless communication or wired communication is possible.

A-2. Configuration of head mounted display device:
FIG. 2 is a block diagram functionally showing the configuration of the head mounted display 100. As shown in FIGS. 1 and 2, the head mounted display 100 includes an image display unit 20 that allows the user to visually recognize a virtual image while being mounted on the user's head, and a control unit (controller) that controls the image display unit 20. ) 10. The image display unit 20 and the control unit 10 are connected by a connection unit 40 and transmit various signals through the connection unit 40. A metal cable or an optical fiber can be used for the connection part 40.

A-2-1. Configuration of control unit:
The control unit 10 is a device for controlling the head mounted display 100. The control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a wireless communication unit 132, a GPS module 134, a CPU 140, an interface 180, and transmission units (Tx) 51 and 52. These parts are connected to each other by a bus (not shown) (FIG. 2).

  The input information acquisition unit 110 detects, for example, a user's gesture using a touch pad, a cross key, a foot switch (a switch operated by a user's foot), gesture detection (a camera or the like), and associates the gesture with the gesture. Command input), eye gaze detection (detecting the user's gaze using an infrared sensor, etc., obtaining command operation input associated with gaze movement), operations on input devices such as microphones, etc. Get the signal according to the input. At the time of gesture detection, a user's fingertip, a ring attached to the user's hand, a tool held by the user, or the like may be used as a mark for motion detection. If the operation input by foot switch, line-of-sight detection, and microphone is acquired, it is necessary to perform on-site work such as medical site, construction industry, and manufacturing industry where it is difficult for the user to operate with hands. The convenience of the user when the head mounted display 100 is used can be greatly improved.

  The storage unit 120 includes a ROM, a RAM, a DRAM, a hard disk, and the like. The storage unit 120 includes a region information storage unit 122. The area information storage unit 122 stores at least one area information. The area information is information that defines the area of the list image generated in the arrangement process (FIG. 5). In other words, the area information is used as a frame for arranging an image of the head mounted display 100 and a display image of the smartphone 300.

  FIG. 3 is an explanatory diagram illustrating an example of area information stored in the area information storage unit 122. The area information AI shown in FIG. 3 has a rectangular area (hereinafter also referred to as “area AI”). The area AI preferably has the same aspect ratio as that of the display elements of the head mounted display 100 (FIG. 2: right LCD 241 and left LCD 242).

  The region AI includes two regions, a first region AR1 and a second region AR2. 1st area | region AR1 is an area | region where the display image of the smart phone 300 is arrange | positioned in arrangement | positioning processing (FIG. 5). The second area AR2 is an area in which an image included in the head mounted display 100 is arranged in the arrangement process. In the example of FIG. 3, the first area AR1 is arranged on one side of the area AI equally divided in the horizontal direction, and the second area AR2 is arranged on the other side of the area AI equally divided in the horizontal direction. That is, the first area AR1 and the second area AR2 are areas having the same size.

  In addition, arrangement | positioning and magnitude | size of 1st area | region AR1 and 2nd area | region AR2 which were shown in FIG. 3 are examples, and can be defined arbitrarily. For example, the first area AR1 and the second area AR2 may have a size obtained by dividing the area AI into n equal parts (n is an integer of 3 or more) in the horizontal direction. In this case, when the list image generated using the area information AI is displayed on the head mounted display 100, the first area AR1 and the second area AR2 are not limited to the user's view. It is preferable to arrange at the end (left and right end) of the region AI. Further, the first area AR1 and the second area AR2 may have a size obtained by equally dividing the area AI into m (m is an integer of 2 or more) in the vertical direction. Here, when m is 3 or more, it is preferable that the first region AR1 and the second region AR2 are arranged at the end portions (upper and lower end portions) of the region AI. The first area AR1 and the second area AR2 may have different sizes. In addition, the first region AR1 and the second region AR2 may overlap at least part of the region.

  The power supply 130 supplies power to each part of the head mounted display 100. As the power supply 130, for example, a secondary battery can be used. The wireless communication unit 132 communicates with an external device in accordance with a predetermined wireless communication standard (for example, infrared, short-range wireless communication exemplified by Bluetooth (registered trademark), wireless LAN exemplified by IEEE 802.11, etc.). Wireless communication with The external device means a device other than the head mounted display 100, and includes a tablet, a personal computer, a game terminal, an AV (Audio Video) terminal, a home appliance, etc. in addition to the smartphone 300 shown in FIG. The GPS module 134 detects the current position of the user of the head mounted display 100 by receiving a signal from a GPS satellite, and generates current position information representing the current position information of the user. The current position information can be realized by coordinates representing latitude and longitude, for example.

  The CPU 140 reads out and executes a computer program stored in the storage unit 120, thereby generating a generation unit 142, a notification unit 144, an operating system (OS) 150, an image processing unit 160, an audio processing unit 170, and a display control unit 190. Function as.

  The generation unit 142 generates a list image using the image of the head mounted display 100 and the display image of the smartphone 300 in the arrangement process (FIG. 4). In the notification process (FIG. 8), the notification unit 144 notifies the smartphone 300 of the operation content when an operation is performed on the display image of the smartphone 300 among the list images.

  The image processing unit 160 generates a signal based on a video signal input via the OS 150 from the generation unit 142, the interface 180, the wireless communication unit 132, or the like. The image processing unit 160 controls the display on the image display unit 20 by supplying the generated signal to the image display unit 20 via the connection unit 40. The signal supplied to the image display unit 20 differs between the analog format and the digital format.

  For example, in the case of a digital format, a video signal in a state where a digital R signal, a digital G signal, a digital B signal, and a clock signal PCLK are synchronized is input. The image processing unit 160 performs various color tone correction processes such as a known resolution conversion process and brightness saturation adjustment on image data Data composed of a digital R signal, a digital G signal, and a digital B signal. Image processing such as stone correction processing is executed. Thereafter, the image processing unit 160 transmits the clock signal PCLK and the image data Data via the transmission units 51 and 52.

  In the case of an analog format, a video signal including an analog R signal, an analog G signal, an analog B signal, a vertical synchronization signal VSync, and a horizontal synchronization signal HSync is input. The image processing unit 160 separates the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the input signal, and generates a clock signal PCLK using a PLL circuit (not shown) or the like according to their periods. The image processing unit 160 converts the analog R signal, the analog G signal, and the analog B signal into digital signals using an A / D conversion circuit or the like. The image processing unit 160 performs well-known image processing on the image data Data composed of the converted digital R signal, digital G signal, and digital B signal as necessary, and then outputs the clock signal PCLK and the image data Data. The vertical synchronization signal VSync and the horizontal synchronization signal HSync are transmitted via the transmission units 51 and 52. Hereinafter, the image data Data transmitted through the transmission unit 51 is also referred to as “right eye image data Data1”, and the image data Data transmitted through the transmission unit 52 is referred to as “left eye image data Data2”. Also called.

  The display control unit 190 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24 included in the image display unit 20. The control signals include driving ON / OFF of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, driving ON / OFF of the left LCD 242 by the left LCD control unit 212, This is a signal for individually switching driving ON / OFF of the left backlight 222 by the left backlight control unit 202. The display control unit 190 controls the generation and emission of image light in the right display drive unit 22 and the left display drive unit 24 based on these control signals. The display control unit 190 transmits the generated control signal via the transmission units 51 and 52.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and supplies the acquired audio signal to a speaker (not shown) of the right earphone 32 and a speaker (not shown) of the left earphone 34.

  The interface 180 is a predetermined wired communication standard (for example, MicroUSB (Universal Serial Bus), USB, HDMI (High Definition Multimedia Interface, HDMI is a registered trademark), DVI (Digital Visual Interface), VGA (Video Graphics Array), composite, In accordance with RS-232C (Recommended Standard 232), a wired LAN exemplified by IEEE 802.3, etc.), wireless communication is performed with an external device. The external device means a device other than the head-mounted display 100, and includes the smartphone 300 shown in FIG. 1, a tablet, a personal computer, a game terminal, an AV terminal, a home appliance, and the like.

A-2-2. Configuration of image display:
The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right optical image display unit 26 (FIG. 1), a left optical image display unit 28 (FIG. 1), and a 9-axis sensor 66. Contains.

  The right display drive unit 22 and the left display drive unit 24 are arranged at positions facing the user's temple when the user wears the image display unit 20. The right display driving unit 22 and the left display driving unit 24 in the present embodiment generate and emit image light representing an image using a liquid crystal display (hereinafter referred to as “LCD”) or a projection optical system. . The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are included.

  The receiving unit 53 receives data transmitted from the transmitting unit 51. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the input clock signal PCLK, right-eye image data Data1, the vertical synchronization signal VSync, and the horizontal synchronization signal HSync. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby converting the illumination light emitted from the right backlight 221 into an image. Modulate into effective image light to represent. The right projection optical system 251 is configured by a collimating lens that converts the image light emitted from the right LCD 241 into a light beam in a parallel state.

  The left display drive unit 24 has substantially the same configuration as the right display drive unit 22 and operates in the same manner as the right display drive unit 22. That is, the left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, and a left LCD control unit 212 that functions as a display element. And a left LCD 242 and a left projection optical system 252. Detailed description is omitted. In this embodiment, the backlight method is adopted, but image light may be emitted using a front light method or a reflection method.

  The right optical image display unit 26 and the left optical image display unit 28 are disposed so as to be positioned in front of the left and right eyes of the user when the user wears the image display unit 20 (see FIG. 1). The right optical image display unit 26 includes a right light guide plate 261 and a light control plate (not shown). The right light guide plate 261 is formed of a light transmissive resin material or the like. The right light guide plate 261 guides the image light output from the right display driving unit 22 to the right eye RE of the user while reflecting the image light along a predetermined optical path. For the right light guide plate 261, a diffraction grating or a transflective film may be used. The light control plate is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20. The light control plate protects the right light guide plate 261 and suppresses damage to the right light guide plate 261 and adhesion of dirt. Further, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The left optical image display unit 28 has substantially the same configuration as the right optical image display unit 26 and operates in the same manner as the right optical image display unit 26. That is, the left optical image display unit 28 includes a left light guide plate 262 and a light control plate (not shown), and guides the image light output from the left display drive unit 24 to the left eye LE of the user. Detailed description is omitted.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of detecting the movement of the user's head of the head mounted display 100 is detected. Functions as a detection unit. Here, the movement of the head includes changes in the speed, acceleration, angular velocity, direction, and direction of the head.

  FIG. 4 is an explanatory diagram illustrating an example of a virtual image visually recognized by the user. FIG. 4 illustrates the visual field VR of the user. As described above, the image light guided to both eyes of the user of the head mounted display 100 forms an image on the retina of the user, so that the user can visually recognize the virtual image VI. In the example of FIG. 4, the virtual image VI is an OS standby screen of the head mounted display 100. Further, the user views the outside scene SC through the right optical image display unit 26 and the left optical image display unit 28. As described above, the user of the head mounted display 100 of the present embodiment can see the virtual image VI and the outside scene SC behind the virtual image VI for the portion of the visual field VR where the virtual image VI is displayed. Further, the user can directly see the outside scene SC through the right optical image display unit 26 and the left optical image display unit 28 for the portion of the visual field VR where the virtual image VI is not displayed. In the present specification, “the head mounted display 100 displays an image” includes allowing the user of the head mounted display 100 to visually recognize a virtual image.

A-3. Placement process:
FIG. 5 is a sequence diagram showing the procedure of the arrangement process. The arrangement process is a process of generating a list image in which an image of the head mounted display 100 and a display image of the smartphone 300 are arranged and displaying the generated list image on the head mounted display 100. The arrangement process is mainly executed by the generation unit 142.

  In step S100, an application for performing arrangement processing is activated. The activation of the application in step S100 may be triggered by the input information acquisition unit 110 detecting the activation operation by the user, or may be triggered by detection of an activation command from another application. By starting the application in step S100, the functions of the generation unit 142 and the notification unit 144 are realized for the CPU 140.

  In step S102, the wireless communication unit 132 or the interface 180 detects that the smartphone 300 is connected. Hereinafter, a case where the head mounted display 100 and the smartphone 300 communicate using a wireless LAN compliant with IEEE 802.11 will be exemplified. In step S <b> 104, the generation unit 142 authenticates the smartphone 300 connected via the wireless communication unit 132. This authentication can be performed using various authentication techniques. For example, the generation unit 142 may authenticate the smartphone 300 using a MAC (Media Access Control) address of the smartphone 300, or may authenticate the smartphone 300 using a user name and a password. Further, the generation unit 142 may authenticate the smartphone 300 using a digital certificate issued by the certificate authority, and recognize the user's physical characteristics (face, fingerprint, voiceprint) and authenticate the smartphone 300. Also good. After the authentication in step S104 is successful, in step S106, the generation unit 142 establishes a connection with the smartphone 300.

  In step S108, the smartphone 300 acquires the display image of the smartphone 300. Specifically, the smartphone 300 renders a screen displayed on the smartphone 300, such as content being played back on the smartphone 300 or an application GUI (Graphical User Interface), according to a predetermined standard, and acquires a frame image. The frame image acquired in step S108 is a display image of the smartphone 300, and is hereinafter also referred to as “first image”.

  In step S110, the smartphone 300 transmits the acquired first image to the head mounted display 100. The generation unit 142 of the head mounted display 100 acquires the first image via the wireless communication unit 132. At this time, the generation unit 142 and the wireless communication unit 132 function as an “acquisition unit”.

  In step S <b> 112, the generation unit 142 acquires an image currently displayed on the display screen of the head mounted display 100 as an image that the head mounted display 100 has. Specifically, the generation unit 142 acquires a frame image using the same method as in step S108. The generation unit 142 may acquire a frame image directly from the image processing unit 160 or the video memory. The frame image acquired in step S112 is an image that the head mounted display 100 has, and is hereinafter also referred to as a “second image”.

  In step S114, the generation unit 142 enlarges or reduces the first image and the second image. Specifically, the generation unit 142 enlarges or reduces the first image acquired in step S110 so as to match the size of the first area AR1 of the area information AI (FIG. 3). Similarly, the generation unit 142 enlarges or reduces the second image acquired in step S112 so as to match the size of the second area AR2 of the area information AI. In addition, it is preferable that the production | generation part 142 is expanded or reduced, maintaining the aspect ratio of a 1st image and a 2nd image, respectively. If it does so, the production | generation part 142 can produce | generate the list image which faithfully reproduced the display image of the smart phone 300, and the image (display image) which the head mounted display 100 has.

  In step S116, the generation unit 142 arranges the first image in the first area AR1 of the area information AI, arranges the second image in the second area AR2 of the area information AI, and generates a list image. . At this time, the generation unit 142 may perform processing as exemplified in the following a1 to a5 on at least one of the first image and the second image. In addition, process a1-a5 may be employ | adopted independently and may be employ | adopted in combination.

(A1) The generation unit 142 changes the shapes of the first and second images. For example, the generation unit 142 can change the rectangular first and second images into a circular shape or a trapezoidal shape.
(A2) The generation unit 142 changes the transmittance of the first and second images. If the transmittances of the first and second images are changed, it is possible to prevent the user's field of view from being obstructed by the list image when the list image displayed as a virtual image is viewed.
(A3) The generation unit 142 performs color conversion processing on the first and second images. For example, the image display unit 20 is configured to include a camera that captures an external scene (external scenery) in the user's visual field direction and acquires an external scene image. Then, the generation unit 142 performs a color conversion process for enhancing or weakening the complementary color of the outside scene image with respect to the first and second images. In this way, the generation unit 142 can make the first and second images stand out from the outside scene.

(A4) The generation unit 142 changes the size of the first and second images. For example, the generation unit 142 enlarges or reduces the first and second images regardless of the size of the first and second areas of the area information AI.
(A5) The generation unit 142 adds decorations such as characters, figures, and symbols to the first and second images. For example, the generation unit 142 can add a character for explaining the image to the first and second images. For example, the generation unit 142 may add a frame that borders the periphery of the first and second images.

  In this way, if processing such as a1 to a5 is performed on one of the first image and the second image, the user who visually recognizes the list image is The first image and the second image can be easily distinguished from each other. Moreover, even if it processes like said a1-a5 using a different aspect with respect to both a 1st image and a 2nd image, the discriminability in a user can be improved similarly. .

  In step S118, the generation unit 142 displays a list image on the head mounted display 100. Specifically, the generation unit 142 transmits the list image generated in step S116 to the image processing unit 160. The image processing unit 160 that has received the list image performs the above-described display processing. As a result, the image light guided to both eyes of the user of the head mounted display 100 forms an image on the retina of the user, so that the user of the head mounted display 100 visually recognizes the virtual image of the list image in the field of view. Can do. In other words, the head mounted display 100 can display a list image.

  FIG. 6 is an explanatory diagram showing a state in which a list image is displayed on the head mounted display 100. As shown in FIG. 6, the user of the head mounted display 100 arranges the first image IM1 in the first area AR1 of the area information AI (FIG. 3) and the second area AR2 in the visual field VR. The list image in which the second image IM2 is arranged can be visually recognized as the virtual image VI. In addition, a decoration using a thick frame BC is added to the second image IM2. For this reason, the user can easily identify the first image and the second image.

  Note that when applying the arrangement processing of FIG. 5 to a moving image, the above steps S108 to S118 are repeatedly executed. When a standard for compressing moving images is used, a difference image (interframe difference) representing a portion changed from the original image or the frame image may be transmitted in step S108. In this case, the generation unit 142 may perform a process of synthesizing the frame image using the previous frame image and the acquired inter-frame difference between steps S110 and S112.

  FIG. 7 is an explanatory diagram for explaining a change in the pointer image in the list image. FIG. 7A shows a pointer image PO1 superimposed on the first image IM1. In the illustrated example, the pointer image PO1 is a figure representing a double circle. FIG. 7B shows a pointer image PO2 superimposed on the second image IM2. In the illustrated example, the pointer image PO2 is a figure in which a smile is drawn in a circle. As described above, in step S118 of the arrangement process (FIG. 5), the generation unit 142 transmits the list image to the image processing unit 160 via the OS 150. At this time, the OS 150 draws a pointer image superimposed on the list image in accordance with a user operation acquired from the input information acquisition unit 110. The generation unit 142 causes the OS 150 to make the pointer image drawn superimposed on the first image IM1 of the list image different from the pointer image drawn superimposed on the second image IM2. To instruct.

  As a result, as shown in FIGS. 7A and 7B, the image display unit displays the pointer image PO1 superimposed on the first image IM1 of the list image and the pointer image superimposed on the second image IM2. The user can visually recognize PO2 as a different virtual image VI. In this way, the user of the head-mounted display 100 is operating the smartphone 300 (external device) represented by the first image IM1 in the list image, or the second image IM2 is It becomes easy to identify whether the head mounted display 100 (head-mounted display device) is represented.

  As described above, according to the arrangement processing of the first embodiment, the image display unit 20 is a display image of the smartphone 300 (external device) connected to the head mounted display 100 (head-mounted display device). A virtual image VI that represents a list image (FIG. 6) including the first image and the second image that is the image of the head mounted display 100 can be formed and made visible to the user. For this reason, the head mounted display 100 which enables a user to visually recognize both the display image of the head mounted display 100 and the display image of the smart phone 300 can be provided.

  Furthermore, according to the arrangement process of the first embodiment, the generation unit 142 uses the first information acquired from the smartphone 300 (external device) for the first area AR1 based on the area information AI (FIG. 3). By arranging the images and arranging the second image of the head mounted display 100 (head-mounted display device) in the second area AR2, it is possible to easily generate a list image.

  Furthermore, according to the arrangement process of the first embodiment, the generation unit 142 displays the frame image currently displayed on the display screen of the head mounted display 100 as an image that the head mounted display 100 (head-mounted display device) has. (Display image) is acquired, and the acquired display image is directly used as the second image to generate a list image. For this reason, the processing content of the arrangement | positioning process in a production | generation part can be simplified.

A-4. Notification processing:
FIG. 8 is a sequence diagram showing the procedure of the notification process. The notification process is a process of notifying the smartphone 300 or the head-mounted display 100 of the operation content when an operation is performed on the first and second images in the list image. The notification process is mainly executed by the notification unit 144.

  In step S200, the input information acquisition unit 110 detects a user operation (for example, click, double-click, drag, on-focus, tap, double-tap, flick, etc.) on the list image, and the operation coordinates (x, y). To get. At this time, the input information acquisition unit 110 functions as an “operation acquisition unit”.

In step S202, the notification unit 144 receives the coordinates (x, y) from the input information acquisition unit 110, and executes the following procedures i and ii.
(I) Based on the received coordinates, it is specified whether the image on which the operation has been performed is the first image or the second image.
(Ii) Based on the received coordinates, the coordinates in the image before enlargement / reduction in step S114 of the arrangement process (FIG. 5) are obtained.

  FIG. 9 is an explanatory diagram for explaining step S202 of the notification process. FIG. 9A is an explanatory diagram for explaining the procedure i. In step S202 of the notification process, the notification unit 144 receives the coordinate CO1 (x1, y1) from the input information acquisition unit 110. The coordinate CO1 is a numerical value representing the amount of change in the x direction and the amount of change in the y direction when the coordinates of the upper left corner of the list image are (0, 0). The notifying unit 144 determines whether the coordinate CO1 (x1, y1) is located on the first image IM1 or the second image IM2 based on the processing content in step S116 of the arrangement processing (FIG. 5). Identify. In the example of FIG. 9A, the coordinate CO1 (x1, y1) is located on the first image IM1, that is, the display image of the smartphone 300.

  FIG. 9B is an explanatory diagram for explaining the procedure ii. The notification unit 144 performs conversion opposite to the conversion performed in step S114 on the list image using the enlargement / reduction ratio used in step S114 of the arrangement process (FIG. 5). That is, if the list image is reduced in step S114, the list image is enlarged. If the list image is enlarged in step S114, the list image is reduced. Thereafter, the notification unit 144 sets the coordinates when the coordinates of the upper left end of the image (that is, the first image IM1 or the second image IM2) specified in FIG. 9A are (0, 0). A coordinate CO2 (x2, y2) corresponding to the same position as CO1 (x1, y1) is obtained.

  In step S204, the notification unit 144 determines whether the image specified in step S202 is a second image, that is, an image (display image) included in the head mounted display 100. In the case of the second image, the notification unit 144 gives the OS 150 the coordinates CO2 (x2, y2) and the operation content (for example, click, double click, drag, on focus, tap, double tap, flick, etc.). Send. In the OS 150, processing such as application activation is performed in accordance with the received coordinates and operation content.

  In step S206, the notification unit 144 determines whether the image specified in step S202 is the first image, that is, the display image of the smartphone 300. In the case of the first image, the notification unit 144 transmits the coordinates CO2 (x2, y2) and the operation content to the smartphone 300. In the smartphone 300, processing such as activation of an application is performed according to the received coordinates and operation content (step S208).

  As described above, according to the notification process of the first embodiment, the notification unit 144 not only notifies the OS 150 of the user's operation on the second image IM2 (FIG. 9) of the list image, but also the first The user's operation on the image IM1 (FIG. 9) can be notified to the smartphone 300 (external device). Therefore, the user not only operates the head mounted display 100 via the input interface of the head mounted display 100 (head mounted display device), but also inputs the head mounted display 100 (head mounted display device). The smartphone 300 can also be operated through the interface. As a result, for example, the user can operate the external device with the external device (the smartphone 300 in this embodiment) connected to the head mounted display 100 in a bag or pocket, Usability in the display 100 can be improved.

B. Second embodiment:
In the second embodiment of the present invention, a configuration in which a second image generated from an icon image is used as the “image of the head-mounted display device” will be described. Below, only the part which has a different structure and operation | movement from 1st Embodiment is demonstrated. In the figure, the same components as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment described above, and detailed description thereof is omitted.

B-1. Image display system configuration:
The schematic configuration of the image display system in the second embodiment is the same as that of the first embodiment shown in FIG.

B-2. Configuration of head mounted display device:
FIG. 10 is a block diagram functionally showing the configuration of the head mounted display 100a in the second embodiment. The difference from the first embodiment shown in FIG. 2 is that a control unit 10 a is provided instead of the control unit 10. The control unit 10 a includes a generation unit 142 a instead of the generation unit 142, includes a notification unit 144 a instead of the notification unit 144, and includes a storage unit 120 a instead of the storage unit 120.

The generation unit 142a is different from the first embodiment described in FIG. The notification unit 144a is different from the first embodiment described in FIG.
The storage unit 120 a further includes frame information 124 in addition to the area information storage unit 122. The frame information 124 stores at least one frame. The frame stored in the frame information 124 is a second image (that is, an image generated by changing the arrangement of the icon images included in the head mounted display 100a) in the arrangement processing (FIG. 12) of the second embodiment. Is used as a frame for arranging icon images.

  In the present embodiment, the “icon image” means an image that expresses the contents of a program or device in an easy-to-understand manner using diagrams, pictures, characters, symbols, and the like. The “icon image” in the present embodiment means an image for starting an application installed in the head mounted display 100a. In addition, the “icon image” includes images drawn by applications (so-called widgets and gadgets) installed in the head mounted display 100a and data (various files stored in the head mounted display 100a). ), An image indicating the presence of a device included in the head mounted display 100a, and the like. May be included. In other words, the icon image can be said to be an abstract symbol of an application (program), data, or device.

  FIG. 11 is an explanatory diagram illustrating an example of a frame stored in the frame information 124. The frame FM1 shown in FIG. 11 has an image list LT1, an image list LT2, and a separation line BA for separating them inside a rectangular area (hereinafter also referred to as “area of the frame FM1”). It has the structure made. The area of the frame FM1 preferably has the same aspect ratio as the second area AR2 of the area information AI.

  The image lists LT1 and LT2 are areas where icon images are actually arranged in the arrangement processing of the second embodiment. In the example of FIG. 3, the image lists LT1 and LT2 are arranged in two stages over the entire lower end of the area of the frame FM1. Note that the image lists LT1 and LT2 can be arranged at any location in the frame FM1. For example, the image lists LT1 and LT2 may be arranged at a part of the lower end of the area of the frame FM1, may be arranged at a part of the entire upper end of the area of the frame FM1, or at the right end of the area of the frame FM1. It may be arranged partly or entirely, may be arranged partly or entirely of the left end of the area of the frame FM1, and may be arranged in all areas of the frame FM1.

  The image list LT1 includes a plurality (five in the illustrated example) of rectangular image frames B1 to B5. The image frames B1 to B5 are arranged so that the long sides of the rectangle are adjacent to each other in the image list LT1. The image list LT2 includes a plurality of rectangular image frames B6 to B10. The image frames B6 to B10 are arranged so that the long sides of the rectangular shape are adjacent to each other in the image list LT2. Image frames B1 to B10 are areas for arranging icon images of the head mounted display 100a in the arrangement processing (FIG. 12) of the second embodiment.

B-3. Placement process:
FIG. 12 is a sequence diagram showing the procedure of the arrangement process in the second embodiment. The only difference from the first embodiment shown in FIG. 5 is that it includes steps S300 and S302 instead of step S112, and the other processing contents are the same as those of the first embodiment.

  In step S300, the generation unit 142a collects icon images included in the head mounted display 100a. Specifically, the generation unit 142a collects a plurality of icon images to be displayed on the standby screen of the head mounted display 100a among the plurality of icon images stored in a predetermined area of the storage unit 120. Examples of the icon image include icons for starting various devices such as a camera and a speaker, various services such as an SNS service, and a mail service.

  In step S302, the generation unit 142a generates the second image by changing the arrangement of the plurality of icon images collected in step S300. Specifically, the generation unit 142a acquires the frame FM1 (FIG. 11) stored in the frame information 124. The generation unit 142a sequentially arranges the acquired icon images in the image frames B1 to B5 of the image list LT1 of the frame FM1 and the image frames B6 to B10 of the image list LT2.

  Note that the generation unit 142a may perform the following processing b1 to b between steps S300 and S302.

(B1) Icon image filtering processing.
For example, the generation unit 142a may obtain a duplicate icon image using image analysis or file name analysis from the icon image collected in step S300, and discard one of the duplicate icon images.
For example, the generation unit 142a may specify the application represented by the icon image from the icon images collected in step S300 using image analysis or file name analysis, and discard the icon image related to the predetermined application.
For example, the generation unit 142a may refer to the information attached to the icon image collected in step S300, obtain the use frequency of the application represented by the icon image, and discard the icon image with a low use frequency.

(B2) Icon image grouping and sorting processing.
For example, the generation unit 142a uses the image analysis or file name analysis from the icon image collected in step S300 to select the type of icon image (application start image, image drawn by an application, data start And the icon images may be grouped or sorted according to the type of the specified icon image.
For example, the generation unit 142a identifies the type and name of the application represented by the icon image using image analysis and file name analysis from the icon image collected in step S300, and displays the icon according to the identified type and name of the application. Images may be grouped or sorted.
For example, the generation unit 142a refers to the information attached to the icon image collected in step S300, obtains the usage frequency of the application represented by the icon image, and groups or sorts the icon images according to the usage frequency. Good.

(B3) Icon image processing.
For example, the generation unit 142a may arbitrarily change the shape, transmittance, and size of the icon image.
For example, the generation unit 142a may perform color conversion processing on the icon image. In this case, the image display unit 20 is configured to include a camera that captures an outside scene (external scenery) in the user's field of view and acquires an outside scene image. Then, the generation unit 142a performs a color conversion process for strengthening the complementary color of the outside scene image with respect to the icon image. In this way, the generation unit 142a can make the icon image stand out from the outside scene.
For example, the generation unit 142a adds decorations such as characters, figures, and symbols to the icon image. Specifically, the generation unit 142a can add a character for explaining the icon image to the icon image. If it does in this way, when the 2nd image currently displayed as a virtual image is visually recognized, it will become easy for a user to identify what each icon image contained in the 2nd image is. The generation unit 142a can also add a frame that borders the icon image to the icon image. In this way, the generation unit 142a can make the icon image stand out from the outside scene.

  FIG. 13 is an explanatory diagram showing a state in which a list image is displayed on the head mounted display 100a. As shown in FIG. 12, the user of the head mounted display 100a of the second embodiment arranges the first image IM1 in the first area AR1 of the area information AI (FIG. 3) in the visual field VR, In the second area AR2, the list image in which the second image IM2 generated using the icon image of the head mounted display 100a is arranged can be visually recognized as a virtual image VI.

  Note that when the arrangement process of FIG. 12 is applied to a moving image, the steps S108 to S118 are repeatedly executed as in the first embodiment. The change of the pointer image in the list image is the same as in the first embodiment.

  As described above, according to the arrangement processing of the second embodiment, the generation unit 142a changes the arrangement of icon images included in the head-mounted display 100 (head-mounted display device), and thus the aspect ratio according to the frame FM1. It is possible to generate a second image IM2 in which is freely changed, and to generate a list image using the generated second image IM2. As a result, the generation unit 142a can generate an optimal second image according to the size of the second area AR2 of the area information AI (FIG. 3) and display it on the image display unit 20.

B-4. Notification processing:
The notification process in the second embodiment is substantially the same as that in the first embodiment described in FIG. However, when the image specified in the procedure i in step S202 is the second image, the procedure ii is replaced with the following iii.
(Iii) The notification unit 144a corresponds to the coordinates CO1 (x1, y1) acquired from the input information acquisition unit 110 based on the processing content (that is, the arrangement of icon images) in step S302 of the arrangement process (FIG. 12). The coordinates CO2 (x2, y2) to be obtained are obtained.

C. Variations:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

・ Modification 1:
In the said embodiment, it illustrated about the structure of the image display system. However, the configuration of the image display system can be arbitrarily determined without departing from the gist of the present invention. For example, addition, deletion, conversion, and the like of each device configuring the image display system can be performed. In addition, it is possible to change the network configuration of the devices constituting the image display system.

  For example, a plurality of external devices (for example, a smartphone and a PDA) may be connected to the head mounted display. In this case, in the same manner as in the first and second embodiments, the generation unit displays the display image of the first external device, the display image of the second external device, and the mth (m is an integer of 3 or more) externals. A list image in which all of the display images of the apparatus are arranged side by side may be generated. In this way, the image display unit visually recognizes a list image in which the image of the head mounted display and the display images of a plurality of external devices connected to the head mounted display are arranged side by side as virtual images. Can be made. For this reason, the convenience of the user in a head mounted display can be improved further.

  For example, a cloud server via the Internet INT may be used instead of the smartphone in the above embodiment. Further, when a plurality of external devices are connected to the head mounted display, a cloud server via the Internet INT may be used as at least one of the external devices. Even in this case, the generation unit can obtain the same effects as those in the first and second embodiments by performing the same processing as in the first and second embodiments.

  For example, the function of the generation unit of the head mounted display of the above embodiment may be provided by an information processing apparatus different from the head mounted display. As the information processing apparatus, for example, a cloud server via the Internet INT may be used. In this case, the information processing apparatus acquires a first image that is a display image of an external device connected to the head-mounted display (head-mounted display device) and a second image that the head-mounted display has. A list image generating unit that generates a list image including the acquiring unit, the acquired first image, and the acquired second image, and a list image that transmits the generated list image to the head mounted display A transmission unit. The list image generation unit generates a list image to be transmitted to the head mounted display by performing the same processing as the head mounted display generation unit described in the above embodiment.

Modification 2
In the said embodiment, it illustrated about the structure of the head mounted display. However, the configuration of the head-mounted display can be arbitrarily determined without departing from the gist of the present invention. For example, each component can be added, deleted, converted, and the like.

  The allocation of components to the control unit and the image display unit in the above embodiment is merely an example, and various aspects can be employed. For example, the following aspects may be adopted. (I) A mode in which processing functions such as a CPU and a memory are mounted on the control unit, and only a display function is mounted on the image display unit. (Ii) Processing functions such as a CPU and a memory are provided in both the control unit and the image display unit. (Iii) a mode in which the control unit and the image display unit are integrated (for example, a mode in which the control unit is included in the image display unit and functions as a glasses-type wearable computer), (iv) instead of the control unit A mode in which a smartphone or a portable game machine is used, and (v) a mode in which the connection unit (code) is eliminated by adopting a configuration in which the control unit and the image display unit can perform wireless communication and wireless power feeding. (Vi) A mode in which the touch pad is eliminated from the control unit, and the touch pad is provided in the image display unit.

  In the above embodiment, for convenience of explanation, the control unit includes a transmission unit, and the image display unit includes a reception unit. However, each of the transmission unit and the reception unit of the above-described embodiment has a function capable of bidirectional communication, and can function as a transmission / reception unit. Further, for example, the control unit illustrated in FIG. 5 is connected to the image display unit via a wired signal transmission path. However, the control unit and the image display unit may be connected by a connection via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth (registered trademark).

  For example, the configurations of the control unit and the image display unit illustrated in FIG. 2 can be arbitrarily changed. Specifically, for example, the control unit includes various input devices (for example, operation sticks, keyboards) in addition to the above-described various input devices (touch pad, cross key, foot switch, gesture detection, line of sight detection, microphone). A mouse, etc.). Moreover, in the said embodiment, although the secondary battery was used as a power supply, as a power supply, not only a secondary battery but a various battery can be used. For example, a primary battery, a fuel cell, a solar cell, a thermal cell, or the like may be used.

  For example, the head mounted display is a binocular transmissive head mounted display, but may be a monocular head mounted display. Further, it may be configured as a non-transmissive head mounted display in which the transmission of the outside scene is blocked when the user wears the head mounted display. In the above embodiment, it is assumed that the image display unit is a head-mounted display that is worn like glasses, but, for example, a head that employs an image display unit of another shape such as an image display unit that is worn like a hat. It is good also as a mount display. The earphone may be an ear-hook type or a headband type, or may be omitted. Furthermore, for example, it may be configured as a head-up display (HUD, Head-Up Display) mounted on a vehicle such as an automobile or an airplane. In addition, it may be configured as a head mounted display built in a body protective device such as a helmet.

  FIG. 14 is an explanatory diagram showing an external configuration of a head mounted display in a modified example. In the example of FIG. 14A, the image display unit 20a includes a right optical image display unit 26a instead of the right optical image display unit 26, and includes a left optical image display unit 28a instead of the left optical image display unit 28. ing. The right optical image display unit 26a and the left optical image display unit 28a are formed smaller than the optical member of the first embodiment, and are disposed obliquely above the right and left eyes of the user when the head mounted display is mounted. Yes. In the example of FIG. 14B, the image display unit 20 b includes a right optical image display unit 26 b instead of the right optical image display unit 26, and includes a left optical image display unit 28 b instead of the left optical image display unit 28. ing. The right optical image display unit 26b and the left optical image display unit 28b are formed smaller than the optical member of the first embodiment, and are disposed obliquely below the right and left eyes of the user when the head mounted display is mounted. Yes. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit completely covers the user's eye. It can also be realized as an unmounted head mounted display. Even in the case of the configuration as shown in FIG. 14, the arrangement of the first image and the second image in the list image is made so as to be suitable for a head-mounted display while improving the visibility for the user. It can be adjusted appropriately. In this case, the arrangement is not limited to the example illustrated in the above embodiment.

  For example, in the above embodiment, the display driving unit is configured using a backlight, a backlight control unit, an LCD, an LCD control unit, and a projection optical system. However, the above aspect is merely an example. The display drive unit may include a component for realizing another method together with these components or instead of these components. For example, the display driving unit may include an organic EL (Organic Electro-Luminescence) display, an organic EL control unit, and a projection optical system. Further, for example, the display drive unit can use a digital micromirror device or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head-mounted display device.

  For example, each function unit such as a generation unit, a notification unit, an image processing unit, a display control unit, and an audio processing unit is realized by a CPU developing and executing a computer program stored in a ROM or a hard disk on the RAM. It was described as a thing. However, these functional units may be configured using an ASIC (Application Specific Integrated Circuit) designed to realize the function.

・ Modification 3:
In the above embodiment, an example of the arrangement process has been described. However, the arrangement processing procedure is merely an example, and various modifications are possible. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

  For example, the generation unit may generate the list image without using the area information. Specifically, in step S116, the generation unit may generate the list image by arranging the first image and the second image at predetermined coordinate positions instead of the area information. As another example, in step S116, the generation unit arranges the first image and the second image at coordinate positions dynamically calculated from the sizes of the acquired first image and second image. A list image may be generated by. In this way, a list image can be generated without requiring area information.

  For example, the enlargement / reduction processing of the first and second images in step S114 may be omitted. Further, for example, either the smartphone authentication in step S104 or the connection establishment in S106 may be omitted, and the execution order may be changed.

  For example, the list image described in the above embodiment is assumed to be an image expressed two-dimensionally. However, the image processing unit may represent the list image three-dimensionally by using different data for the right eye image data and the left eye image data.

  For example, the generation unit changes the pointer images drawn to be superimposed on the first and second images of the list image into different images. However, the generation unit causes the OS to make the pointer image drawn superimposed on the first area of the list image and the pointer image drawn drawn superimposed on the second area different from each other. You may instruct.

  For example, the pointer image drawn superimposed on the first image of the list image and the pointer image drawn superimposed on the second image have the same shape and have changed colors and decorations. An image may be used.

  For example, instead of making the pointer image drawn superimposed on the first image of the list image and the pointer image drawn superimposed on the second image different from each other, another method is used. Thus, the image operated by the user can be viewed. Specifically, the transmittance of one image operated by the user may be reduced and the transmittance of the other image may be increased. In addition, color conversion that enhances the image compared to the outside scene is performed on one image being operated by the user, and color conversion is performed to assimilate the outside scene and the image on the other image. Processing may be performed. Alternatively, one image being operated by the user may be decorated, and the other image may be left undecorated. In this way, the image operated by the user can be visually recognized using the transmittance, color, and presence / absence of decoration of the first image and the second image.

  For example, in the second embodiment, the generation unit may generate the second image without using a frame. Specifically, the generation unit may generate the second image by arranging an icon image at a predetermined coordinate position. As another example, the generation unit may generate the second image by arranging the icon image at a coordinate position dynamically calculated from the size of the acquired icon image. In this way, the second image can be generated without requiring frame information.

  For example, in the second embodiment, the generation unit may dynamically generate the second image so as to avoid the user's line-of-sight direction. Specifically, in the head mounted display, a configuration for detecting the user's line-of-sight direction (also referred to as “line-of-sight direction detection unit”) such as a camera or an infrared sensor that captures the user's eyes is added. The generation unit can preferentially select an image list far from the detected line-of-sight direction from among a plurality of image lists provided in the frame, and can arrange icon images. In this way, in the second image, it is possible to arrange a dynamic icon image avoiding the user's line-of-sight direction.

-Modification 4:
In the above embodiment, an example of the notification process has been described. However, the procedure of the notification process is merely an example, and various modifications can be made. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

-Modification 5:
In the said embodiment (FIG. 3), the example of the area | region information memorize | stored in the area | region information storage part was shown. However, the details of the area information are merely examples, and various changes can be made. For example, a component can be added, deleted, or changed.

  For example, the area information storage unit may store a plurality of area information. In addition, frames used when generating a list image according to arbitrary conditions such as the user's preference (setting) of the head mounted display, the movement of the user's line of sight, the movement of the user's head, the ambient brightness, etc. May be selected.

Modification 6:
In the second embodiment (FIG. 11), examples of frames stored in the frame information are shown. However, the details of the frame are merely examples, and various changes can be made. For example, a component can be added, deleted, or changed.

  For example, a plurality of frames may be stored in the frame information. In addition, it is used when generating the second image according to arbitrary conditions such as the user's preference (setting) of the head mounted display, the movement of the user's line of sight, the movement of the user's head, the ambient brightness, etc. A frame may be selected.

  For example, a frame has two image lists LT1 and LT2. However, the frame may have one image list or three or more image lists. Further, the shape, size, and number of image frames in the image list can be arbitrarily determined. Further, the size (aspect ratio) of the region of the frame may not be the same size (aspect ratio) as the second region of the region information.

Modification 7:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Control part 20 ... Image display part 22 ... Right display drive part 24 ... Left display drive part 26 ... Right optical image display part 28 ... Left optical image display part 32 ... Right earphone 34 ... Left earphone 40 ... Connection part 51 ... Transmission Unit 52 ... Transmitter 53 ... Receiver 100 ... Head mounted display (head-mounted display device)
110: Input information acquisition unit (operation acquisition unit)
DESCRIPTION OF SYMBOLS 120 ... Memory | storage part 122 ... Area | region information storage part 124 ... Frame information 130 ... Power supply 132 ... Wireless communication part (acquisition part)
140 ... CPU
142 ... generating unit (acquiring unit, generating unit)
144 ... Notification section (notification section)
DESCRIPTION OF SYMBOLS 160 ... Image processing part 170 ... Sound processing part 180 ... Interface 190 ... Display control part 201 ... Right backlight control part 202 ... Left backlight control part 221 ... Right backlight 222 ... Left backlight 251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate 300 ... Smartphone (external device)
1000 ... Image display system Data ... Image data B1 to B10 ... Image frame BC ... Thick frame SC ... Outside view RE ... Right eye LE ... Left eye AI ... Region information VI ... Virtual image VR ... Field of view FM1 ... Frame IM1 ... First image IM2 ... second image PO1 ... pointer image PO2 ... pointer image AR1 ... first area AR2 ... second area LT1 ... image list LT2 ... image list INT ... internet

Claims (11)

A head-mounted display device that allows a user to visually recognize a virtual image and an outside scene,
An acquisition unit that acquires a first image that is an image displayed on a display screen of the external device from an external device connected to the head-mounted display device ;
Said first image, said head mounted display device can be closed, generator for generating a list image and a second image that is an image relating to the operation of the head-mounted display device,
An image display unit for forming the virtual image representing the generated list image;
Bei to give a,
The generating unit arranges the first image with respect to the first area according to the definition of area information including a first area and a second area different from the first area, and the second area A head-mounted display device that generates the list image in which the second image is arranged with respect to the region . The head-mounted display device according to claim 1 ,
The generation unit is a head-mounted display device that uses the image displayed on the head-mounted display device as the second image when generating the list image . The head-mounted display device according to claim 1 ,
The generation unit generates an icon image to be displayed on a standby screen of the head-mounted display device when the list image is generated. A head-mounted display device that generates the second image by arranging in accordance with frame information indicating an arrangement position of the image. The head-mounted display device according to claim 3 ,
When the generation unit generates the second image, the generation unit further changes the shape, the transmittance, the color, and the size of the icon image included in the head-mounted display device. A head-mounted display device that performs at least one of the change of the height and the addition of the decoration to obtain an icon image arranged according to the frame information . The head-mounted display device according to any one of claims 1 to 4 ,
The generation unit further changes the size of the first image when generating the list image, sets the changed image as the first image, and sets the size of the second image. A head-mounted display device that implements at least one of changing the height and setting the changed image as a second image . The head-mounted display device according to any one of claims 1 to 5 ,
The generation unit further generates a change in shape, a change in transmittance, a change in color, a change in size, and a decoration on the first image when generating the list image. And executing the image after the execution as the first image, changing the shape, changing the transmittance, and changing the color with respect to the second image. And a head-mounted display device that performs at least one of a change in size and an application of decoration . The head-mounted display device according to any one of claims 1 to 6 , further comprising:
An operation acquisition unit for acquiring an operation of the user with respect to the list image;
A head-mounted display device including a notification unit that notifies the external device of the operation when the acquired operation is an operation on the first image. The head-mounted display device according to any one of claims 1 to 7 ,
The image display unit further forms a virtual image in which a pointer image is superimposed on the list image,
The head-mounted display device, wherein the generation unit causes the pointer image to be superimposed on the first image and the pointer image to be superimposed on the second image to be different images. A method for controlling a head-mounted display device,
(A) obtaining a first image that is an image displayed on a display screen of the external device from an external device connected to the head-mounted display device ;
(B) said first image, said head mounted display device is Yes, a step of generating list image and a second image that is an image relating to the operation of the head-mounted display device ,
(C) forming a virtual image representing the generated list image;
Bei to give a,
In the step (b), the first image is arranged with respect to the first region in accordance with the definition of region information including a first region and a second region different from the first region, A method of generating the list image in which the second image is arranged with respect to a second region . An image display system,
A head-mounted display device that allows a user to visually recognize a virtual image and an outside scene;
An external device connected to the head-mounted display device;
With
The external device is
A transmission unit configured to acquire a first image that is an image displayed on a display screen of the external device, and to transmit the acquired first image to the head-mounted display device;
The head-mounted display device is
Said first image, said head mounted display device can be closed, generator for generating a list image and a second image that is an image relating to the operation of the head-mounted display device,
An image display unit for forming the virtual image representing the generated list image;
Bei to give a,
The generating unit arranges the first image with respect to the first area according to the definition of area information including a first area and a second area different from the first area, and the second area An image display system that generates the list image in which the second image is arranged with respect to the area of the image . An information processing apparatus that is connected to a head-mounted display device and generates an image to be displayed on the head-mounted display device,
A first image that is an image acquired by an external device connected to the head-mounted display device and displayed on a display screen of the external device, and the head-mounted display device an acquisition unit but chromatic, and acquires a second image that is an image relating to the operation of the head-mounted display device,
A list image generation unit that generates a list image including the acquired first image and the acquired second image;
A list image transmission unit for transmitting the generated list image to the head-mounted display device;
Bei to give a,
The list image generation unit is configured to arrange the first image with respect to the first region in accordance with a rule of region information including a first region and a second region different from the first region, An information processing apparatus that generates the list image in which the second image is arranged with respect to a second region .

Images