JP2006267604A - Composite information display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a composite information display device which avoids a peep in browsing an electronic document. <P>SOLUTION: The composite information display device includes a light-transmissive head mount display (HMD) having an image display area having a wide viewing angle, a main display having a resolution and a screen size adapted to OA activities, and a means for displaying image information segmented for HMD, on an area resulting from excluding image information displayed on the main display from a display area of the HMD. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention generally relates to an information processing system device, and more particularly, a light-transmissive head-mounted display (also referred to as a head-mounted display device, HMD or HMD device), and a conventional display that performs display in conjunction with the display device. The present invention relates to a device and a composite information display device using these display devices.

  As is well known, office automation (OA) work using computers in the office has recently increased.

  As a general form of OA work, a computer, a keyboard, and a CRT monitor are used as one system, and a user performs OA work using a system placed on a desk.

  In executing the OA operation, data is input from a keyboard or data processing is instructed while displaying data on a CRT having a relatively high resolution screen.

  When the desired data is not displayed on the CRT screen, the data range displayed by the arrow keys of the keyboard or the mouse device can be moved to continue the OA operation.

  On the other hand, as one of display devices, there is a head mounted display device (HMD) that can display an image to a user while wearing on the head like goggles or glasses.

  This head-mounted display device can provide a large screen that covers a relatively wide field of view while being small and light.

Further, when the head-mounted display device is of a light transmission type (optical see-through type), the image from the computer is superimposed on the image taken from the computer by superimposing the image from the computer. Many devices have been proposed that realize mixed reality (MR) for the purpose of seamless connection between real space and virtual space by combining images and displaying them on HMD (video see-through method). Yes.
JP 2002-271694 A JP 2003-141571 A JP-A-7-59032

  In a general OA work system, items to be displayed on the screen and the amount of information are increasing with the recent complication of OA work, so that the resolution of display devices and the increase in screen size are being promoted. In general, a desktop work space in an office is limited, and it is not easy to introduce a large display device.

  In recent years, the necessity of strengthening information security has been screamed within companies, but during OA work, since confidential information is displayed on the display device, information leakage due to peeping etc. I was worried.

  On the other hand, if a head mounted display device is used, the risk of peeping is reduced, and a relatively large screen can be displayed. However, it is said that a person usually has a field of view with a viewing angle of ± 40 ° about the center of the eyeball. Accordingly, it is necessary to introduce a display device having a screen that covers the maximum field of view of a person. However, since the number of scanning lines for displaying an image on a display is generally limited (for example, 480 scanning lines in a normal VGA size). The resolution of the image is reduced when a large screen display is used. This reduction in resolution causes a reduction in the amount of information that can be displayed, and reduces the efficiency of OA work by presenting a blurred image to a person.

  The solution to this problem is to develop a display device with a large number of scanning lines that can display on the screen a high-resolution image that does not impair the human eyesight. It is technically difficult to develop.

  The present invention has been made to provide an information display device having a wide field of view using such a current head-mounted display having a low resolution.

  Humans can see the gaze point (the center axis of the eyeball, that is, the position of the object looking at the line of sight) and only in the narrow surroundings (usually an area of ± 5 ° around the line of sight), and other areas Then, the phenomenon that the image looks blurred is known. The present invention relates to an apparatus capable of presenting an operator with an image equivalent to the use of a wide field of view and high resolution display by simple image information processing operation, paying attention to such characteristics of a living body.

  In order to display such a person's gazing point and its narrow surroundings with high resolution, Patent Document 3 detects the person's line of sight and operates the optical system in response to the movement of the eyeball so that the line of sight is always displayed. At the same time as looking at the central point of the monitor of the device, the central area of the monitor displays an image of the area that the eyeball should be gazing at with high resolution on the image display panel when the monitor is placed directly in front of the eyeball. There has been proposed an image display device that presents and presents an image around the region with a transparent liquid crystal panel at a low resolution.

  However, in the above proposal, it is necessary to physically move the monitor or mirror in order to follow the movement of the eyeball so that the display on the monitor for the narrow viewing angle coincides with the center of the line of sight. It is inevitable that a small time lag occurs between them. It is well known that the continuous occurrence of such a small shift acts on the autonomic nerve and causes unpleasant symptoms similar to seasickness.

  The present invention has been made in view of the circumstances as described above, and is capable of presenting a wide viewing angle image and presenting a high-resolution image suitable for OA work, and mixed reality. It is an object of the present invention to provide a composite information display device that can realize a feeling well and has a simple device configuration.

  In order to achieve this object, the composite information display device according to the present invention is suitable for a light transmission type head-mounted display having an image display area with a wide viewing angle worn on the operator's head and OA work. A main display device having a resolution and a screen size, and means for displaying image information divided for HMD in a region excluding the image information displayed on the main display device from the display region of the HMD device. It is characterized by that.

  In general, a worker who is performing OA work often performs a work while maintaining a constant distance and posture with respect to the main display device within a fixed time. Therefore, a fixed region including the center of the visual field in the display area of the HMD device. By designating the area in advance as the display area of the main display device, the display area of the main display device can be removed from the display area of the HMD device.

  In this case, since the operator's gazing point is placed on the main display device, the image display area of the wide viewing angle of the HMD device is located in the peripheral visual field of the worker, and thus has a relatively low resolution. There is no problem even if it is a display device, and as a result, an information processing device with a wide viewing angle and high resolution can be obtained.

  According to claim 2, which is a preferred aspect of the present invention, the composite information display device includes a video camera device having a shooting angle of view equivalent to a viewing angle of a display area of the head mounted display, Means for recognizing the main display device seen through the display area of the light-transmissive head-mounted display device and determining an apparent position and size of the display area of the main display device with respect to the display area of the HMD device; It is characterized by having.

  Since the video camera device is installed together with the HMD device so that the display area of the HMD device has a field angle of view and a field angle center equivalent to the field angle given to the worker, Even when the posture or the positional relationship with the main display device is changed, it is possible to perform image display excluding a portion corresponding to the display region of the main display device that is always seen through the HMD display region. Therefore, it is possible to obtain a composite information display device having a sense of unity.

  In general, in a system that presents mixed reality (hereinafter referred to as MR (Mixed Reality) or AR (Augmented Reality)), how accurately and stably extracting markers placed in the target space is extracted from the real space. In order to perform alignment with the virtual space, that is, to obtain the observer's viewpoint position and orientation, it is extremely important to use a marker because it is substantially inevitable.

  In the present invention, an image that is captured by a video camera and used to determine a relative positional relationship with the HMD display area, that is, a marker, is a main display that has been used by an operator for OA work. Since it is the case itself of the apparatus, there is an advantage that it does not give a sense of incongruity to the operator.

  According to a third aspect of the present invention, the composite information display device is disposed in the HMD having a field angle of view equivalent to a viewing angle of a display area of the head mounted display. A synchronization signal that synchronizes the display of the scanning line of the main display device and the timing of image capturing of the video camera is generated. Means for comparing the image of the first luminescent spot of the first scanning line of the main display device and the last luminescent spot of the final scanning line taken by the video camera, so that the display area of the main display device A means for discriminating an apparent position and size with respect to a display area of the HMD device is provided.

  Using the synchronization signal, the first luminescent spot of the first scanning line and the last luminescent spot of the last scanning line are photographed by the video camera within one refresh cycle of the main display device. An accurate position of the display area can be calculated. Therefore, the display area of the HMD display device can be controlled more precisely.

  According to the present invention, a wide viewing angle image can be presented, a high-resolution image suitable for OA work can be presented, and mixed reality can be well realized. Thus, it is possible to provide a composite information display device having a simple device configuration.

  Hereinafter, a composite information display device according to a preferred embodiment to which the present invention is applied will be described with reference to the accompanying drawings.

  FIG. 1 shows the relationship between the field of view 104 of an operator, that is, an HMD wearer, the main display device 101, the display area 102 of the main display device, and the displayable area 103 of the HMD in the composite information display device of the present invention. A portion excluding the display area 102 of the main display device in the shaded area from the displayable area 103 of the light transmission type HMD becomes a part where an image is actually displayed on the HMD display apparatus, and the HMD wearer can display the HMD display screen. The display area 102 of the main display device 101 can be visually recognized through the hatched portion 102 through the display portion of the light transmission type HMD without being obstructed.

  In the system of claim 1, the size of the display area 102 of the main display device and the positional relationship of the HMD with respect to the displayable area 103 can be arbitrarily set by the HMD wearer.

  FIG. 2 shows a relative positional relationship between the eyeball 206 of the HMD wearer, the display surface 201 of the light transmission type HMD, and the display surface 202 of the main display device. Since the display surface 201 of the HMD is installed at a position close to the eyeball 206 of the HMD wearer, even if it is relatively small, it can cover the maximum field of view 204 of the HMD wearer and is displayed on the HMD. The virtual image 205 to be formed is constituted.

  In FIG. 2, the display 201 of the HMD is described as if it is directly installed at a position close to the eyeball 206 of the HMD wearer. Of course, this includes optical mechanisms such as a half mirror, a prism, and a lens. It is self-evident that it does not hinder the contrivance to use and install the display surface in a place away from the eyeball of the HMD wearer.

  FIG. 3 shows a system configuration diagram in a case where the display range position of the main display device is determined by comparing the main display device housing photographed in advance with the display device housing photographed by the video camera.

  The information processing device 302 is a computer device used by an OA worker for OA work, and the input device 301 is generally used for inputting data such as a keyboard and a mouse and for instructing data processing. The main display device 309 is also a display device having a relatively high resolution screen that is generally used for OA work.

  When the desired data is not displayed on the screen of the main display device 309, the input device 301 can move the displayed data range and continue the OA operation.

  The information processing apparatus 302 includes an image information generation unit 303 that generates a window system, a virtual screen, and the like. The image information generation unit 303 generates image information for an area corresponding to the HMD displayable area 103 and transmits the image information to the image generation apparatus 313.

  On the other hand, the video camera 312 mounted on the HMD device 310 so that the display area of the HMD device 310 has the same field angle of view and center of field of view as the field angle given to the operator is the main display device 309. The video image input unit 304 of the image generation device 313 captures this, and the main display device position determination unit 305 determines the relative position of the main display device 309 with respect to the HMD image display unit 311.

  The image information dividing unit 306 divides the image information generated by the image information generating unit 303 according to the relative position information determined by the main display device position determining unit 305, and the main display image generating unit 307 and the HMD image generating unit 308 are divided. In FIG. 5, the main display device 309 and the light transmission type HMD image display unit 311 generate image signals having appropriate resolutions, so that the main display device 309 and the light transmission type HMD image display unit 311 can display composite information with a sense of unity. Realize.

  FIG. 4 is a flowchart showing an algorithm for realizing these processes.

  The following processing may be performed inside the image generation apparatus 313 or may be realized as a program stored in the information processing apparatus 302.

  First, in step S001, the HMD wearer wears the aforementioned HMD device. In step S002, the main display device is turned off in order to easily determine the casing of the main display device as a marker. This may be simply changed to temporarily blacking out the display screen of the main display device.

  Next, in step S003, a guide image representing the outer frame of the main display device is displayed on the display screen of the HMD. The size and aspect ratio of this guide image may be switched by the operation of the HMD wearer. The HMD wearer changes the posture so that the outer frame of the main display device 101 seen through the HMD displayable area 103 and the guide image overlap each other, and when the image overlaps successfully, the HMD device 310 moves the housing of the main display device in step S004. Shoot with the video camera 312 attached to the camera.

  In step S005, an image of a portion excluding the display area from the housing of the main display device, that is, a frame image is cut out from the image taken in step S004. Since the main display device is turned off in step S002, the display surface and the frame portion can be easily distinguished from each other. Since the position of the outer frame is designated in step S003, the frame image can be easily cut out. Can be done.

  At the end of the cut-out, the main display device is turned on in step S006 to enable image display.

  The processes from step S002 to step S006 need only be performed once when the HMD wearer uses the composite information display device according to the present invention for the first time or when the main display device is replaced with another one. The frame image cut out in step S005 may be recorded.

  In step S007, the video camera 312 is used to photograph the field of view of the HMD wearer. In step S008, which is the main display device position determination unit 305, the frame image cut out in step S005 is compared with the image of the field of view of the HMD wearer photographed in S007 to identify the main display device. In step S009, it is determined in which position the display area 102 of the main display device is located in the displayable area 103 of the HMD.

  In step S 010, the display element of the virtual screen generated by the image information generation unit 303 is acquired from the information processing apparatus 202. In step S011 of the image information separation unit 306, the coordinates of each display element are confirmed, and it is determined whether the display element is in the display range of the main display screen or the display range of the HMD. If this is within the range of the main display screen, the display element is rendered in accordance with the resolution of the main display device in step S012 of the main display image generation unit 307, and an image is output to the main display device 309 in step S013. . On the other hand, if the display element is within the range of the HMD display screen, the display element is rendered in accordance with the resolution of the light transmission type HMD image display unit 311 in step S014 of the HMD image generation unit 308, and the process proceeds to step S015. The image is output to the HMD image display unit 311.

  In step S016, it is determined whether or not all of the display elements of the virtual screen have been processed. If there are unprocessed elements, the process returns to step S010 to repeat the process. When all the display elements have been processed, the processing from step S007 is repeated.

  By repeating the above processing, an image having an appropriate resolution is generated and displayed for each of the main display area and the HMD display area, and the HMD wearer can obtain a good field of composite information display.

  FIG. 5 shows an example of composite information display that an OA worker who is performing OA work using the composite information display apparatus of the present invention will obtain.

  The main display device 501 is seen through the displayable area 503 of the HMD. The main display device 501 actually displays only the portion surrounded by the display area boundary 502 of the main display device, and all other displays are displayed in the displayable area 503 of the HMD. The field of view 504 of the HMD wearer is indicated by a broken line in the figure, but this is not displayed on the HMD.

  In the displayable area 503 of the HMD, information that should be kept from being noticed by others, such as the information display area 507, a personal memorandum displayed on the virtual sticky note 508, etc., and the virtual calendar 505 are shown. Elements that need to protect information security and privacy, such as schedule information, a virtual keyboard 506, a virtual clock 512, and the like are displayed.

  Information display areas 509 and 510 are displayed on the display area boundary 502 of the main display device.

  On the other hand, the information display area 511 indicates that the information display area 511 is displayed across the display area of the main display device and the HMD.

  All these display elements can be controlled in which display area they should be displayed by changing the coordinate position occupied on the virtual screen 513 in accordance with an instruction input from the input device 302.

  Since the display elements displayed within the display area boundary 502 of the main display device are displayed at a relatively high resolution, they are suitable for OA work. On the other hand, the display elements displayed outside the display area boundary 502 of the main display device are displayed with a relatively low resolution.

  All the display elements 505 to 512 occupy predetermined display coordinate positions in the virtual screen 513 generated by the image information generation unit 303. The coordinate position occupied by the display area boundary 502 of the main display device on the virtual screen 513 is constant and does not change even if the posture of the worker and the relative position between the worker and the main display device 501 change. The relative position of the displayable area 502 of the main display device in the displayable area 504 changes. For this reason, since the position of the subjective main display device 501 is always constant and the display content is stable for the worker, no unpleasant symptom is caused and information security and privacy are protected. An OA working environment is obtained.

  FIG. 6 is an enlarged view of the information display area 511 displayed across the main display device and the display area of the HMD.

  In the image information dividing unit 306 and step S011, it is determined whether each display element is inside or outside the display area boundary 602 of the main display device. At this time, the display element is as in the information display area 603. When it is determined that both the display area boundary 602 of the main display device and the display area of the HMD are applied, the display element is displayed in the main display device display area 604 of the information display area 603, and the information display area 603 is divided into a portion 605 displayed in the HMD display area, and each of them is used as a separate display element in step S012 of the main display image generation unit 307 and in step S014 of the HMD image generation unit 308, respectively. Therefore, the display element display is not lost.

  FIG. 7 shows a system configuration diagram when the video camera is photographed in synchronization with the screen display scanning line of the main display device and the display range position of the main display device is determined.

  The information processing device 702 is a computer device used by an OA worker for OA work, and the input device 701 is generally used for inputting data or instructing data processing such as a keyboard and a mouse. The main display device 710 is also a display device having a relatively high resolution screen that is generally used for OA work.

  Further, when desired data is not displayed on the screen of the main display device 701, the displayed data range can be moved by the input device 701, and the OA operation can be continued.

  The information processing apparatus 702 includes an image information generation unit 703 that generates a window system, a virtual screen, and the like. The image information generation unit 703 generates image information for an area corresponding to the displayable area 103 of the HMD, and transmits the image information to the image generation apparatus 714.

  On the other hand, the image generation device 714 includes a synchronization signal generation unit 707, and the imaging angle of view equivalent to the visual field center and the visual field center that the display area of the main display image generation unit 708 and the HMD device 711 gives the synchronization signal to the operator. A synchronization signal is given to the video camera 713 mounted on the HMD device 711 and the main display device position determination unit 705 so as to have a photographing field angle center. The video camera 713 captures the display surface of the main display device 710 in accordance with the timing indicated by the synchronization signal. The video image input unit 704 of the image generation device 714 takes this in, and the main display device position determination unit 705 takes into account the timing of the synchronization signal obtained from the synchronization signal generation unit 707, and the HMD image display unit 311 of the main display device 710. The relative position with respect to is determined.

  The image information dividing unit 706 divides the image information generated by the image information generating unit 703 according to the relative position information determined by the main display device position determining unit 705, and the main display image generating unit 708 and the HMD image generating unit 709. , By generating an image signal having a resolution suitable for the main display device 7109 and the light transmission type HMD image display unit 712, a composite information display having a sense of unity in the main display device 710 and the light transmission type HMD image display unit 712 can be obtained. Realize.

  FIG. 8 shows a display method of the main display device which is a scanning line method. The scanning line type display device used in this embodiment is a CRT display device which is a known scanning line type display device.

  The main display device 801 shows the main display device 710 in FIG. 7 in detail. In the main display device 801, the display image is displayed as an aggregate of small bright spots. Each bright spot is given brightness only for a very short time, but an afterimage remains in the human eye, so the aggregate of bright spots is recognized as a single image. Each luminescent spot is displayed while moving to the right from the position indicated by the upper left corner of the display surface, the starting point 802 of the first scanning line, and the brightness is reduced when the display is completed to the right end of the display surface. The display continues while returning to the left end and moving to the right again sequentially from the point slightly lowered. This operation is called horizontal regression 805. In this manner, the display of the nth scanning line 804 is finished, the horizontal regression 805 is performed, and the operation of displaying the (n + 1) th scanning line is repeated, and the display is performed up to the lower right corner of the display surface and the end point 803 of the bright spot of the last scanning line. Now, the display of one screen is finished, the brightness is reduced, and the display screen moves again to the upper left corner of the display surface for the next bright spot display. This operation is called vertical regression 807.

  In this embodiment, the synchronization signal is transmitted from the synchronization signal generation unit 707 to the main display image generation unit 708, or the display device 710, and the start point 802 of the bright spot of the first scanning line is displayed at the same time as the video camera 713. Since the sync signal is also sent, the video camera 713 can photograph the position of the start point 802 of the bright spot of the first scanning line. Similarly, the video camera 713 captures an image in accordance with the timing at which the bright spot end point 803 of the final scanning line is displayed by the synchronization signal from the synchronization signal generation unit 707. The captured image is sent to the video image input unit 704 of the image generation device 714.

  Since the main display device position determination unit 705 has received the synchronization signal from the synchronization signal generation unit 707 first, the image on the display surface of the main display device 710 from the video image input unit 704 is the brightness of the first scanning line. It is possible to know whether the start point 802 of the point is captured or the end point 803 of the bright spot of the final scanning line is captured. Since it can be assumed that the display surface of the main display device 710 is rectangular, the main display device position determination unit 705 easily determines the relative position of the main display device 710 with respect to the HMD image display unit 712 from the information of these two points. Can do.

  When the configuration of the present embodiment is used, in FIG. 4, step S 007 is performed by setting the bright spot start point 802 of the first scan line and the bright spot end point 803 of the final scan line according to the synchronization signal from the synchronization signal generation unit 707. If it is read as shooting, the preparatory work from step S002 to step S006 can be omitted, and at the same time position determination with higher accuracy can be performed.

The figure which shows the relationship between the display area of a main display apparatus, and the display area of HMD The figure which shows the positional relationship of an HMD wearer's eyeball and each display surface System configuration diagram for determining the display range position of the main display device by comparing the main display device case photographed in advance with the display device case photographed by the video camera Flowchart showing algorithm for implementing the present invention The figure which shows an example of the composite information display during OA work Enlarged view of the information display area 511 displayed across the main display device and the display area of the HMD. System configuration diagram when the video camera is photographed in synchronization with the screen display scanning line of the main display device and the display range position of the main display device is determined. The figure which shows the display method of the main display apparatus which is a scanning line system

Explanation of symbols

101 Main display device 102 Display area (shaded area) of main display device
103 Displayable area 104 of light transmission HMD 104 Field of view 201 of HMD wearer Display surface 202 of light transmission HMD Display surface 202 of main display device Center of line of sight of HMD wearer 204 Maximum view 205 of HMD wearer Displayed on HMD Virtual image 206 Eyeball 301 of HMD wearer 301 Input device 302 Information processing device 303 Image information generation unit 304 Video image input unit 305 Main display device position determination unit 306 Image information division unit 307 Main display image generation unit 308 HMD image generation unit 309 Main display device 310 HMD device 311 Light transmission type HMD image display unit 312 Video camera 313 Image generation device 501 Main display device 502 Main display device display area boundary 503 HMD displayable area 504 HMD wearer's field of view 505 HMD display area Displayed virtual calendar 506 HMD display Virtual keyboard 507 displayed in the area Information display area 508 displayed in the HMD display area Virtual sticky note 509 displayed in the HMD display area Information display area 510 displayed in the main display apparatus display area 510 Inside the main display apparatus display area Information display area 511 displayed on the main display device and information display area 512 displayed across the display area of the HMD Virtual clock 513 displayed on the HMD display area Virtual screen 601 Main display device 602 Display area boundary of the main display device 603 Information display area 604 displayed across the display area of the main display device and the HMD 604 Part of the information display area 603 displayed in the main display apparatus display area 605 Part of the information display area 603 displayed in the HMD display area 701 Input device 702 Information processing device 703 Image information generation unit 704 Video image input unit 7 5 Main display device position determination unit 706 Image information division unit 707 Synchronization signal generation unit 708 Main display image generation unit 709 HMD image generation unit 710 Main display device 711 HMD device 712 HMD image display unit 713 Video camera 714 Image generation device 801 Main display Device 802 Bright point start point 803 of the first main scan line Bright point end point 804 of the last main scan line nth main scan line 805 Horizontal regression 806 n + 1 main scan line 807 Vertical regression

Claims (3)

  Light-transmitting head-mounted display, main display device, and image information segmented for head-mounted display are displayed in an area excluding image information displayed on the main display device from the display area of the head-mounted display. And a composite information display device.   Recognizing the video camera device having a shooting angle of view equivalent to the viewing angle of the display area of the head-mounted display and the main display device seen through the display area of the light-transmissive head-mounted display, 2. The composite information display apparatus according to claim 1, further comprising means for discriminating an apparent position and size of the display area with respect to the display area of the head mounted display.   Having a video camera device disposed on the head mounted display having a field angle of view equivalent to a viewing angle of a display area of the head mounted display, the main display device being a scanning line system, Means for generating a synchronization signal for synchronizing the display of the scanning line and the timing of image capturing of the video camera; the first bright spot and the final scanning line of the first scanning line of the main display device captured by the video camera; And a means for discriminating an apparent position and size of the display area of the main display device with respect to the display area of the head-mounted display by comparing the images of the last bright spots of the main display device. 1. The composite information display device according to 1.

Images