JP2017107359A - Image display device, program, and method that displays object on binocular spectacle display of optical see-through type - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display device that can increase the visibility of an object for users when displaying an object on a spectacle optical see-through type binocular display, in a case where right and left calibration parameters contain estimation errors.SOLUTION: The present invention includes: fixation point position acquisition means for acquiring a depth position from a display to a fixation point of a user; overlapping region calculation means for calculating an overlapping region where a right-eye screen and a left-eye screen overlap with each other, recognized at a depth position through the display from the right eye and left eye of the user; drawing region selecting means for selecting one of the right-eye screen and the left-eye screen for a virtual object to be displayed in the overlapping region; and object drawing means for drawing the virtual object to the selected one of the right-eye screen and the left-eye screen.SELECTED DRAWING: Figure 2

Description

  The present invention uses a glasses-like optical see-through type binocular display to add character and image objects (CG (Computer Graphics)) to an augmented reality (AR (augmented reality)) in a real space viewed from the user. As a superimposing display technology. In particular, the present invention relates to a technique for improving the visibility of an object viewed from a user.

  As a use of AR, for example, when a user holds the camera of a terminal he / she owns over a product, the product is specified, and information such as a price associated with the product is displayed at an appropriate position on the display. be able to. In recent years, with the improvement of processing performance of terminals such as mobile phones and smartphones, an environment capable of executing AR has been provided.

  For example, an HMD (Head Mounted Display) is a spectacle-like device provided with a small display worn on the user's head. The camera mounted on the HMD does not require the user to consciously hold the camera by himself / herself in order to continuously capture the image of the real space that appears in front of the user's line of sight. Therefore, the AR image can be displayed in a superimposed manner on the real space ahead of the user's line of sight. There is a work support system as an application in which HMD and AR are linked. For example, the worker can check the object of the work instruction reflected on the display in front of the eyes while performing the work of the hand.

HMD can be mainly classified into the following two types.
Video see-through type: Superimpose an object on the camera preview image Optical see-through type: Superimpose an object on the real space itself

  The video see-through HMD can use the same alignment method as the existing smartphone application because the preview image of the camera is the target of alignment. However, the viewing angle of the wearer and the visibility of the real environment are limited to the angle of view and resolution of the camera. Therefore, for example, in the case of a work support system, the restriction of the field of view for the worker may reduce the work efficiency.

  On the other hand, in the optical see-through HMD, the real space itself that the wearer visually recognizes through the virtual screen is the target of alignment. The optical see-through type HMD has a drawback that the display viewing angle is narrower than the video see-through type. On the other hand, since the field of view of the wearer is open, only the work instruction information necessary for the real space can be AR-displayed, which is suitable for the work support system.

  According to the optical see-through HMD, it is necessary to correct the display position of the object so that it overlaps the real space in the field of view of the wearer. Such correction is an operation unique to the optical see-through HMD, and is referred to as “HMD calibration” (see, for example, Patent Document 1 and Non-Patent Document 1). According to the technique described in Patent Document 1, a calibration parameter is estimated with high accuracy for a monocular HMD.

  Among optical see-through HMDs, there is a technique that compensates for narrowness of the viewing angle of a single eye by displaying on a virtual screen that can be seen by both eyes (see, for example, Non-Patent Document 2).

  FIG. 1 is an external view showing a user's field of view as seen from an optical see-through HMD.

  Specifically, according to MOVERIO BT-200 manufactured by EPSON (see Non-Patent Document 3, for example), when the user focuses both eyes on a predetermined depth position, the viewing angle of a single eye, for example, the left eye The viewing angle of the left eye with respect to the screen is 23 °. Here, the viewing angle of the monocular is constant regardless of the focus position. On the other hand, the viewing angle of both eyes corresponding to the binocular screen was about 1.5 times the viewing angle of a single eye as a result of visual confirmation with a depth distance l = 50 cm. That is, when the focus is adjusted to the position where the depth distance is 1 = 50 cm, and the right eye is closed and only the left eye is opened, the viewing angle seems to be increased by 1.5 times. Visible to. As the user moves away from the gazing point (focus position), the magnification rate of the viewing angle of both eyes decreases. Then, the viewing angle becomes 1.0 times at a certain depth distance l, and the left eye screen and the right eye screen completely overlap. The depth distance l is referred to as the focal length of the HMD.

The degree to which the viewing angle is expanded using the binocular screen differs depending on the depth position where the wearer focuses.
The closer the user sees, the greater the viewing angle magnification when using a binocular screen.
As the user looks farther, the magnification of the viewing angle when using the binocular screen decreases.

  FIG. 2 is an external view showing the relationship between the two screens and the user's field of view for the optical see-through HMD.

  According to FIG. 2, in the case of MOVERIO BT-200, half mirrors are embedded on the left and right sides of the spectacles portion in the HMD. The light irradiated in the horizontal direction from the inside of the HMD is input to the user's eyes through the reflecting surface of the half mirror. On the other hand, the light in the real environment is directly input to the user's eyes through the non-reflecting surface of the half mirror. As a result, the user visually recognizes a screen (virtual screen) as a virtual image in the real environment. Unlike a physical display, the virtual screen has a feature that the size of the screen looks different depending on the depth position viewed by the user. For example, when viewed from the user, a display screen equivalent to 320 type can be viewed at a position at a depth distance of 20 m. The optical see-through type HMD often takes advantage of this property to enjoy video viewing on a very large screen even in a narrow environment. Hereinafter, “screen” refers to “virtual screen”.

According to FIG.1 and FIG.2, the binocular screen is represented by the depth position of a user's gaze point. The gaze point is a position where both eyes of the user are in focus. This binocular screen is divided into the following three areas.
Left eye area (area only for left eye screen)
Overlapping area (area where the right eye screen and left eye screen overlap)
Right eye area (area only for right eye screen)

  For example, according to the use of the work support system, the depth position is often a short distance (about 50 cm) as the work range of the hand. This is different from the above-described usage method assumed by the HMD manufacturer, and the screen position of the left eye and the right eye is shifted to the left and right, so that the operator can see the left eye region and overlap as shown in FIGS. The region and the right eye region are viewed at the same time. Here, the binocular screen is a combination of these three areas and is a screen visually recognized by the user's eyes.

  For each of the left-eye screen and right-eye screen constituting the binocular screen, by calculating the HMD calibration parameters in advance, the drawing position of the appropriate object that matches the user's eye position is calculated. can do. The calibration parameter takes a constant value regardless of the focus position of the user if the position of the user's eyes is fixed with respect to the HMD to be worn. Therefore, even when the screen positions of both eyes are shifted to the left and right during short distance viewing, if the calibration parameters can be accurately calculated in advance, the object can be superimposed and displayed at an appropriate position.

  FIG. 3 is an explanatory diagram showing a screen shown in the user's field of view when the calibration parameter does not include an estimation error. The balloons “Do not touch this” and “Press this” in the figure are examples of work instruction objects when a work support system is assumed. The wearer of the HMD appears to overlap this actual work target equipment.

  According to FIG. 3, the calibration parameters of the HMD are calculated completely without error, and in the overlapping area of the binocular screen, the objects drawn on each screen are completely overlapped. Visible from the user's eyes without any discomfort.

JP 2014-170374 A

Hirokazu Kato, M. Billinghurst, Koichi Asano, Keihachiro Tachibana, “Augmented Reality System Based on Marker Tracking and Its Calibration”, Transactions of the Virtual Reality Society of Japan, Vol4, No4, pp.607-616, (1999) , [Online], [October 9, 2015 search], Internet <http://intron.kz.tsukuba.ac.jp/tvrsj/4.4/kato/p-99_VRSJ4_4.pdf> Yuya Maki, Tatsuya Kobayashi, Haruhisa Kato, Hiromasa Yanagihara, “Hands-free remote operation support AR system by HMD calibration and on-site learning,” Research Report Audio Visual Complex Information Processing (AVM), 2015-AVM-88 (2), 1-6 (2015-02-20), [online], [October 9, 2015 search], Internet <URL: http://ci.nii.ac.jp/naid/110009877748> EPSON, “MOVERIOBT-200AV / BT-200”, [online], [October 9, 2015 search], Internet <http://www.epson.jp/products/moverio/bt200/>

  Actually, the HMD mounting position may be shifted in the middle. In addition, the calibration parameters vary from user to user depending on the shape of the user's head and whether or not glasses for correcting vision are worn. However, since the calibration work is time-consuming, it may be possible to use calibration parameters calculated by a third party. As a result, calibration parameters often contain errors.

  FIG. 4 is an explanatory diagram showing a screen shown in the user's field of view when the calibration parameter includes an error.

When the left and right calibration parameters (P _1, P _r ) include an error, as shown in FIG. 4, the drawing position of the object (CG) for each screen is relative to the real environment as seen from the user. There is a gap. Here, in the binocular HMD, in addition to the displacement of the drawing position, a problem relating to the visibility of the object peculiar to both eyes arises. Specifically, as shown in FIG. 4, the object does not completely overlap in the overlapping area of the left eye screen and the right eye screen, and the object is visually recognized twice.

  Even when the left and right calibration parameters do not include an estimation error, the objects drawn on the left and right screens overlap each other in the overlapping area, thereby increasing the luminance of only the overlapping area. As a result, the presence of boundary lines at both ends of the overlapping area is conspicuous, and the visibility of the entire object is lowered.

  Accordingly, the present invention provides an image display that can improve the visibility of an object in an overlapping area of each screen, which is visually recognized by a user through the display when the object is displayed on an eyeglass-shaped optical see-through type binocular display. An object is to provide an apparatus, a program, and a method.

According to the present invention, there is an optical see-through display that is arranged for each of the user's right eye and left eye and is visible through the real space, and matches the gazing point in the real object that the user gazes at. An image display device for controlling a virtual object to be displayed on a display,
Gazing point position acquisition means for acquiring a depth position from the camera to the user's gazing point;
Overlapping area calculation means for calculating an overlapping area where the right eye screen and the left eye screen that are visually recognized at a depth position through the display from each of the user's right eye and left eye,
A drawing area selection means for selecting only one of the right eye screen and the left eye screen for the virtual object to be displayed in the overlapping area;
And object drawing means for drawing the virtual object on the selected right eye screen or left eye screen.

According to another embodiment of the image display device of the present invention,
The drawing area selection means
A binocular screen composed of a right eye screen and a left eye screen is divided into a right eye area, an overlapping area, and a left eye area,
Using both end coordinates of the virtual object with respect to the binocular screen and both end coordinates of the overlapping area,
The virtual object to be displayed across the right eye region and the overlapping region is selected as the screen to be drawn in the overlapping region, and the right eye screen is selected.
For the virtual object to be displayed across the left eye region and the overlapping region, it is also preferable to select the left eye screen as a screen to be drawn in the overlapping region.

According to another embodiment of the image display device of the present invention,
The drawing area selection means
For the virtual object to be displayed in the overlap area,
When the accuracy of the calibration parameter P _{r for} the right eye is higher than the accuracy of the calibration parameter P _l for the left eye, the right eye screen is selected as a screen to be drawn in the overlapping region,
Conversely, if the accuracy of the calibration parameter P _r for the right eye is lower than the accuracy of the calibration parameter P _l for the left eye, it is also preferable to select the left eye screen as the screen to be drawn in the overlapping area. .

ｓ：スケール係数 According to another embodiment of the image display device of the present invention,
The back projection-projection calculation of the overlapping area calculation means is performed by performing back projection to a depth position in the three-dimensional space via an arbitrary point from the right eye to the right eye screen (or from the left eye to the left eye screen), and then to the other left eye. back projection is projected on the screen (or the right-eye screen) - by projection calculation, the right eye screen coordinate system (u _{r, v r),} the left eye screen coordinate system (u _{l, v l)} the positional relationship between the It is also preferable to calculate the transformation matrix H _{l, r} shown below.

s: Scale factor

According to another embodiment of the image display device of the present invention,
It is also preferable that the overlapping area calculation means calculates a displacement amount d defined by the following transformation matrix H _{l, r} by back projection-projection calculation when the screens of both eyes are displaced only in the left-right direction.

ｓ：スケール係数
（ｕ_ｒ，ｖ_ｒ）＝（０，０）及びＺ_C＝Ｚ_α（奥行き距離のスカラー値）を代入することで、ｕ_ｌを変位量ｄとして算出するか、
又は、
（ｕ_ｌ，ｖ_ｌ）＝（０，０）及びＺ_C＝Ｚ_α（奥行き距離のスカラー値）を代入することで、ｕ_ｒを変位量−ｄとして算出することも好ましい。 According to another embodiment of the image display device of the present invention,
Backprojection overlapping area calculating means - projection calculations, the right eye screen coordinate system (u _{r, v r)} and left-eye screen coordinate system (u _{l, v l)} and, in the right eye calibration parameters P using a _r, a calibration parameter P _l of the left eye, a camera coordinate system used by the gazing point position acquisition unit (Xc, Yc, Zc) and was defined by the following equation,

s: scale factor _{(u r, v r) =} (0,0) and Z _C = Z _alpha by substituting (scalar value depth distance), or to calculate the _{u l} a displacement d,
Or
_{(U l, v l) =} (0,0) and Z _C = Z _alpha by substituting (scalar value depth distance), it is also preferable to calculate the _{u r} as the displacement amount -d.

ｓ：スケール係数
注視点位置取得手段は、世界座標系における注視点の座標を、変換行列Ｗc,wを用いてカメラ座標系へ変換し、そのＺcを奥行き距離のスカラー値Ｚ_αとして算出することも好ましい。 According to another embodiment of the image display device of the present invention,
When using a visible light camera, the gazing point position acquisition means stores in advance a marker image that identifies the gazing point, and detects the marker image from the photographed image, whereby the world coordinates where the marker is arranged are as follows: A transformation matrix (position and orientation parameter) Wc, w from the system (Xw, Yw, Zw) to the camera coordinate system (Xc, Yc, Zc) is calculated;

s: Scale coefficient The gazing point position acquisition means converts the coordinates of the gazing point in the world coordinate system into the camera coordinate system using the transformation matrix Wc, w, and calculates the Zc as a scalar value Z _α of the depth distance. Is also preferable.

According to another embodiment of the image display device of the present invention,
It is also preferable that the gazing point position acquisition unit sets the barycentric coordinates in the marker image area as the gazing point.

According to another embodiment of the image display device of the present invention,
Object drawing means
For each virtual object, draw only on one screen that is not selected as the overlap area drawing target,
Fill each overlapping area of the left eye screen and right eye screen with the transparent color of the display,
It is also preferable to draw the virtual object on the other non-drawing screen.

According to the present invention, in an actual object that is arranged in front of each of the user's right eye and left eye and has an optical see-through binocular display that can be seen through the real space, and the user gazes at. A program for causing a computer mounted on a device that controls a virtual object to be displayed on a display to function in accordance with a gaze point,
Gazing point position acquisition means for acquiring a depth position from the camera to the user's gazing point;
Overlapping area calculation means for calculating an overlapping area where the right eye screen and the left eye screen that are visually recognized at a depth position through the display from each of the user's right eye and left eye,
A drawing area selection means for selecting only one of the right eye screen and the left eye screen for the virtual object to be displayed in the overlapping area;
The computer is caused to function as an object drawing means for drawing the virtual object on the selected right eye screen or left eye screen.

According to the present invention, in an actual object that is arranged in front of each of the user's right eye and left eye and has an optical see-through binocular display that can be seen through the real space, and the user gazes at. An image display method for a device that controls a virtual object to be displayed on a display according to a gaze point,
The device
A first step of obtaining a depth position from the camera to the user's point of interest;
A second step of calculating, from each of the user's right eye and left eye, an overlapping area where the right eye screen and the left eye screen viewed at a depth position through the display intersect;
A third step of selecting only one of the right eye screen and the left eye screen for the virtual object to be displayed in the overlapping area;
And a fourth step of drawing the virtual object on the selected right eye screen or left eye screen.

  According to the image display device, program, and method of the present invention, when an object is displayed on a spectacle-shaped optical see-through type binocular display, the visibility of the object in the overlapping region of each screen that the user visually recognizes through the display. Can be improved. That is, the present invention pays attention to the fact that the binocular viewing angle of the optical see-through HMD can be expanded by utilizing the property at the time of short distance viewing not assumed by the HMD.

It is an external view showing a user's field of view seen from an optical see-through HMD. It is an external view showing the relationship between two screens and a user's field of view for an optical see-through HMD. It is explanatory drawing showing the screen reflected in a user's visual field when a calibration parameter does not contain an estimation error. It is explanatory drawing showing the screen reflected in a user's visual field in case a calibration parameter contains an error. It is a functional block diagram of the image display apparatus in this invention. It is explanatory drawing showing the relationship between a world coordinate system and a camera coordinate system. It is explanatory drawing showing the displacement amount between both screens. It is explanatory drawing showing the positional relationship and each area | region of a left eye screen, a right eye screen, and a binocular screen. It is the 1st explanatory view showing the drawing field to a virtual object. It is the 2nd explanatory view showing the drawing field to a virtual object. It is the 3rd explanatory view showing the drawing field to a virtual object. It is the 4th explanatory view showing the drawing field to a virtual object. It is the 5th explanatory view showing the drawing field to a virtual object. It is explanatory drawing showing the drawing procedure of an object drawing part. It is explanatory drawing showing the screen reflected in the user's visual field in this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 5 is a functional configuration diagram of the image display apparatus according to the present invention.

  According to FIG. 5, the function of the image display device 1 may be built in the optical see-through HMD, or may be an external terminal connected to the optical see-through HMD. Examples of the terminal include a smartphone, a tablet, and a personal computer. In this case, the function of the image display device is provided as an application that can be installed in the terminal.

  According to FIG. 5, the display of the HMD main body is an optical see-through type that can be seen through the real space, and is disposed in front of each of the left and right eyes of the user. The image display device 1 includes a gazing point position acquisition unit 11, an overlapping region calculation unit 12, a drawing region selection unit 13, and an object drawing unit 14. These functional components are realized by executing a program that causes a computer installed in the apparatus to function. Further, the flow of processing of these functional components can be understood as an image display method of the apparatus.

[Gaze point acquisition unit 11]
The gazing point position acquisition unit 11 acquires the depth position to the gazing point as viewed from the user wearing the HMD. By acquiring the depth position up to the point of sight, the virtual object can be displayed at the focus position of the user's line of sight while controlling the position and size. Specifically, the depth position is a distance from the camera equipped with the HMD to the user's gazing point with respect to the real space. Then, the acquired gazing point distance is output to the overlapping area calculation unit 12.

The gazing point position acquisition unit 11 can acquire the gazing point distance to the depth position using various devices. Examples of devices include the following.
Visible light camera Depth sensor (depth camera)
Ultrasonic sensor and magnetic sensor (consisting of a dedicated transmitter and sensors attached to the HMD body and the object to be watched)

<When using a visible light camera>
The gazing point position acquisition unit 11 stores in advance a marker image that identifies the gazing point. The marker image may be an image of the target object itself to be watched by the user, or may be an image in which a dedicated marker image is pasted on the target object. The marker image is detected by image recognition from the image captured by the visible light camera. The marker image may be a QR code (registered trademark), for example. At this time, it is also preferable to set the center-of-gravity coordinates in the marker image area as the point of gaze.

  FIG. 6 is an explanatory diagram showing the relationship between the world coordinate system and the camera coordinate system.

ｓ：スケール係数
そして、注視点位置取得部１１は、世界座標系における注視点の座標を、変換行列Ｗc,wを用いてカメラ座標系へ変換し、そのＺcを奥行き距離のスカラー値Ｚ_αとして算出する。 The gazing point position acquisition unit 11 acquires the position and orientation of the target object (see, for example, Non-Patent Document 1). This calculates a transformation matrix (position and orientation parameter) Wc, w from the world coordinate system (Xw, Yw, Zw) in which the marker image is arranged to the camera coordinate system (Xc, Yc, Zc) by the following formula ( For example, refer nonpatent literature 1).

s: Scale factor The gazing point position acquisition unit 11 converts the coordinates of the gazing point in the world coordinate system into the camera coordinate system using the transformation matrix Wc, w, and uses the Zc as the scalar value Z _α of the depth distance. calculate.

  The transformation matrix Wc, w may be calculated from a point correspondence set between the world coordinate system and the pixel coordinate system of the image plane using a robust estimation algorithm such as RANSAC (RAndom SAmple Consensus). It is also possible to remove erroneous point correspondences from the point correspondence set.

  As shown in FIG. 6, when the optical axis direction of the camera coordinate system of the visible light camera can be approximated to the positive or negative direction of the normal vector of the virtual screen, it is between the origin of the coordinate system of the visible light camera and the gazing point. Can be the gaze point distance.

Note that an average value or a median value of vertex groups constituting the virtual object may be used for the three-dimensional position X ¹ w of the point of interest in the world coordinate system. Further, for example, in the case of the work support system, when the position of the work instruction target can be acquired by another embodiment, this may be X ¹ w.

<When using a depth sensor>
When using a depth sensor (depth camera), the depth value of the entire visual recognition area can be acquired as well as the specific target object to which the marker image is pasted. The depth sensor can acquire the position of the real object as the position of the gazing point by capturing the reflection of the infrared ray emitted from the infrared irradiation sensor.

[Overlapping region calculation unit 12]
The overlapping area calculation unit 12 calculates, from each of the user's right eye and left eye, an overlapping area where the right eye screen and the left eye screen that are visually recognized through the display at a “depth distance” intersect. The calculated coordinate information of the overlapping area is output to the drawing area selecting unit 13.

ｓ：スケール係数 The overlapping area calculation unit 12 performs back projection on the depth position in the three-dimensional space via an arbitrary point from the right eye to the right eye screen (or the left eye to the left eye screen), and then the other left eye screen (or the right eye). The following transformation showing the positional relationship between the right eye screen coordinate system (u _r , v _r ) and the left eye screen coordinate system (u _l , v _l ) by back projection projected onto the screen) The matrix H _{l, r} is calculated.

s: Scale factor

The overlapping area calculation unit 12 can calculate the displacement amount d defined by the following transformation matrix H _{l, r} by back projection-projection calculation when the screens of both eyes are displaced only in the left-right direction.

  FIG. 7 is an explanatory diagram showing the amount of displacement between both screens.

  According to FIG. 7, after the back projection from the right eye to the depth position of the three-dimensional space via the upper left vertex of the right eye screen, the back projection-projection calculation projects to the other left eye screen. The quantity d is represented.

ｓ：スケール係数 Further, the back projection-projection calculation of the overlapping area calculation unit 13 may be defined by the following equation.
Right eye screen of the coordinate system _{(u r, v} r)
Left eye screen coordinate system (u _l , v _l )
Camera coordinate system (Xc, Yc, Zc) used for gazing point position acquisition means
Right eye calibration parameter _Pr
Left eye calibration parameter P _l

s: Scale factor

The displacement amount d is calculated by either of the following.
(1) By substituting (u _r , v _r ) = (0, 0) and Z _C = Z _α (scalar value of depth distance), u _l is calculated as the displacement d.
_{(2) (u l, v} l) = (0,0) and Z _C = Z _alpha by substituting (scalar value depth distance) to calculate the _{u r} as the displacement amount -d.
As a result, a back projection position on the binocular screen plane defined by Z _C = Z _α is obtained. The corresponding points on the other screen are calculated by substituting the three-dimensional coordinates of the backprojection position into the other equation.

[Drawing area selection unit 13]
The drawing area selection unit 13 selects only one of the right eye screen and the left eye screen for the virtual object to be displayed in the overlapping area.

  FIG. 8 is an explanatory diagram showing the positional relationship and each region of the left eye screen, right eye screen, and binocular screen.

According to FIG. 8, the binocular screen is divided into four regions A to D by combining the left eye screen and the right eye screen according to the displacement d [pixel] between the screens and the horizontal width w [pixel] of the screen. Has been.
Area A: Area displayed only on the left eye screen Area B: Area overlapped with the right eye screen on the left eye screen Area C: Area overlapped with the left eye screen on the right eye screen Area D: Area displayed only on the right eye screen

  The range of this overlapping area changes in real time depending on the focus position of the user's eyes. Specifically, the size becomes smaller as the wearer visually recognizes the distance, and becomes larger as the user visually recognizes the distance. For this purpose, the gazing point position acquisition unit 11 specifies the depth distance that the user gazes at, and determines the range of the overlapping region corresponding to the distance in real time.

For each virtual object to be drawn (one object may be divided into two), only the left screen area (B area) when the drawing position of that object spans the overlapping area between the left and right screens Or display only in the area on the right screen (C area). Here, in the binocular screen, the position of the boundary line in the range of the overlapping region is important. The drawing area selection unit 13 selects a screen on which a virtual object is to be drawn by one of the following two processes.
<Screen selection method based on visibility of virtual object>
<Screen selection method based on calibration parameter accuracy>

<Screen selection method based on visibility of virtual object>
A binocular screen composed of a right eye screen and a left eye screen is divided into a right eye region, an overlapping region, and a left eye region.
For each object, the drawing area is limited to one of the following.
Left eye screen (area A, B), right eye screen (area D), or
Left eye screen (region A) Right eye screen (regions C and D)
The virtual object to be displayed across the right eye region and the overlapping region is selected as the screen to be drawn in the overlapping region using the both end coordinates of the virtual object with respect to the binocular screen and the both end coordinates of the overlapping region. To do.
Further, a virtual object to be displayed across the left eye region and the overlap region using the both end coordinates of the virtual object with respect to the binocular screen and the end point coordinates of the overlap region is a left eye screen as a screen to be drawn in the overlap region. Select.

<Screen selection method based on calibration parameter accuracy>
When the accuracy of the calibration parameter P _{r for} the right eye is higher than the accuracy of the calibration parameter P _l for the left eye, the right eye screen is selected as the screen to be drawn in the overlapping area.
Left eye screen (region A) Right eye screen (regions C and D)
Conversely, when the accuracy of the calibration parameter P _r for the right eye is lower than the accuracy of the calibration parameter P _l for the left eye, the left eye screen is selected as a screen to be drawn in the overlapping area.
Left eye screen (areas A and B), right eye screen (area D)
Alternatively, when a specific condition regarding the display position of the virtual object is satisfied, a screen selection method based on the visibility of the virtual object is used, and otherwise, a screen selection method based on the accuracy of the calibration parameter is used. May be.

  FIG. 9 is a first explanatory diagram showing a drawing area for a virtual object.

  According to FIG. 9, for example, the projection position is calculated using equation (3) for the vertex coordinates of each polygon constituting the virtual object. This calculation is for grasping the approximate drawing position of the virtual object before actually rendering the virtual object. The calculation cost may be reduced by down-sampling the vertex group.

<< Selection Criteria when Virtual Object Exists Only in Two Areas (Left Eye Area and Overlapping Area, Overlapping Area and Right Eye Area) on Binocular Screen >> (FIGS. 9 to 11) >>
According to FIG. 9, the minimum value and the maximum value of the u coordinate are defined as follows for the point group that projects the point group constituting the virtual object on the screen based on Expression (3).
Drawing range of u coordinate of virtual object on left-eye screen: (u _l ^min , u _l ^max )
Drawing range of u coordinates of the virtual object in the right-eye ^{_{screen: (u r min, u r}} max)
Here, when the virtual object is drawn by the left eye screen of the region A and the right eye screen of the region C, the continuity of the virtual object is divided at the overlap region boundary.

  FIG. 10 is a second explanatory diagram illustrating a drawing area for a virtual object.

According to FIG. 10, the virtual object is drawn only on the left eye screens in the areas A and B, as compared with FIG. According to FIGS. 9 and 10, the following conditions are satisfied.
u _l ^min <d, d <u _l ^max , u _r ^max <w−d
In this case, the virtual object is preferably displayed only on the left eye screen (regions A and B). Thereby, the continuity of the virtual object only on the left eye screen is maintained, and visibility can be improved.

  FIG. 11 is a third explanatory diagram illustrating a drawing area for a virtual object.

According to FIG. 11, the minimum value and the maximum value of the u coordinate satisfy the following conditions for the projected point group of the virtual object.
^{_{d <u l min, u r}} min <w-d, w-d <u r max
In this case, in order to maintain the continuity of the virtual object on the right eye screen, the virtual object is preferably displayed only on the right eye screen (regions C and D).

<< Selection Criteria when Virtual Object Exists Only in Three Areas (Left Eye Area, Overlapping Area, and Right Eye Area) on Binocular Screen >> (FIGS. 12 to 13) >>
FIG. 12 is a fourth explanatory diagram illustrating a drawing area for a virtual object.

Than the length range of the d-u _l ^min can not be displayed only in the left eye screen, the right eye can not be displayed only on the screen u _r ^max - the longer of (w-d) of the length of the range.
^{_{d-u l min <u r}} max - (w-d)
u _l ^min <d, w <u _l ^max , u _r ^min <0, w−d <u _r ^max
In this case, in order to give priority to the range that can be displayed only on the right eye screen, the virtual object preferably displays the areas C and D on the right eye screen and the area A on the left eye screen.

  FIG. 13 is a fifth explanatory diagram illustrating a drawing area for a virtual object.

According to FIG. 13, among the virtual objects, the continuity of the text part can be maintained by displaying the text part to be claimed as the display to the user in the areas A and B of the left eye screen. In this case, u _l ^min , u _l ^max , u _r ^min , and u _r ^max are extracted for only the text object, and the following conditions are satisfied.
^{_{d-u l min> u r}} max - (w-d)
In this case, the virtual object preferably displays the areas A and B on the left eye screen and the area D on the right eye screen in order to give priority to the range that can be displayed only on the left eye screen. That is, the screen to be drawn is selected according to the size of the text object among the virtual objects.

[Object drawing unit 14]
The object drawing unit 14 draws the virtual object on the selected right eye screen or left eye screen.

  FIG. 14 is an explanatory diagram illustrating a drawing procedure of the object drawing unit.

The object drawing unit 14 executes the following drawing process on the binocular screen. FIG. 14 shows a case where the right eye screen is selected as a drawing target of the overlapping area.
(S11) For each virtual object, drawing is performed only on one screen that is not selected as a drawing target of the overlapping area. For example, the objects of FIGS. 12 and 14 and CG2 of FIG. 15 are drawn only on the left eye screen. 13 and CG1 in FIG. 15 are drawn only on the right eye screen.
(S12) The overlapping areas of the left eye screen and the right eye screen are filled with the transparent color of the display.
(S13) The virtual object is drawn on the other screen not drawn in (S11). For example, the objects of FIGS. 12 and 14 and CG2 of FIG. 15 are drawn only on the right eye screen. 13 and CG1 in FIG. 15 are drawn only on the left eye screen.

  As described above in detail, according to the image display device, the program, and the method of the present invention, when an object is displayed on the binocular optical see-through binocular display in the form of glasses, as viewed from the user, The visibility of the object in the overlapping area of the screen can be improved.

  FIG. 15 is an explanatory diagram showing a screen shown in the user's field of view according to the present invention.

  According to FIG. 15, either the left eye screen or the right eye screen is selected for each virtual object in order to maintain the visibility. For this reason, even if the viewing angle of the optical see-through HMD is increased (even if it is a manual operation), the visibility of the virtual object in the overlapping region of the binocular screen is not impaired. That is, the virtual object is not displayed twice in the overlapping area, and the luminance is also unified. According to the present invention, it is possible to prevent a decrease in visibility due to a calibration parameter estimation error and a luminance difference.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

1 Image display device, head mounted display, terminal 11 Gaze point acquisition unit
12 Overlapping region calculation unit 13 Drawing region selection unit 14 Object drawing unit

Claims (12)

It is arranged in front of each of the user's right eye and left eye, has an optical see-through type binocular display that is visible through the real space, and matches the gazing point in the real object that the user gazes, An image display device for controlling a virtual object to be displayed on the display,
Gazing point position acquisition means for acquiring a depth position from the camera to the user's gazing point;
An overlapping area calculation means for calculating an overlapping area where a right eye screen and a left eye screen viewed at the depth position through the display intersect from each of the user's right eye and left eye;
A drawing area selecting means for selecting only one of the right eye screen and the left eye screen for the virtual object to be displayed in the overlapping area;
An image display device comprising: an object drawing unit that draws the virtual object on a selected right eye screen or left eye screen. The drawing area selecting means includes
The binocular screen composed of the right eye screen and the left eye screen is divided into a right eye region, an overlapping region, and a left eye region,
Using both end coordinates of the virtual object with respect to the binocular screen and both end coordinates of the overlapping area,
The virtual object to be displayed across the right eye region and the overlapping region is selected as the screen to be drawn in the overlapping region, and the right eye screen is selected.
The image display apparatus according to claim 1, wherein a virtual object to be displayed across the left eye region and the overlapping region selects a left eye screen as a screen to be drawn in the overlapping region. The drawing area selecting means includes
For the virtual object to be displayed in the overlap area,
When the accuracy of the calibration parameter P _{r for} the right eye is higher than the accuracy of the calibration parameter P _l for the left eye, the right eye screen is selected as a screen to be drawn in the overlapping region,
Conversely, when the accuracy of the calibration parameter P _r for the right eye is lower than the accuracy of the calibration parameter P _l for the left eye, the left eye screen is selected as a screen to be drawn in the overlapping area.
The image display apparatus according to claim 1. The back projection-projection calculation of the overlapping area calculation means is performed by performing back projection to a depth position in the three-dimensional space via an arbitrary point from the right eye to the right eye screen (or from the left eye to the left eye screen), and then to the other left Positional relationship between the coordinate system (u _r , v _r ) of the right eye screen and the coordinate system (u _l , v _l ) of the left eye screen by back projection-projection calculation that projects onto the eye screen (or right eye screen) Calculate the following transformation matrix H _{l, r}

The image display device according to claim 1, wherein s is a scale factor. The overlapping area calculating means calculates a displacement amount d defined by the following transformation matrix H _{l, r} by back projection-projection calculation when the screens of both eyes are displaced only in the left-right direction.

The image display device according to claim 4. The overlapping area calculation means performs back projection on the depth position in the three-dimensional space via an arbitrary point from the right eye to the right eye screen (or from the left eye to the left eye screen), and then to the other left eye screen (or right The displacement amount d defined by the transformation matrix H _{l, r} is calculated when the screen is shifted only in the left-right direction by back projection-projection calculation projected onto an eye screen). The image display device according to 4 or 5. The back projection-projection calculation of the overlapping area calculation means includes a right eye screen coordinate system (u _r , v _r ), a left eye screen coordinate system (u _l , v _l ), and a right eye calibration parameter. using the P _r, a calibration parameter P _l of the left eye, the gaze point the camera coordinate system used by the position acquisition unit (Xc, Yc, Zc) and was defined by the following equation,

s: scale factor _{(u r, v r) =} (0,0) and Z _C = Z _alpha by substituting (scalar value depth distance), or to calculate the _{u l} a displacement d,
Or
_{(U l, v l) =} (0,0) and Z _C = Z _alpha by substituting (scalar value depth distance), claim and calculates a _{u r} as the displacement amount -d 6 The image display device described in 1. When using the visible light camera, the gazing point position acquisition unit stores in advance a marker image that identifies the gazing point, and detects the marker image from the photographed image, whereby the world in which the marker is arranged is as follows. A transformation matrix (position and orientation parameter) Wc, w from the coordinate system (Xw, Yw, Zw) to the camera coordinate system (Xc, Yc, Zc) is calculated,

s: Scale coefficient The gazing point position acquisition unit converts the coordinates of the gazing point in the world coordinate system into the camera coordinate system using the transformation matrix Wc, w, and calculates Zc as a scalar value Z _α of the depth distance. The image display device according to claim 1, wherein the image display device is an image display device. The image display apparatus according to claim 8, wherein the gazing point position acquisition unit uses a centroid coordinate in an area of the marker image as the gazing point. The object drawing means includes
For each virtual object, draw only on one screen that is not selected as the overlap area drawing target,
Fill each overlapping area of the left eye screen and right eye screen with the transparent color of the display,
The image display device according to claim 1, wherein the virtual object is drawn on the other screen that is not drawn. It is arranged in front of each of the user's right eye and left eye, has an optical see-through type binocular display that is visible through the real space, and matches the gazing point in the real object that the user gazes, A program for causing a computer mounted on a device for controlling a virtual object to be displayed on the display to function,
Gazing point position acquisition means for acquiring a depth position from the camera to the user's gazing point;
An overlapping area calculation means for calculating an overlapping area where a right eye screen and a left eye screen viewed at the depth position through the display intersect from each of the user's right eye and left eye;
A drawing area selecting means for selecting only one of the right eye screen and the left eye screen for the virtual object to be displayed in the overlapping area;
A program that causes a computer to function as object drawing means for drawing the virtual object on a selected right-eye screen or left-eye screen. It is arranged in front of each of the user's right eye and left eye, has an optical see-through type binocular display that is visible through the real space, and matches the gazing point in the real object that the user gazes, An image display method of an apparatus for controlling a virtual object to be displayed on the display,
The device is
A first step of obtaining a depth position from the camera to the user's point of interest;
A second step of calculating, from each of the user's right eye and left eye, an overlapping region where the right eye screen and the left eye screen viewed at the depth position through the display intersect;
A third step of selecting only one of the right eye screen and the left eye screen for the virtual object to be displayed in the overlapping area;
And a fourth step of drawing the virtual object on the selected right-eye screen or left-eye screen.

Images