JP7217206B2 - Image display device, image display system and image display method - Google Patents

Description

The present invention relates to an image display device, an image display system and an image display method.

A head-mounted display connected to a game machine is worn on the head, and a game is played by operating a controller or the like while viewing a screen displayed on the head-mounted display. When the user wears the head-mounted display, the user does not see anything other than the image displayed on the head-mounted display, so that the sense of immersion in the image world is enhanced, which has the effect of further enhancing the entertainment of the game. In addition, a virtual reality (VR) image is displayed on a head-mounted display, and when the user wearing the head-mounted display rotates his or her head, a 360-degree panoramic view of the virtual space is displayed. As a result, the feeling of being immersed in the video is further enhanced, and the operability of applications such as games is also improved.

In this way, when a head-tracking function is provided to a head-mounted display and a virtual reality image is generated by changing the viewpoint and the line-of-sight direction in conjunction with the movement of the user's head, the time from generation to display of the virtual reality image is as follows. Due to the delay, there is a discrepancy between the orientation of the user's head, which was assumed when the image was generated, and the orientation of the user's head when the image was displayed on the head-mounted display, causing the user to feel intoxicated. feeling (called “VR sickness (Virtual Reality Sickness)”, etc.).

Therefore, a process called "time warp" or "reprojection" is performed to correct the rendered image to match the latest position and orientation of the head-mounted display, and measures have been taken to make it difficult for the user to perceive the deviation. there is

In addition, since virtual reality requires a high frame rate video experience, the frame rate is pseudo-improved by performing reprojection asynchronously with low frame rate rendering.

In conventional reprojection processing, the entire image is transformed on the assumption that the depth is uniform even in areas with different depths, so the reprojected image may give a sense of incongruity. Especially in the case of images that include areas with greatly different depths, interpolation of one frame is the limit at a frame rate of 120 fps (frames per second) in order to avoid discomfort due to reprojection. There is a limit. In addition, there are cases where it is not desired to uniformly reproject menus and dialogs whose display positions are fixed on the screen.

The present invention has been made in view of these problems, and an object of the present invention is to provide an image display device, an image display system, and an image display method capable of suppressing discomfort due to image conversion.

In order to solve the above-described problems, an image display device according to one aspect of the present invention performs reprojection processing for transforming an image including depth value information so as to match a viewpoint position or line-of-sight direction according to a plurality of different depth values. and a reprojection unit for generating reprojected images according to a plurality of different depth values.

Another aspect of the present invention is also an image display device. This device performs reprojection processing to transform a UV texture storing UV coordinate values for sampling an image including depth value information so as to match the viewpoint position or line-of-sight direction according to a plurality of different depth values. a reprojection unit for generating a plurality of UV textures reprojected according to a plurality of different depth values; sampling the image using the plurality of UV textures converted by the reprojection processing; and a distortion processing unit that performs distortion processing for deforming the .

Yet another aspect of the invention is an image display system. This image display system includes an image display device and an image generation device, wherein the image generation device renders an object in a virtual space to generate a computer graphics image including depth value information. and a transmitting unit configured to transmit the computer graphics image including the depth value information to the image display device. The image display device comprises: a receiving unit that receives the computer graphics image including the depth value information from the image generating device; and a reprojection unit that performs reprojection processing for converting to match the viewpoint position or line-of-sight direction using a reprojection unit, and generates a computer graphics image that has undergone reprojection processing according to a plurality of different depth values.

Yet another aspect of the present invention is an image display method. This method comprises the steps of: performing reprojection processing for transforming an image including depth value information so as to match a viewpoint position or a line-of-sight direction according to a plurality of different depth values; and generating a processed image.

Any combination of the above constituent elements, and expressions of the present invention converted between methods, devices, systems, computer programs, data structures, recording media, etc. are also effective as aspects of the present invention.

According to the present invention, it is possible to suppress discomfort due to image conversion.

1 is an external view of a head mounted display; FIG. 1 is a configuration diagram of an image generation system; FIG. 1 is a functional configuration diagram of a head mounted display; FIG. 1 is a functional configuration diagram of an image generation device; FIG. It is a figure explaining a structure of an image generation system. FIG. 10 is a diagram illustrating a procedure of asynchronous reprojection processing; FIG. 7A is a diagram for explaining reprojection processing and distortion processing according to the conventional method, and FIG. 7B is a diagram for explaining reprojection processing and distortion processing according to the method of this embodiment. It is a figure explaining a depth reprojection process and a distortion process. It is a figure explaining depth UV reprojection processing and distortion processing.

FIG. 1 is an external view of the head mounted display 100. FIG. The head-mounted display 100 is an image display device that is worn on the user's head to view still images and moving images displayed on the display and to listen to sounds and music output from headphones.

A gyro sensor or an acceleration sensor built in or external to the head mounted display 100 measures the position information of the head of the user wearing the head mounted display 100 and the orientation information such as the rotation angle and inclination of the head. be able to.

A camera unit is mounted on the head mounted display 100, and the user can photograph the outside world while wearing the head mounted display 100. FIG.

Head-mounted display 100 is an example of a "wearable display." Here, a method of generating an image to be displayed on the head mounted display 100 will be described, but the image generating method of the present embodiment is not limited to the head mounted display 100 in a narrow sense. It can also be applied when wearing headphones, headsets (headphones with a microphone), earphones, earrings, ear cameras, hats, hats with cameras, hair bands, and the like.

FIG. 2 is a configuration diagram of an image generation system according to this embodiment. The head mounted display 100 is connected to the image generation device 200 via an interface 300 such as HDMI (registered trademark) (High-Definition Multimedia Interface), which is a communication interface standard for transmitting video and audio as digital signals. .

The image generation device 200 predicts the position/orientation information of the head-mounted display 100 from the current position/orientation information of the head-mounted display 100 in consideration of the delay from image generation to display. Based on the position/orientation information, an image to be displayed on the head mounted display 100 is drawn and transmitted to the head mounted display 100 .

An example of the image generation device 200 is a game machine. The image generation device 200 may also be connected to a server via a network. In that case, the server may provide the image generating device 200 with an online application such as a game in which multiple users can participate via a network. The head mounted display 100 may be connected to a computer or mobile terminal instead of the image generation device 200. FIG.

FIG. 3 is a functional configuration diagram of the head mounted display 100 according to this embodiment.

The control unit 10 is a main processor that processes and outputs signals such as image signals and sensor signals, commands and data. The input interface 20 receives operation signals and setting signals from the user and supplies them to the control unit 10 . The output interface 30 receives image signals from the control unit 10 and displays them on the display panel 32 .

The communication control unit 40 transmits data input from the control unit 10 to the outside by wired or wireless communication via the network adapter 42 or the antenna 44 . The communication control unit 40 also receives data from the outside by wired or wireless communication via the network adapter 42 or the antenna 44 and outputs the data to the control unit 10 .

The storage unit 50 temporarily stores data, parameters, operation signals, and the like processed by the control unit 10 .

The orientation sensor 64 detects position information of the head mounted display 100 and orientation information such as the rotation angle and inclination of the head mounted display 100 . The attitude sensor 64 is realized by appropriately combining a gyro sensor, an acceleration sensor, an angular acceleration sensor, and the like. A motion sensor that combines at least one of a 3-axis geomagnetic sensor, a 3-axis acceleration sensor, and a 3-axis gyro (angular velocity) sensor may be used to detect forward/backward, left/right, and up/down movements of the user's head.

The external input/output terminal interface 70 is an interface for connecting peripheral devices such as a USB (Universal Serial Bus) controller. The external memory 72 is an external memory such as flash memory.

The transmission/reception unit 92 receives an image generated by the image generation device 200 from the image generation device 200 and supplies the image to the control unit 10 .

Based on the latest position/orientation information of the head-mounted display 100 detected by the orientation sensor 64, the reprojection unit 84 performs reprojection processing on the UV texture storing the UV coordinate values for sampling the image. Convert to UV texture according to the latest viewpoint position/line-of-sight direction of the mount display 100 .

Referencing the image using the converted UV texture provides the same effect as reprojection of the image, but the degree of deterioration in image quality is different. Direct sampling of an image inevitably degrades the image by performing an interpolation process such as bilinear interpolation between adjacent pixel values. On the other hand, the UV texture is a texture in which UV values that change linearly are arranged, so unlike pixel values, the linearity of the UV values obtained is not lost. UV texture reprojection has the advantage that non-linear conversion of pixel values due to bilinear interpolation, which occurs in image reprojection, does not occur. However, since textures that store UV values also have resolution limitations, the UV values obtained by bilinear interpolation of UV textures differ from true UV values and have a certain amount of rounding error. Therefore, by making the resolution of the texture that stores the UV values larger than that of the image, the error due to interpolation during sampling can be reduced, or by storing the UV values in a texture with a large bit length such as 32 bits per color. It is also possible to reduce the conversion error. By increasing the resolution and accuracy of the UV texture in this manner, it is possible to suppress deterioration of the image.

The distortion processing unit 86 samples the image with reference to the reprojection-processed UV texture, and deforms the sampled image according to the distortion caused by the optical system of the head-mounted display 100 . Then, the distortion-processed image is supplied to the control unit 10 .

The head-mounted display 100 employs an optical lens with a high curvature in order to display an image with a wide viewing angle in front of and around the user's eyes, and the user looks into the display panel through the lens. If a lens with a high curvature is used, the image will be distorted due to lens distortion. Therefore, in order to make it look correct when viewed through a lens with a high curvature, distortion processing is applied to the rendered image in advance, and the image after distortion processing is transmitted to the head-mounted display and displayed on the display panel, where the user can view the image. Make it look normal when viewed through a lens with high curvature.

The control unit 10 can supply image and text data to the output interface 30 for display on the display panel 32, and can supply the data to the communication control unit 40 for external transmission.

The current position/orientation information of the head mounted display 100 detected by the orientation sensor 64 is notified to the image generation device 200 via the communication control section 40 or the external input/output terminal interface 70 . Alternatively, the transmitting/receiving section 92 may transmit the current position/orientation information of the head mounted display 100 to the image generation device 200 .

FIG. 4 is a functional configuration diagram of the image generation device 200 according to this embodiment. The figure is a block diagram focusing on functions, and these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

At least part of the functions of the image generation device 200 may be implemented in the head mounted display 100 . Alternatively, at least part of the functions of image generation device 200 may be implemented in a server connected to image generation device 200 via a network.

The position/posture acquisition unit 210 acquires the current position/posture information of the head mounted display 100 from the head mounted display 100 .

The viewpoint/line-of-sight setting unit 220 uses the position/orientation information of the head mounted display 100 acquired by the position/orientation acquisition unit 210 to set the user's viewpoint position and line-of-sight direction.

The image generation unit 230 reads data necessary for generating computer graphics (CG) from the image storage unit 260, renders objects in the virtual space to generate a CG image, performs post-processing, and stores the data in the image storage unit 260. Output.

The image generation section 230 includes a rendering section 232 and a post-processing section 236 .

The rendering unit 232 renders an object in the virtual space seen in the line-of-sight direction from the viewpoint position of the user wearing the head-mounted display 100 according to the user's line-of-sight position and line-of-sight direction set by the viewpoint/line-of-sight setting unit 220, and performs CG rendering. An image is generated and provided to the post-processor 236 .

The post-processing unit 236 performs post-processing such as depth-of-field adjustment, tone mapping, and anti-aliasing on the CG image, performs post-processing so that the CG image looks natural and smooth, and stores it in the image storage unit 260. do.

The transmission/reception unit 282 reads the frame data of the CG image generated by the image generation unit 230 from the image storage unit 260 and transmits it to the head mounted display 100 . The transmitting/receiving unit 282 may read the frame data of the CG image including the alpha value and the depth information, and transmit it to the head mounted display 100 as an RGBAD image via a communication interface capable of transmitting RGBAD image signals. Here, the RGBAD image signal is an image signal obtained by adding an alpha value and a depth value to red, green, and blue color values for each pixel.

FIG. 5 is a diagram for explaining the configuration of the image generation system according to this embodiment. Here, in order to simplify the description, the main configurations of the head mounted display 100 and the image generating device 200 for generating and displaying CG images will be illustrated and described.

The user's viewpoint position and line-of-sight direction detected by the orientation sensor 64 of the head-mounted display 100 are transmitted to the image generating device 200 and supplied to the rendering section 232 .

The rendering unit 232 of the image generation device 200 generates a virtual object viewed from the viewpoint position/line-of-sight direction of the user wearing the head-mounted display 100 and gives the CG image to the post-processing unit 236 .

The post-processing unit 236 performs post-processing on the CG image, transmits it to the head mounted display 100 as an RGBAD image including alpha values and depth information, and supplies it to the reprojection unit 84 .

The reprojection unit 84 of the head-mounted display 100 acquires the latest viewpoint position and line-of-sight direction of the user detected by the orientation sensor 64, and converts the UV texture storing the UV coordinate values for sampling the CG image to the latest viewpoint. It is converted so as to match the position and line-of-sight direction, and is supplied to the distortion processing unit 86 .

The distortion processor 86 samples the CG image with reference to the reprojected UV texture, and applies distortion processing to the sampled CG image. A CG image that has undergone distortion processing is displayed on the display panel 32 .

In the present embodiment, the case where reprojection unit 84 and distortion processing unit 86 are provided in head mounted display 100 has been described, but reprojection unit 84 and distortion processing unit 86 may be provided in image generation device 200 . Providing the reprojection unit 84 and the distortion processing unit 86 in the head mounted display 100 is advantageous in that the latest orientation information detected by the orientation sensor 64 can be used in real time. However, if the processing capability of the head mounted display 100 is restricted, a configuration in which the reprojection unit 84 and the distortion processing unit 86 are provided in the image generation device 200 can be adopted. In that case, the latest orientation information detected by orientation sensor 64 is received from head mounted display 100 , reprojection processing and distortion processing are performed in image generation device 200 , and the resulting image is transmitted to head mounted display 100 .

FIG. 6 is a diagram for explaining the procedure of asynchronous reprojection processing according to this embodiment.

A head tracker including the attitude sensor 64 of the head mounted display 100 estimates the attitude of the user wearing the head mounted display at the timing of the n-th vertical synchronization signal (VSYNC) (S10).

A game engine runs a game thread and a rendering thread. The game thread generates a game event at the timing of the nth VSYNC (S12). The rendering thread executes scene rendering based on the pose estimated at the timing of the n-th VSYNC (S14), and post-processes the rendered image (S16). Since scene rendering generally takes time, it is necessary to perform reprojection based on the latest posture until the next scene rendering is performed.

Reprojection occurs asynchronously with rendering by the rendering thread at the timing of GPU interrupts. The head tracker estimates the posture at the (n+1)th VSYNC timing (S18). Based on the posture estimated at the timing of the (n+1)th VSYNC, reprojection processing is performed on the UV texture for referring to the rendered image at the timing of the nth VSYNC. The timing UV texture is converted to the (n+1)-th VSYNC timing UV texture (S20). Referring to the reprojected UV texture, the rendered image is sampled at the n-th VSYNC timing, distortion processing is performed (S22), and a distortion image at the (n+1)-th VSYNC timing is output.

Similarly, the head tracker estimates the attitude at the (n+2)th VSYNC timing (S24). Based on the posture estimated at the timing of the (n+2)th VSYNC, reprojection processing is performed on the UV texture for referring to the rendered image at the timing of the nth VSYNC. The timing UV texture is converted to the (n+2)-th VSYNC timing UV texture (S26). Referring to the reprojected UV texture, the image rendered at the nth VSYNC timing is sampled and distortion processing is performed (S28), and a distortion image at the (n+2)th VSYNC timing is output.

Although the case where asynchronous reprojection is performed twice before executing the next scene rendering has been described here, the number of times asynchronous reprojection is performed varies depending on the time required for scene rendering.

Reprojection processing and distortion processing according to the conventional method and reprojection processing and distortion processing according to the method of the present embodiment will be compared and described with reference to FIGS. 7(a) and 7(b).

In a GPU having a programmable shader function, a vertex shader processes attribute information of polygon vertices, and a pixel shader processes an image on a pixel-by-pixel basis.

FIG. 7A shows reprojection processing and distortion processing according to the conventional method. In the first pass of rendering, the vertex shader performs reprojection processing on image 400 to generate image 410 after reprojection processing. Next, in a second pass, the pixel shader distorts the reprojected image 410 to produce a distorted image 420 . Distortion processing includes chromatic aberration correction for each color of RGB.

In the conventional method of FIG. 7(a), when the vertex shader performs reprojection processing in the first pass, pixels are sampled from the image 400 and a post-reprojection image 410 is generated by bilinear interpolation or the like. Next, when the pixel shader performs distortion processing in the second pass, pixels are sampled from the post-reprojection image 410 to generate a post-distortion image 420 by bilinear interpolation or the like. That is, since pixel sampling and interpolation are performed twice, ie, the first pass and the second pass, the image quality is inevitably degraded.

Note that if the reprojection process is performed by the vertex shader, the distortion process cannot be performed by the pixel shader in the same rendering pass. This is because the pixel shader cannot sample other pixels generated in the same pass. Therefore, it is divided into two passes, the first pass and the second pass. In the first pass, the vertex shader performs reprojection, once the image after reprojection processing is written to memory, and in the second pass, the pixel shader performs reprojection. Distortion processing is applied to the processed image. In that case, deterioration of image quality due to two pixel samplings cannot be avoided.

If reprojection processing and distortion processing are performed in one pass, there is no choice but to execute reprojection processing and distortion processing in the vertex shader. , can handle only one screen coordinate, so it is not possible to calculate different distortions for each RGB color for each pixel at once in the vertex shader. That is, in order to correct the chromatic aberration of each color of RGB with the vertex shader and the pixel shader, there is no choice but to correct the chromatic aberration of each color of RGB with the pixel shader of the second pass, and the number of times of sampling must be two.

FIG. 7B shows reprojection processing and distortion processing according to the method of this embodiment. In the first pass, the vertex shader performs reprojection processing on a UV texture 500 storing UV coordinate values for sampling an image to generate a post-reprojection UV texture 510 . Next, in a second pass, the pixel shader samples the image 400 with reference to the reprojected UV texture 510 and generates a distorted image 420 by bilinear interpolation or the like.

In this method, reprojection is performed not on the image, but on the UV texture that is the reference source of the texture when the image is texture-mapped (referred to as "UV reprojection"). UV reprojection does not sample the image when reprojecting the UV texture. Since image sampling and interpolation are performed only once when performing distortion processing in the second pass, deterioration in image quality is less than in the conventional method.

Also, when the UV texture is subjected to the reprojection process in the first pass, the size of the UV texture may be small because a sufficient approximate solution can be obtained by linear interpolation when the reprojection angle is small. Compared to the conventional method in which the image is directly reprojected and the converted image is stored in the memory, the memory capacity can be reduced and the power consumption required for memory access can be reduced.

As described above, according to the UV reprojection of this method, the original image before deformation is referred to based on the UV texture deformed by reprojection without directly sampling the image during reprojection. No degradation of image quality occurs.

Next, reprojection (referred to as “depth reprojection”) will be described so that an image containing information on depth values is reprojected so as to match the viewpoint position or line-of-sight direction according to a plurality of different depth values.

In depth reprojection, the reprojection unit 84 performs reprojection processing to convert an image to match the viewpoint position or line-of-sight direction according to a plurality of different depths. Synthesize multiple images to generate a composite image. A distortion processing unit 86 performs distortion processing on the synthesized image.

FIG. 8 is a diagram for explaining depth reprojection processing and distortion processing.

A depth value of each pixel of the image 400 is stored in the depth buffer. Here, as an example, the representative depths are set to f=0, 1, and 5 (units are meters as an example), and the depth range of the image is set to d=0, 0<d<3, 3≦d. Divide into 3 stages. An image region with a greater depth is displaced more by the reprojection process.

The reprojection unit 84 does not perform reprojection on the image in the area of the depth 600 value d=0. The original image 400 is used as it is for the region where the value d=0 of the depth 600 . A region with a value d=0 of the depth 600 is, for example, a menu or dialog displayed in front of the virtual space. Since the region with the value d=0 of the depth 600 is not subjected to reprojection processing, it is not affected by reprojection processing and does not move on the screen.

The reprojection unit 84 performs the reprojection process of f=1 for an area where the value d of the depth 600 is not 0 and the value d of the depth 602 after the reprojection process of f=1 is in the range of 0<d<3 after the reprojection process of f=1. Generate image 402 .

The reprojection unit 84 generates an image 404 after reprojection processing with f=5 for an area where the value d of the depth 600 is not 0 and the value d of the depth 602 after reprojection processing with f=1 is in the range of 3≦d. to generate

In the above description, an image subjected to reprojection processing is generated by applying reprojection processing to an image for each representative depth and synthesizing a plurality of images after reprojection processing for each representative depth. Alternatively, a reprojected image may be generated by 3D transforming each pixel of the image as a point cloud, or by generating a simple mesh from the depth buffer and performing 3D rendering. .

The distortion processing unit 86 performs distortion processing on the synthesized image 408 and generates an image 420 after the distortion processing.

By dividing the depth range of the image into a plurality according to a plurality of representative depths, performing reprojection processing for each representative depth, and synthesizing the plurality of images after the reprojection processing for each representative depth, the depth is not taken into account. Compared to the case of uniformly reprojecting the entire image, it is possible to generate a more natural image with less sense of incongruity. As a result, even if the frame rate of the image is increased by reprojection, it is possible to prevent unnatural movement.

The representative depth can be set arbitrarily, and may be divided into three or more. If there is no area where reprojection is not desired, such as a position-fixed menu, it is not necessary to set the case where the depth is zero. Also, the value or number of representative depths may be dynamically changed according to the depth distribution of the rendered image. The depth distribution valleys may be detected based on the depth histogram included in the image, and the value and number of representative depths may be determined so that the depth range is divided by the depth distribution valleys.

In the above description, the image is reprojected according to a plurality of different depths, but the UV reprojection technique may be applied here. The reprojection unit 84 performs reprojection processing on UV textures according to a plurality of different depths, and generates a plurality of reprojection processed UV textures according to a plurality of different depths. The distortion processor 86 samples an image using a plurality of UV textures converted by the reprojection process, performs distortion processing, and generates a distortion-processed image. This is called "depth UV reprojection".

FIG. 9 is a diagram for explaining depth UV reprojection processing and distortion processing.

By using UV reprojection, it is possible to generate a reprojection image with less sense of incongruity by reprojection according to the depth while avoiding deterioration of image quality due to sampling.

For depth, the representative depths are set to f = 0, 1, and 5, and the depth range of the image is divided into three stages of d = 0, 0 < d < 3, and 3 ≤ d. is reprojected. For the UV texture, the representative depths are set to f = 0, 1, 5, and 20, and the depth range of the image is set to d = 0, 0 < d < 3, 3 ≤ d < 10, 10 ≤ d. Divided into stages, reprojection is applied to the UV texture for each representative depth.

The reprojection unit 84 does not perform reprojection on the region of the depth 600 where the value d=0. The image 400 is sampled using the UV texture 500 as it is for the region of the value d=0 of the depth 600 .

The reprojection unit 84 uses the UV texture 502 for a region where the value d of the depth 600 is not 0 and the value d of the depth 602 after the reprojection process of f=1 satisfies 0<d<3. The image 400 is sampled.

In the reprojection unit 84, the value d of the depth 600 is not 0, the value d of the depth 602 after the reprojection process of f=1 is 3≦d, and the value of the depth 504 after the reprojection process of f=5 is For regions where d satisfies 3≦d<10, the UV texture 504 is used to sample the image 400 .

In the reprojection unit 84, the value of the depth 600 is not 0, the value d of the depth 602 after the reprojection process of f=1 is 3≦d, and the value d of the depth 604 after the reprojection process of f=5 is For regions where d is 10≦d, the UV texture 506 is used to sample the image 400 .

Since the user is not very sensitive to the error in the depth direction of the image, even if the number of representative depths is reduced for depth reprojection, the effect on the image quality is small.

In the description of FIG. 9, reprojection was applied to each of the depth and UV textures, but a (U, V, D) texture (called “UVD texture”) is generated by combining UV and depth, and UVD textures are generated. Reprojection may be applied to the texture. For example, in an image buffer that stores three colors of RGB, if a U value is stored in R (red), a V value is stored in G (green), and a depth value is stored in B (blue), the RGB image buffer UVD textures can be stored in . More efficient than reprojecting depth and UV textures separately.

Next, a method for coping with an occlusion area generated by depth reprojection will be described. Normal reprojection without considering depth (fixed depth) deforms the entire image and does not cause occlusion problems. However, when depth reprojection is performed, the amount of displacement varies depending on the depth, and the closer the object is, the greater the movement. Generally speaking, the movement of the foreground object creates an occlusion area that was previously invisible. Since the occlusion area cannot be drawn, if it is left as it is, it will be filled with black, etc., and it will be unnatural.

Therefore, in order not to make the occlusion area unnatural, the past frame (for example, the frame one frame before) is used as the initial value, and the image reprojected according to the depth by depth reprojection is displayed on it. Overwrite. As a result, even if an occlusion area is generated by depth reprojection, since the past frame is drawn as an initial value in the occlusion area, unnaturalness can be avoided.

Instead of the past frame as the initial value before depth reprojection, a post-reprojection past frame obtained by subjecting the past frame to normal reprojection with a fixed depth may be used. Since the past frame is the one that matches the viewpoint position or line-of-sight direction at the past point in time as it is, it is more natural to use the one that matches the current viewpoint position or line-of-sight direction by normal reprojection with fixed depth. can be obtained. Note that normal reprojection with a fixed depth does not generate an occlusion area unlike depth reprojection, so there is no problem in using it as an initial value.

Next, addition reprojection will be described. The resolution of the image can be increased by reprojecting an image obtained by past depth reprojection so as to match the current viewpoint position or line-of-sight direction and then adding it to the image obtained by the current depth reprojection. Here, instead of simply adding a plurality of frames, weighted addition, average value, and median value of a plurality of frames may be obtained. Additive reprojection is more effective when combined with ray tracing. Rendering by ray tracing takes time, so the frame rate is low, but by reprojecting and adding past rendering results, it is possible to increase the resolution in both the temporal and spatial directions. By additive reprojection, in addition to improving resolution, effects such as reduction of noise and aliasing, and conversion to HDR (High Dynamic Range) of an image by improving color depth can be obtained.

The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that the embodiments are examples, and that various modifications can be made to combinations of each component and each treatment process, and that such modifications are within the scope of the present invention. .

In the above embodiment, the distortion processing has been described on the premise that non-linear distortion occurs in the displayed image as in the optical system of the head-mounted display 100. However, not only non-linear distortion but also linear distortion This embodiment can be applied. For example, this embodiment can also be applied when at least part of the displayed image is enlarged or reduced. When projecting an image onto a wall or the like with a projector, the projector is installed obliquely so as to look up at the wall. This embodiment can also be applied to apply such linear distortion to an image.

In the above embodiment, the case of performing reprojection in accordance with the viewpoint of the head mounted display 100 has been described. For applications other than head-mounted displays, for example, when displaying on a television monitor, the UV reprojection and depth projection of the present embodiment are used in order to increase the frame rate by performing reprojection to match the viewpoint of the camera. Reprojection, depth UV reprojection can be used.

10 control unit 20 input interface 30 output interface 32 display panel 40 communication control unit 42 network adapter 44 antenna 50 storage unit 64 attitude sensor 70 external input/output terminal interface 72 external memory 84 reprojection Section 86 Distortion Processing Section 92 Transmission/Reception Section 100 Head Mounted Display 200 Image Generation Device 210 Position/Orientation Acquisition Section 220 Viewpoint/Line of Sight Setting Section 230 Image Generation Section 232 Rendering Section 236 Post-Processing Section 260 image storage unit, 282 transmission/reception unit, 300 interface.

Claims (13)

Executes reprojection processing that transforms an image containing depth value information so that it matches the viewpoint position or line-of-sight direction according to multiple different depth values, and generates reprojection processed images according to multiple different depth values. including a reprojection unit that
The image display device, wherein the reprojection unit generates the reprojection-processed image by overwriting a drawing result of a frame of a past image as an initial value. Executes reprojection processing that transforms an image containing depth value information so that it matches the viewpoint position or line-of-sight direction according to multiple different depth values, and generates reprojection processed images according to multiple different depth values. including a reprojection unit that
The reprojection unit converts a past image so as to match the current viewpoint position or line-of-sight direction, and then obtains an added value, an average value, and a median value between the reprojected image and the reprojected image. image display device. 3. The image display device according to claim 1, wherein the reprojection unit generates a composite image by synthesizing a plurality of images that have undergone reprojection processing according to a plurality of different depth values. 3. The image display device according to claim 1, wherein the reprojection unit does not perform reprojection processing on an area having a predetermined depth value. 3. The image display device according to claim 1, wherein said plurality of different depth values are determined based on a distribution of depth values included in said image. The reprojection unit overwrites the image obtained by performing the reprojection process of transforming the frame of the past image so as to match the viewpoint position or the line-of-sight direction without dividing the frame into a plurality of different depth values as an initial value. 3. The image display device according to claim 1, wherein the image is generated by said reprojection processing. An image display system including an image display device and an image generation device,
The image generation device is
a rendering unit that renders an object in a virtual space to generate a computer graphics image including depth value information;
a transmitting unit configured to transmit the computer graphics image including the depth value information to the image display device;
The image display device is
a receiving unit that receives the computer graphics image including the depth value information from the image generation device;
performing reprojection processing for transforming the computer graphics image including the depth value information so as to match a viewpoint position or a line-of-sight direction according to a plurality of different depth values, and performing reprojection processing according to the plurality of different depth values. a reprojection unit that generates a rendered computer graphics image;
The image display system according to claim 1, wherein the reprojection unit generates the reprojection-processed computer graphics image by overwriting the rendering result of a frame of a past image as an initial value. An image display system including an image display device and an image generation device,
The image generation device is
a rendering unit that renders an object in a virtual space to generate a computer graphics image including depth value information;
a transmitting unit configured to transmit the computer graphics image including the depth value information to the image display device;
The image display device is
a receiving unit that receives the computer graphics image including the depth value information from the image generation device;
performing reprojection processing for transforming the computer graphics image including the depth value information so as to match a viewpoint position or a line-of-sight direction according to a plurality of different depth values, and performing reprojection processing according to the plurality of different depth values. a reprojection unit that generates a rendered computer graphics image;
The reprojection unit obtains an addition value, an average value, and a median value between the past image and the reprojection-processed computer graphics image after converting the past image so as to match the current viewpoint position or line-of-sight direction. An image display system characterized by: 9. The image display system according to claim 7 , wherein reprojection processing by said image display device is performed asynchronously with rendering by said image generation device. a step of performing a reprojection process for transforming an image including depth value information so as to match a viewpoint position or line-of-sight direction according to a plurality of different depth values;
generating reprojected images according to a plurality of different depth values;
The image display method, wherein the step of generating generates the reprojection-processed image by overwriting the drawing result of a frame of a past image as an initial value. a step of performing a reprojection process for transforming an image including depth value information so as to match a viewpoint position or line-of-sight direction according to a plurality of different depth values;
generating reprojected images according to a plurality of different depth values;
The step of generating is characterized by obtaining an added value, an average value, and a median value between the past image that has been converted to match the current viewpoint position or line-of-sight direction and the reprojection processed image. image display method. A function of executing a reprojection process for transforming an image including depth value information so as to match the viewpoint position or line-of-sight direction according to a plurality of different depth values;
a function of generating an image that has been reprojected according to a plurality of different depth values;
The program, wherein the generating function generates the reprojection-processed image by overwriting the rendering result of a frame of a past image as an initial value. A function of executing a reprojection process for transforming an image including depth value information so as to match the viewpoint position or line-of-sight direction according to a plurality of different depth values;
a function of generating an image that has been reprojected according to a plurality of different depth values;
The function to generate is characterized in that the past image is converted to match the current viewpoint position or line-of-sight direction, and the added value, average value, and median value are obtained between the reprojected image and the image. A program to

Images