JP5755571B2 - Virtual viewpoint image generation device, virtual viewpoint image generation method, control program, recording medium, and stereoscopic display device - Google Patents

Description

  The present invention relates to a virtual viewpoint image generation device, a virtual viewpoint image generation method, a control program, a recording medium, and a virtual viewpoint image that generate a virtual viewpoint image viewed from an arbitrary viewpoint position from two or more images having different viewpoint positions. The present invention relates to a stereoscopic display device including a generation device.

  Conventionally, various forms have been proposed for stereoscopic video display methods. In particular, research on a multi-view autostereoscopic display system has become active. In the multi-view autostereoscopic display method, an image when viewed from a plurality of viewpoint positions (hereinafter referred to as a multi-view image) is generated with respect to a subject, and different directions are provided through a special lens or the like. This is a method of performing stereoscopic display by displaying the image toward the screen.

  The multi-view autostereoscopic display method has an advantage that a plurality of people can view a stereoscopic video without wearing special glasses. Furthermore, in the multi-viewpoint autostereoscopic display method, since multi-viewpoint images are displayed according to the respective viewpoint positions, the images change according to the viewing position, and a good motion parallax can be felt. Here, the motion parallax is a situation in which, when the position of the face is moved to the left or right, a movement corresponding to the distance occurs in the visible object, and is one of the causes for humans to feel a stereoscopic effect. For example, when looking on the outside of a window on a train, distant objects appear to move slowly, and nearby objects appear to move faster, which is an example of motion parallax. Therefore, the multi-viewpoint autostereoscopic display method has an advantage that good motion parallax can be obtained. Therefore, the use of the multi-viewpoint autostereoscopic display method is spreading.

  In the multi-view autostereoscopic display method, it is ideal to shoot the subject using multiple cameras to generate multi-view images, but there are restrictions such as the size of the imaging system and the setting of the optical axis of the camera. Therefore, the number of cameras that can be used practically is limited. In addition, the amount of information in the transmission and storage process increases in proportion to the increase in the number of cameras, which takes time and cost. Therefore, a technique for generating a multi-viewpoint image while suppressing the number of cameras is desired.

  Currently, techniques for generating virtual viewpoint images that should be seen at an arbitrary viewpoint position from a plurality of images with different viewpoint positions on the display side are being developed one after another.

  For example, Patent Document 1 discloses a method for generating a stereoscopic image from a plurality of input parallax images. Specifically, for example, a parallax value distribution image (or a depth image replaced from the parallax value distribution image) serving as an index of the warp amount in virtual viewpoint image generation is generated from input parallax images captured from two viewpoints. A virtual viewpoint image is generated at a virtual viewpoint on a straight line connecting the viewpoints of the input parallax image or on an extension line thereof. Then, a multi-viewpoint image composed of the input parallax image and / or the virtual viewpoint image is used to synthesize a stereoscopic image.

JP 2003-209858 A (published July 25, 2003)

  As described above, with the technique disclosed in Patent Document 1, it is necessary to generate a parallax value distribution image or a depth image in order to generate a virtual viewpoint image. Specifically, a corresponding point that represents the same subject portion as a point-to-point and pixel-to-pixel correspondence between input parallax images (for example, a left image and a right image) is extracted, and between the input parallax images The parallax value or depth amount for the target pixel is obtained from the difference in the coordinate values of the corresponding points.

  However, in order to extract the corresponding points, a matching area is searched from a partial area of a predetermined size centered on a predetermined point in the input parallax image, so that the search accuracy is the size of the partial area. Depends on. That is, the search accuracy decreases as the size of the partial region increases.

  The corresponding points may be obtained for all points in the input parallax image. However, when the image size is large, the time is enormous, which is not desirable. Therefore, particularly when the image size is large, there is a problem that it is difficult to calculate the parallax value or the depth amount with high accuracy. Therefore, when generating a virtual viewpoint image based on an inaccurate parallax value or depth amount, a part of the object is missing in the vicinity of the outline of the object in the virtual viewpoint image or extraneous around the outline of the object May have a special pattern. Such a state should not occur originally and leads to deterioration of image quality.

  Next, an example of the phenomenon in which the image quality deteriorates and one of the causes will be described with reference to FIGS. FIG. 9 is a diagram illustrating an example of a phenomenon in which deterioration in image quality occurs in the generation of a virtual viewpoint image according to the conventional technique. 9A shows the left image, FIG. 9B shows the right image, and FIG. 9C shows the calculation based on the left image shown in FIG. 9A. An example of the depth amount (depth information) is shown.

  As shown in FIG. 9, first, a case where a left image (FIG. 9A) and a right image (FIG. 9B) are input will be described. It is assumed that the L-shaped part is an object on the near side, and the other exists on the back side in the background. Also, with respect to the part A on the left image, the part C has the same background position on the right image, and the part B has the same position on the near side.

  Next, on the basis of the left image, a matching portion (matching block) is searched from the right image by looking in the horizontal direction. That is, a matching block search is performed. The depth amount is calculated based on the result of matching block search. When the obtained depth amount is shown in the figure, it often has a shape as shown in (c) of FIG. 9 (leading to deterioration in image quality). In FIG. 9C, the depth amount is represented by a color, and white is the depth amount portion determined to be the near side, and gray is the depth amount portion determined to be the back side.

  In (c) of FIG. 9, in the vicinity of the portion corresponding to the portion A in (a) of FIG. 9, a portion that is originally a background and the depth amount must be gray is partially white in an arc shape. Yes. The depth amounts of these portions are considerably close to the depth amount of the front object.

  On the other hand, the corner portion other than the above of the L-shaped object on the near side is cut into a round shape, and the depth amount (gray) on the far side originally becomes the depth amount on the near side. The depth amount of these portions is a value that is very close to the depth amount on the background side.

  Next, one of the causes of the phenomenon shown in FIG. 9C will be described with reference to FIG. FIG. 10 is an explanatory diagram for performing a matching block search from the right image shown in FIG. 9B based on the left image shown in FIG. 10A is an enlarged view of a portion A in the left image shown in FIG. 9A, and FIG. 10B is an enlarged view of a portion C in the right image shown in FIG. 9B. FIG. 10 (c) is an enlarged view of a portion B in the right image shown in FIG. 9 (b).

  Here, as shown in FIG. 10A, a range of m × m pixels and a range of n × n pixels are set in advance. The m × m pixel range is a search range when a matching block search is performed, and the n × n pixel range is an area to which a depth amount obtained based on the result of the matching block search is assigned. Details of the m × m pixel range and the n × n pixel range will be described later.

  As shown in FIG. 10A, when viewed in the search range (that is, a m × m pixel range) on the reference left image, the area of the object in front is larger than the area of the background portion. Yes.

  As shown in (a) and (b) of FIG. 10, in the matching block search, when the matching degree is compared at the position where the background part originally matches, the matching degree is minimized (good) for the background part. However, the degree of coincidence increases (becomes worse) in other parts. On the other hand, as shown in FIGS. 10A and 10C, when a matching block search is performed at a position where the front object matches, the matching degree is minimized (good) for the front portion, but otherwise The degree of coincidence increases (becomes worse) in the part. Overall, since the area of the front part is larger than the area of the background part, the degree of coincidence of the front part is better. Therefore, as a result of the matching block search, the block on the right image that matches the range of m × m pixels on the left image shown in FIG. 10A is located at the position where the object in the foreground matches in the right image. Become a block.

  Therefore, when the depth amount is obtained based on the result of the matching block search and is assigned to the range of n × n pixels on the left image shown in FIG. 10A, as shown in FIG. The depth amount of the arc-shaped portion corresponding to A (see FIG. 9A) is a value that is very close to the depth amount of the object in front.

  In addition to the reasons described above, it is relatively common that the position of the contour of the object does not coincide with the position of the contour of the depth amount for some reason.

  Another example of such deterioration in image quality will be described with reference to FIG. FIG. 11 is a diagram illustrating an example of a phenomenon in which image quality deterioration occurs when an input parallax image is converted into a virtual viewpoint image. FIG. 11A is a diagram illustrating a positional relationship when an object is viewed from above, and is a diagram illustrating a relationship between an original viewpoint position and a converted viewpoint position. FIG. 11B is a diagram illustrating deterioration in image quality when an input parallax image is converted into a virtual viewpoint image.

  First, at the position of the original viewpoint shown in FIG. 11A, the image is taken by a camera, and an image as shown in the left figure of FIG. 11B (hereinafter referred to as “original image”) is acquired. . Next, an image (hereinafter referred to as “image after conversion”) when viewed from a different viewpoint from the original image (after-conversion viewpoint) is created. Specifically, first, a depth amount in each pixel of the original image is obtained, and an interpolation vector is obtained from the depth amount and the converted viewpoint position. Next, as shown in FIG. 11B, the converted image is generated by moving each pixel of the original image according to the obtained interpolation vector.

  Note that the depth amount and the interpolation vector calculation require at least two original images, but details thereof will be described later, and the description thereof is omitted here.

  However, as described above, it is difficult to accurately calculate the depth amount. Therefore, when the depth amount is calculated, a depth amount close to the depth amount outside the object contour is calculated even though the depth amount is within the object contour. The amount of depth goes inside the contour. As a result, the interpolation vector of the pixel in that portion is different from the interpolation vector of the object, and the converted image becomes an image in which a part of the object is missing.

  On the other hand, when calculating the depth amount, if a depth amount close to the depth amount inside the contour of the object is calculated despite being outside the contour of the object, The depth amount protrudes outside the contour. As a result, the pixels around the object move together with the object, and the converted image becomes an image with an extra pattern around the object. As shown in the right diagram of FIG. 11B, in the converted image, the lower side of the foreground object is missing, while the upper side moves part of the background together with the foreground object.

  As described above, since it is difficult to accurately calculate the parallax value or the depth amount in the related art, there is a problem in that the image quality deteriorates when the input parallax image is converted into the virtual viewpoint image.

  In Patent Document 1, a pixel in an unexpected region is selected by selecting a specific range in the image for an input parallax image obtained by capturing a complex scene and adjusting only the parallax value or the depth amount of the specific region. Although it is described that the parallax value or depth amount is not changed, symptoms (deterioration of image quality) that occur when the parallax value or depth amount does not match the input parallax image accurately and countermeasures Is not listed.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a virtual viewpoint image generation apparatus, a virtual viewpoint that can reduce deterioration in image quality that occurs when an input parallax image is converted into a virtual viewpoint image. A viewpoint image generation method, a control program, a recording medium, and a stereoscopic display device are provided.

  In order to solve the above-described problem, a virtual viewpoint image generation device according to the present invention is a virtual viewpoint image generation device that generates a new viewpoint image that is an image corresponding to a virtual viewpoint position, and two or more at different viewpoint positions. An acquisition unit for acquiring a plurality of images, and selecting one image as a reference image from among the images acquired by the acquisition unit, and generating the new viewpoint image from the acquired plurality of images And a new viewpoint for generating the new viewpoint image by moving each pixel of the reference image based on the movement amount calculated by the movement amount calculator. An attention generating block which is a target pixel or a set of a plurality of pixels of the new viewpoint image generated by the image generation unit and the new viewpoint image generation unit is specified, and the target pixel or the target block is included in a predetermined region. Then, the variation of the movement amount in each pixel is obtained, and a smoothing process is performed on the predetermined area according to the degree of the variation, and the value of the target pixel or the target block is set to the smooth value obtained by the smoothing process. A smoothing processing unit to be replaced is provided.

  In order to solve the above problems, a virtual viewpoint image generation method according to the present invention is a virtual viewpoint image generation method by a virtual viewpoint image generation device that generates a new viewpoint image that is an image according to a virtual viewpoint position. An acquisition step of acquiring two or more images at different viewpoint positions, and one image is selected as a reference image from the images acquired in the acquisition step, and a new viewpoint is selected from the acquired images A moving amount calculating step for calculating a moving amount of each pixel of the reference image to generate an image, and moving each pixel of the reference image based on the moving amount calculated in the moving amount calculating step; , A new viewpoint image generation step for generating the new viewpoint image, and a target block that is a target pixel or a set of a plurality of pixels of the new viewpoint image generated in the new viewpoint image generation step In the predetermined area including the target pixel or the target block, a variation in the movement amount in each pixel is obtained, and a smoothing process is performed on the predetermined area according to the degree of the variation, and the target pixel or the target block And a smoothing step that replaces the value with a smooth value obtained by the smoothing process.

  In the new viewpoint image, the symptom that a part of the object is missing or an extra pattern is added around the outline of the object is that the amount of movement calculated for each pixel of the same subject part of the multiple images follows the outline of the object This is likely to occur near the contour of the object. That is, since the object and the background are adjacent to each other in the vicinity of the contour of the object, the difference in movement amount tends to increase with the contour of the object as a boundary. This difference induces an erroneous determination as to whether the pixel of interest or the block of interest belongs to the object or the background. The symptom becomes more conspicuous as the difference (variation) in the movement amount for each pixel is larger.

  According to said structure, the smoothing process with respect to the said area | region can be performed in the area | region containing the pixel with the big variation | change_quantity of movement amount which is an area | region where said symptom is conspicuous. Therefore, an image (particularly, a contour shift) in an area in which the above-described symptom is conspicuous is blurred by the smoothing process and becomes inconspicuous, so that it is possible to reduce deterioration in image quality.

  If there are at least two images at different viewpoint positions, the movement amount calculation unit can obtain the depth information of the part by comparing the positions of the same part of the same subject on each image. it can. In addition, the movement amount calculation unit calculates the amount of movement required to move the pixels of the part corresponding to a new viewpoint position different from the different viewpoint position, and the amount of change of the viewpoint position and the obtained depth information. You can get it using.

  In addition, when an attention block that is a set of a plurality of pixels is used instead of a single attention pixel, there is an effect that the processing speed can be improved when the image size is large (the number of pixels is large). .

  In the virtual viewpoint image generation device, it is preferable that the smoothing processing unit determines a smoothing strength in accordance with the degree of variation, and performs smoothing in accordance with the strength.

  According to the above configuration, the pixel value can be smoothed according to the degree of conspicuous deterioration in image quality. That is, when image quality deterioration occurs, the value of the target pixel or target block included in the area where the image quality deterioration has occurred is not uniformly smoothed. Can be changed. Therefore, there is an effect that image clarity can be kept to the maximum and deterioration of image quality can be reduced.

  In the virtual viewpoint image generation device, it is preferable that the smoothing processing unit obtains an absolute value of a difference between the maximum value and the minimum value of the movement amount as the degree of variation.

  According to said structure, the dispersion | variation degree of the said moving amount can be calculated | required with the absolute value of the difference of the maximum value of moving amount, and the minimum value. Therefore, compared to a case where the degree of variation is obtained by calculation such as standard deviation, no complicated calculation is required and the processing speed is increased. Further, when the smoothing processing unit is realized by a circuit, there is an effect that the circuit scale is reduced because fewer arithmetic units and the like are required.

  The virtual viewpoint image generation apparatus may be realized by a computer. In this case, the virtual viewpoint image generation apparatus is operated by the computer by causing the computer to operate as each unit of the virtual viewpoint image generation apparatus. A control program to be realized and a computer-readable recording medium on which the control program is recorded also fall within the scope of the present invention.

  Further, if the stereoscopic display device includes the virtual viewpoint image generation device and performs stereoscopic display using the virtual viewpoint image generated by the virtual viewpoint image generation device, the same effects as the virtual viewpoint image generation device can be obtained. .

  As described above, the present invention identifies a target pixel of the generated new viewpoint image or a target block that is a set of a plurality of pixels, and moves in each pixel in a predetermined region including the target pixel or target block. A smoothing process is performed on the area in accordance with the degree of variation.

  Therefore, an image in an area where a symptom in which a part of the object is missing or an extra pattern around the outline of the object is easily noticeable is blurred by the above-described smoothing process and becomes inconspicuous. There is an effect that can be.

It is a block diagram which shows the principal part structure of the virtual viewpoint image generation apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the flow of a process after the virtual viewpoint image generation apparatus which concerns on one Embodiment of this invention acquires a left image and a right image, and produces | generates a virtual viewpoint image. FIG. 6 is a diagram illustrating an example of dividing a left image in an embodiment of the present invention. It is a figure explaining the example which calculates the amount of depths by a coincidence block in the virtual viewpoint image generation device concerning one embodiment of the present invention. In one Embodiment of this invention, it is a figure which shows the value showing a viewpoint position. In the virtual viewpoint image generation device according to an embodiment of the present invention, a change in an image when a virtual viewpoint image is generated by generating a new viewpoint image by moving pixels in the left image and performing smoothing. FIG. It is a figure which shows an example of the glasses-type three-dimensional display apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the multiview autostereoscopic display apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the phenomenon which image quality deterioration generate | occur | produces in the generation of the virtual viewpoint image by a prior art. FIGS. 9A and 9B are explanatory diagrams for performing matching block search from the right image based on the left image, FIG. 9A is an enlarged view of a portion A in the left image shown in FIG. 9B, and FIG. It is an enlarged view of the part A in the right image shown to (a). It is a figure which shows an example of the phenomenon in which deterioration of an image quality generate | occur | produces, when converting from an input parallax image to a virtual viewpoint image by a prior art.

Embodiment 1
The virtual viewpoint image generation device 1 according to an embodiment of the present invention will be described below with reference to FIGS.

(Outline of processing)
First, an outline of processing in which the virtual viewpoint image generation device 1 according to the present embodiment reduces deterioration in image quality will be described.

  In the new viewpoint image, the symptom that a part of the object is missing or an extra pattern is added around the outline of the object is that the interpolation vector (movement amount) obtained for each pixel of the same subject part in multiple images is It is likely to occur near the contour of the object because it is not along the contour of the object. That is, since the object and the background are adjacent to each other in the vicinity of the contour of the object, the difference between the interpolation vectors tends to increase with the contour of the object as a boundary. This difference induces an erroneous determination as to whether the pixel of interest or the block of interest belongs to the object or the background. Therefore, the above-mentioned symptom becomes more conspicuous as the difference in the interpolation vector for each pixel increases.

  Here, the interpolation vector indicates a movement amount for each pixel in the input parallax image in order to generate a virtual viewpoint image. That is, the interpolation vector indicates how much each pixel on the input parallax image is generated when an image viewed from a virtual viewpoint position (hereinafter sometimes referred to as “new viewpoint position”) is generated from the input parallax image. This parameter indicates whether it should be moved.

  The input parallax images are the left image corresponding to the viewpoint position of the left eye or the left camera and the right eye or the right camera when the same object is viewed with both eyes or taken with two cameras. It refers to at least one of the right images corresponding to the viewpoint position.

  Therefore, the virtual viewpoint image generation device 1 according to the present invention interpolates a converted image converted from an input parallax image (hereinafter sometimes referred to as “new viewpoint image”) in a predetermined region. When the variation of the vector (for example, the absolute value of the difference between the maximum value and the minimum value) is large, the region of the image is blurred by executing the smoothing process on the region, thereby reducing the deterioration of the image quality.

  Specifically, the virtual viewpoint image generation device 1 first obtains an interpolation vector for each pixel when creating a new viewpoint image from the depth amount and the new viewpoint position, and stores the obtained interpolation vector for each pixel.

  Then, a new viewpoint image is generated by moving each pixel in the input parallax image according to the obtained interpolation vector. Next, with respect to the new viewpoint image, an interpolation vector in a predetermined area centered on a certain pixel is referred to. Among them, when the variation of the interpolation vector (for example, the absolute value of the difference between the minimum value and the maximum value) is equal to or greater than a predetermined threshold value, a smoothed value (for example, an average value) of each pixel value in the region Value) to obtain a new pixel value of the target pixel included in the region. Note that the target pixel may be one pixel, but a target block that is a set of a plurality of pixels may be treated as the target pixel.

  The process of examining the variation of the interpolation vector and smoothing as necessary is repeated for all the pixels constituting the new viewpoint image.

  Finally, the virtual viewpoint image is generated by changing the pixel values of some pixels of the new viewpoint image to the smoothed values.

  By the above processing, in the generated virtual viewpoint image, the image of the region where the variation of the interpolation vector is large is blurred by the smoothing so that the above-mentioned symptoms of the deterioration of the image quality become inconspicuous and the deterioration of the image quality is reduced. can do.

(Configuration of Virtual Viewpoint Image Generation Device 1)
Below, the principal part structure of the virtual viewpoint image generation apparatus 1 is demonstrated based on FIG.

  FIG. 1 is a block diagram showing a main configuration of the virtual viewpoint image generation device 1. As illustrated in FIG. 1, the virtual viewpoint image generation device 1 includes an image acquisition unit 10 (acquisition unit), a matching block search unit 20, a depth amount calculation unit 30, a new viewpoint position acquisition unit 40, and an interpolation vector calculation unit 50 ( Movement amount calculation unit), new viewpoint image generation unit 60, and boundary smoothing unit 70 (smoothing processing unit).

  A specific example of the processing performed by each unit (10 to 70) will be described in detail later, so the outline of each unit (10 to 70) will be described here.

  The image acquisition unit 10 acquires two or more images at different viewpoint positions, for example, a left image and a right image that are input parallax images. Here, the left image and the right image are obtained by shooting the subject from two viewpoint positions on a line parallel to the horizontal direction of the shooting plane of the subject. The acquisition method of the left image and the right image is not particularly limited, and for example, an image captured by a binocular camera (not shown) built in the virtual viewpoint image generation device 1 may be acquired, or communication You may acquire from a delivery server by a means (not shown). The image acquisition unit 10 transmits the acquired left image and right image to the matching block search unit 20.

  The matching block search unit 20 searches for a matching portion (matching block) in the horizontal direction from the left image and the right image received from the image acquisition unit 10. Specifically, for example, the left image is set as the reference image. First, the matching block search unit 20 divides the left image into a plurality of blocks, and searches the right image for blocks that match the divided blocks in the horizontal direction. Then, the matching block search unit 20 transmits the upper left coordinate value of the matching block in each of the left image and the right image to the depth amount calculation unit 30.

  The depth amount calculation unit 30 calculates a difference from the coordinate values in each left image and right image received from the matching block search unit 20 to obtain a depth amount. Then, the depth amount calculation unit 30 transmits the obtained depth amount to the interpolation vector calculation unit 50.

  The new viewpoint position acquisition unit 40 acquires a new viewpoint position of the new viewpoint image to be generated. The method for acquiring the new viewpoint position is not particularly limited. For example, a predetermined viewpoint position may be acquired from a storage unit (not shown) provided in the virtual viewpoint image generation device 1 or from the user. You may acquire what was designated by the operation part (not shown). Then, the new viewpoint position acquisition unit 40 transmits the acquired new viewpoint position to the interpolation vector calculation unit 50.

  The interpolation vector calculation unit 50 generates a new viewpoint image at the new viewpoint position from the received depth amount and the new viewpoint position, that is, a movement amount for each pixel in the left image as the reference image, that is, an interpolation vector. Is what you want. Then, the interpolation vector calculation unit 50 transmits the obtained interpolation vector for each pixel to the new viewpoint image generation unit 60.

  The new viewpoint image generation unit 60 generates a new viewpoint image when viewed from the new viewpoint position. Specifically, the new viewpoint image generation unit 60 acquires a left image as a reference image from the image acquisition unit 10, and based on the interpolation vector for each pixel received from the interpolation vector calculation unit 50, the left image Each pixel in is moved to generate a new viewpoint image. Then, the new viewpoint image generation unit 60 transmits the generated new viewpoint image together with the interpolation vector of each pixel to the boundary smoothing unit 70.

  When the difference between the maximum value and the minimum value of the interpolation vector at each pixel in the predetermined area is large, the boundary smoothing unit 70 executes a smoothing process on the predetermined area, and calculates the value of the target pixel or target block. The virtual viewpoint image is generated by replacing the smoothed value obtained by the smoothing process.

(Process flow)
Hereinafter, processing details of each unit (10 to 70) of the virtual viewpoint image generation device 1 will be described with reference to FIGS.

  First, the flow of processing of the virtual viewpoint image generation device 1 will be described with reference to FIG. FIG. 2 is a flowchart showing a flow of processing from when the virtual viewpoint image generation device 1 acquires a left image and a right image to when a virtual viewpoint image is generated.

<Match block search>
As shown in FIG. 2, first, the image acquisition unit 10 acquires a left image and a right image (S101).

  Then, the matching block search unit 20 searches for matching blocks in each of the left image and the right image (S102).

  Specifically, first, the matching block search unit 20 divides the left image into rectangular regions (n) of n × n pixels (n is an integer of 1 or more) as shown in FIG. FIG. 3 is a diagram illustrating an example of dividing the left image. Then, the matching block search unit 20 also sets a rectangular area (n) of n × n pixels having the same horizontal position and vertical position as one of the divided areas on the right image.

  Next, the matching block search unit 20 sets a rectangular area (m) of m × m pixels centered on a rectangular area (n) of n × n pixels of each of the left image and the right image. However, m is an integer greater than or equal to n.

  Hereinafter, the reason for setting the rectangular area (m) of m × m pixels separately from the rectangular area (n) of n × n pixels will be described.

  It is considered that the accuracy of matching increases as the size of the rectangular area when the matching block search unit 20 searches for a matching block increases to a certain size. This is because as the size is larger, the amount of information of the pattern is larger, and the degree of matching is confirmed with respect to the larger amount of information, resulting in higher accuracy of matching. Therefore, it is advantageous that the size of the rectangle when searching is large.

  However, simply increasing the size of the rectangular area increases the granularity of the resulting depth. For example, when the size of the rectangular area is set to a large size such as 64 × 64 pixels (hereinafter, referred to as a rectangular area (64)), the resulting depth amount also becomes the same depth amount in the rectangular area (64). Produces a rough result like a rattling mosaic image in units of 64 × 64 pixels. In this way, no matter how accurate the matching block search is, the amount of depth along the pattern of the input image is not fine.

  In order to achieve both of the above two points (search accuracy and image clarity), the matching block search itself is performed with m × m pixels of a large size, and the depth amount thus obtained is a small size of n × m. Reflected in n pixels. As a result, the accuracy of the search is good, and the resulting depth amount can be finely matched to the pattern of the input image.

  The size of the smaller rectangular area (n) is ideally 1 × 1 pixel. However, there is no problem when it is performed by software. However, when the circuit is assembled and performed by hardware, there is a possibility that the number of circuits may increase considerably. Therefore, n is preferably an integer of 1 or more.

  Then, the matching block search unit 20 calculates the degree of matching between the m × m pixel rectangular area (m) on the left image and the m × m pixel rectangular area (m) on the right image. The degree of coincidence is the difference between each pixel value of m × m pixels included in the rectangular area (m) on the left image and each pixel value of m × m pixels included in the rectangular area (m) on the right image. It is a value indicating the degree of. The degree of coincidence is obtained as a sum of absolute values of differences between pixel values in the rectangular area (m) as in the following formula (1).

  Here, R and L indicate the pixel value of each pixel included in the rectangular area (m) of the right image and the rectangular area (m) of the left image, respectively, and (x, y) indicate the coordinate value of the pixel. For example, R (x, y) is a pixel value of a pixel located at coordinates (x, y) in the right image.

  According to Expression (1), the closer the calculated matching degree is to 0, the higher the matching degree.

  Further, the matching block search unit 20 shifts each rectangular area (n) while shifting the rectangular area (n) of n × n pixels on the right image to the right by a predetermined shift amount of one pixel or more in the horizontal direction. ) And a rectangular area (m) of m × m pixels is newly acquired, and the degree of coincidence with the rectangular area (m) in the acquired left image is calculated by the above method.

  The shift amount may be a value of one pixel or more. However, when the calculated depth amount is highly accurate, it is preferable to set a smaller value (for example, one pixel).

  The reason will be described by taking the case where the left image is the reference in the matching block search as an example. When the left image is a reference, the entire screen of the left image is divided every n × n pixels. Therefore, the left image is processed n pixels at a time. On the other hand, in the right image, a rectangular area (m) of m × m pixels having the same size as the rectangular area (m) on the left image is set and compared with the rectangular area (m) on the left image. Here, the number of pixels for shifting the rectangular area (m) on the right image can be arbitrarily set, but the fineness (accuracy) of the depth amount determined by the shift amount varies. In other words, the smaller the shift amount, the higher the accuracy of the depth amount.

  Therefore, in order to improve both the processing speed of the matching block search and the accuracy of the depth amount to be obtained, the rectangular area of n × n pixels (and m × m pixels) is set coarsely (largely), and the above It is preferable to set the shift amount finely (smallly).

Next, the matching block search unit 20 has the rectangular area (n) included in the rectangular area (m) on the right image having the smallest degree of matching, and the rectangular area (m) included in the rectangular area (m) on the left image. The area (n) is determined as a matching block. The matching block search unit 20 then determines the upper left coordinate value (x _R , y) of the rectangular area (n) in the right image determined as the matching block and the upper left coordinate value (x) of the rectangular area (n) in the left image. _L , y) are stored in association with the matching block correspondence.

Alternatively, the rectangular area (m) having the smallest matching degree is determined as a matching block, and the upper left coordinate value (x _R , y) of the rectangular area (m) in the right image determined to be the matching block, and the rectangular area in the left image The upper left coordinate value (x _L , y) of (m) may be stored in association with the matching block correspondence.

  Then, the matching block search unit 20 obtains a matching block correspondence relationship with the rectangular area (n) on the right image in the same manner as described above for all the rectangular areas (n) on the left image. Finally, the matching block search unit 20 transmits the stored matching block correspondence relationship to the depth amount calculation unit 30.

<Depth calculation>
Next, the depth amount calculation unit 30 calculates the depth amount from the upper left coordinate value of the rectangular area (n) of the left image and the upper left coordinate value of the rectangular area (n) of the right image that are in the matching block correspondence relationship. (S103). Specifically, a description will be given based on FIG. FIG. 4 is a diagram illustrating an example in which the depth amount is calculated using the matching block. 4A is a diagram showing the right image, and FIG. 4B is a diagram showing the left image.

For example, the upper left coordinate value of the rectangular area (n) of the left image in the matching block correspondence relationship is (x _L , y), and the upper left coordinate value of the rectangular area (n) of the right image is (x _R , y). To do. As shown in FIG. 4, the depth amount calculation unit 30 calculates the depth amount by the following equation (2).

Depth amount = x _R −x _L (2)
Next, the depth amount calculation unit 30 assigns the obtained depth amount to all the pixels included in the rectangular region (n) of interest in the left image, that is, each of n × n pixels.

  Then, the depth amount calculation unit 30 assigns the depth amount to all the pixels on the left image by obtaining the depth amount for the upper left coordinate values of all the rectangular areas (n) on the left image. Then, the depth amount calculation unit 30 transmits the depth amount assigned to each pixel in the left image (hereinafter sometimes referred to as “depth amount of each pixel”) to the interpolation vector calculation unit 50.

  In the above example, the rectangular area (n) and the rectangular area (m) are squares, but are not limited thereto, and may be rectangular. Furthermore, the size of the rectangular area (m) may be the same size as the rectangular area (n).

<Interpolation vector calculation>
Next, the new viewpoint position acquisition unit 40 acquires a new viewpoint position of the new viewpoint image to be generated (S104). The value representing the new viewpoint position will be described with reference to FIG. FIG. 5 is a diagram illustrating values representing viewpoint positions.

  As shown in FIG. 5, the new viewpoint position corresponding to the viewpoint position of the left image is set to 0.0, and the new viewpoint position corresponding to the viewpoint position of the right image is set to 1.0. The other new viewpoint positions are represented by relative values with respect to the viewpoint positions of the left image and the right image.

  For example, as shown in FIG. 5, when creating a new viewpoint image viewed from an intermediate position between the left image and the right image, the new viewpoint position is set to 0.5 and viewed from the right side of the right image. When creating an image, the new viewpoint position is set to 1.5. Furthermore, when it is desired to create an image viewed from the left side of the left image, the new viewpoint position is set to -0.5.

  Then, the new viewpoint position acquisition unit 40 transmits the acquired new viewpoint position to the interpolation vector calculation unit 50.

  Next, the interpolation vector calculation unit 50 calculates the inner position of each pixel of the left image from the depth amount of each pixel acquired from the depth amount search unit 30 and the new viewpoint position acquired from the new viewpoint position acquisition unit 40. An insertion vector is obtained (S105). Specifically, the interpolation vector calculation unit 50 obtains an interpolation vector for each pixel based on the following equation (3).

(Interpolation vector) = (depth amount) × (new viewpoint position) (3)
Then, the interpolation vector calculation unit 50 transmits the interpolation vector of each pixel obtained by the equation (3) to the new viewpoint image generation unit 60.

<New viewpoint image generation>
Next, the new viewpoint image generation unit 60 acquires the left image from the image acquisition unit 10. In addition, the new viewpoint image generation unit 60 acquires the interpolation vector of each pixel in the left image from the interpolation vector calculation unit 50.

  Subsequently, the new viewpoint image generation unit 60 generates a new viewpoint image when viewed from the new viewpoint position (S106). Specifically, for example, let one pixel of interest in the left image be (x, y), and let the interpolation vector at that pixel of interest be V. The new viewpoint image generation unit 60 moves the noted pixel (x, y) to the position of (x + V, y). Then, as described above, the new viewpoint image generation unit 60 performs the above movement for all the pixels in the left image. Therefore, a new viewpoint image when viewed from the new viewpoint position is generated. The generated new viewpoint image will be described with reference to FIG.

  FIG. 6 is a diagram illustrating changes in an image when a virtual viewpoint image is generated by generating a new viewpoint image by moving pixels in the left image and performing smoothing. (A) of FIG. 6 is a figure which shows the left image as an input parallax image. FIG. 6B is a diagram illustrating a new viewpoint image generated by moving a pixel in the left image. For example, the left image is viewed from a new viewpoint position of −0.5 illustrated in FIG. In this case, a new viewpoint image is shown. Moreover, (c) of FIG. 6 is a figure which shows the object area | region which performs the below-mentioned smoothing process, in order to produce | generate a virtual viewpoint image.

  As shown in FIG. 6, after each pixel in the left image shown in FIG. 6A is moved by the interpolation vector, a new viewpoint image shown in FIG. 6B is generated.

  Then, the new viewpoint image generation unit 60 moves the value of the interpolation vector from the original pixel to the moved pixel simultaneously with the movement of the pixel. That is, the moved pixel and the interpolation vector are stored in association with each other. This interpolation vector is required for smoothing processing by the boundary smoothing unit 70 described later.

  Then, the new viewpoint image generation unit 60 transmits the generated new viewpoint image and the interpolation vector of each pixel to the boundary smoothing unit 70.

  In addition, when moving the pixel in the left image, a plurality of pixels may move to the same coordinate position. In that case, it is preferable to move by referring to the depth amount in each original pixel. Specifically, the movement of the pixel having the smallest depth amount is given priority. This means that the smaller the depth amount, the closer the pixel corresponding to that depth amount is, and when multiple pixels corresponding to different depth amounts overlap, the back pixel is replaced by the previous pixel. Because it should be hidden.

  Here, instead of referring to the interpolation vector, it is necessary to confirm the depth amount obtained before that. This is because the interpolation vector is a value in which the viewpoint position is added in addition to the depth amount. Therefore, when the positive / negative relationship between the depth amount and the viewpoint position is reversed, the relationship between the size of the interpolation vector and the depth amount is reversed. Because it will do.

For example,
(A) Front object depth = 20
(B) Depth amount of back background = 100
If the new viewpoint position is -1.5 on the left side of the left image,
(A ′) Interpolation vector of front object = −30 (= 20 × −1.5)
(B ′) Backside background interpolation vector = −150 (= 100 × −1.5)
It becomes.

  In this case, it can be seen that the magnitude relationship between the depth amounts is opposite to the magnitude relationship between the interpolation vectors.

  Next, the reason why the smaller the depth amount is, the closer the pixel corresponding to the depth amount is.

Before that, first, a position where a stereoscopic image is formed on the screen will be described. The position at which the stereoscopic image is formed is determined by the position where the same object is displayed on the left image and the right image. For example,
(A) When the position on the right image is on the right side compared to the position on the left image, the image is formed on the back side from the screen.

  (B) When the position on the left image and the position on the right image are the same, an image is formed at the position of the screen.

  (C) When the position on the right image is on the left side compared with the position on the left image, the image is formed on the near side of the screen. That is, it appears to jump out of the screen.

  In the present embodiment, after the matching block search, the depth amount is calculated by the above equation (2). That is, the depth amount is calculated by calculating the depth amount = (position on the right image) − (position on the left image).

  Therefore, in the case of (A) above, when the depth amount is calculated using the above equation (2), the depth amount becomes a positive value, and the larger the depth amount, the deeper the image is formed. In the case of (C), when the depth amount is calculated using the above equation (2), the depth amount becomes a negative value. The smaller the depth amount, the larger the pop-out amount, and the image is formed on the near side. . Thus, in this embodiment, the depth amount on the back side is large, and the depth amount on the near side is small.

  This is because the calculation is performed by subtracting the position on the left image from the position on the right image, and if the subtraction is determined in the reverse direction, the depth relationship is reversed.

  The direction of the subtraction is not particularly limited, and may be determined by a program or circuit design specification.

<Smoothing>
Next, when the difference between the interpolation vectors in the predetermined region is large, the boundary smoothing unit 70 performs a smoothing process on the region (S107).

  Specifically, the boundary smoothing unit 70 applies a certain area (for example, a 5 × 5 pixel rectangle) centered on a certain target pixel (one pixel) to the new viewpoint image generated by the new viewpoint image generation unit 60. Reference is made to the interpolation vector at each pixel in the region.

  Then, the boundary smoothing unit 70, when the variation of the interpolation vector in the region exceeds the reference amount, for example, the absolute value of the difference between the minimum value and the maximum value of the interpolation vector is greater than or equal to a predetermined threshold value In this case, a value obtained by smoothing the pixel value in the region (for example, an average value of each pixel in a rectangular region of 5 × 5 pixels) is obtained, and the value is set as a new pixel value of the pixel of interest (one pixel). . These processes are performed for all the pixels of the new viewpoint image, and the value of the pixel to be changed in the new viewpoint image is changed to a new pixel value to generate a virtual viewpoint image.

  Here, the target pixel is described as one pixel, but the target pixel is not limited to one pixel, and may be a target block that is a set of a plurality of pixels. Therefore, particularly when the image size is large (the number of pixels is large), the processing speed can be improved.

  Further, the smoothing process is not particularly limited, and for example, a general low-pass filter can be used. In smoothing processing that blurs such fine patterns to some extent, in the new viewpoint image, there is a symptom that the outline of the object is fused with the background, part of the object is missing, or an extra pattern is added around the outline of the object. It is less noticeable and has the effect of reducing image quality deterioration.

  As a result, as shown in FIG. 6C, a portion where the difference in the interpolation vector is large in the vicinity of the contour of the object in the image is blurred, and deterioration in image quality can be reduced.

[Modification (1)]
In the above, an example has been described in which the virtual viewpoint image generation device 1 performs the smoothing process on a region when the difference between the interpolation vectors in the predetermined region is large. As described above, the larger the difference between the interpolation vectors, the more conspicuous the image quality is. Therefore, the virtual viewpoint image generation device 1 may change the strength of smoothing in proportion to the absolute value of the difference between the interpolation vectors in a predetermined region. That is, the boundary smoothing unit 70 included in the virtual viewpoint image generation device 1 is a strength for smoothing the region based on the absolute value of the difference between the minimum value and the maximum value of the interpolation vector in the predetermined region. Adjust.

  According to the above configuration, the strength of smoothing can be changed according to the conspicuous degree of deterioration in image quality. That is, when image quality deterioration occurs, the area where the image quality deterioration has occurred is not uniformly smoothed, and the smoothing strength is increased for each part in the area according to the degree of conspicuous deterioration in image quality in the area. Can be changed. Therefore, there is an effect that the sharpness of the image can be kept to the maximum and the deterioration of the image quality can be reduced.

  In the above, the virtual viewpoint image generation device 1 has been described with respect to an example in which the left image and the right image are used as the input parallax image. However, the input parallax image is not limited to the left image and the right image, and two or more images are used. Any image viewed from a plurality of viewpoint positions may be used.

[Embodiment 2]
A virtual viewpoint image generation apparatus 1 and a new viewpoint position detection unit that detects positions of left and right eyes of dedicated glasses for stereoscopic viewing as two new viewpoint positions. Regarding the glasses-type stereoscopic display device 2 that generates two virtual viewpoint images corresponding to the two new viewpoint positions detected by the detection unit and displays the two virtual viewpoint images generated by the virtual viewpoint image generation device 1. Are included in the technical scope of the present invention.

  Hereinafter, an example of the glasses-type stereoscopic display device 2 will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of the glasses-type stereoscopic display device 2, and when the glasses-type stereoscopic display device 2 is viewed from above, the viewer views the glasses with dedicated glasses at two different viewpoint positions. Indicates.

  In the current 3D television, the left image and the right image are the same regardless of where they are viewed from the TV screen.

  On the other hand, as shown in FIG. 7, the glasses-type stereoscopic display device 2 according to the present embodiment detects the position of the eye of the dedicated glasses or the viewer, and the position of the glasses or the eye (new viewpoint position). Accordingly, the virtual viewpoint image for the left eye and the virtual viewpoint image for the right eye are generated and displayed by the virtual viewpoint image generation device 1, respectively. Therefore, even when the viewer moves his / her head, a stereoscopic image corresponding to the position of the head (new viewpoint position) can be provided to the viewer.

  In the above description, the glasses-type stereoscopic display device 2 of the present embodiment has been described with respect to detecting the position of the conventional dedicated glasses attached to a commercially available 3D-compatible television or the like or the viewer's eyes. In order to improve the detection accuracy, it is preferable to mark the conventional dedicated glasses for detection. By doing so, it is possible to improve the detection accuracy and eliminate the need for a complicated detection method and easily perform detection.

  In the above description, the glasses-type stereoscopic display device 2 detects the position of the dedicated glasses or the viewer's eyes. However, the present invention is not limited to this. For example, the dedicated glasses have their own position as the new viewpoint position. It may be detected and transmitted to the glasses-type stereoscopic display device 2. Special glasses as described above are also included in the technical scope of the present invention.

[Embodiment 3]
A virtual viewpoint image generation device 1 is provided. The virtual viewpoint image generation device 1 generates a plurality of virtual viewpoint images corresponding to a plurality of predetermined new viewpoint positions, and the virtual viewpoint image generation device 1 generates the virtual viewpoint image generation device 1. The multi-view autostereoscopic display device 3 that performs multi-view autostereoscopic display by displaying the virtual viewpoint image in the direction corresponding to each viewpoint position is also included in the technical scope of the present invention.

  Hereinafter, an example of the multi-view autostereoscopic display device 3 will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of the multi-viewpoint autostereoscopic display device 3, and nine virtual viewpoint images at nine different viewpoint positions are displayed when the multi-viewpoint autostereoscopic display device 3 is viewed from above. Show the state.

  Specifically, as shown in FIG. 8, the multi-viewpoint autostereoscopic display device 3 can output virtual viewpoint images having different viewpoint positions in nine different directions (viewpoint positions). The virtual viewpoint image created by the virtual viewpoint image generation device 1 is output according to each viewpoint position. The viewer receives one of the nine virtual viewpoint images with the left eye and the other with the right eye. Since the images received by the left and right eyes are images that are appropriately created in consideration of the shift of the viewpoint position, the viewer can enjoy the effect that the image can be felt in three dimensions due to binocular parallax.

  Note that the plurality of new viewpoint positions determined in advance may be stored in the storage unit at the time of shipment, or may be input in advance by the user.

[Modification (2)]
A stereoscopic display device that includes the virtual viewpoint image generation device 1, generates a virtual viewpoint image from an input parallax image, and performs stereoscopic display using the virtual viewpoint image is also included in the technical scope of the present invention.

  Further, the stereoscopic display device including the virtual viewpoint image generation device 1 may be realized by incorporating the virtual viewpoint image generation device 1 in the stereoscopic display device, or the stereoscopic display device may be the virtual display device. It may be realized by being separated from the viewpoint image generation device 1 and connected to the virtual viewpoint image generation device 1 by wired or wireless communication means.

[Supplement]
The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  Finally, each block of the virtual viewpoint image generation device 1, in particular, the matching block search unit 20, the depth amount calculation unit 30, the interpolation vector calculation unit 50, the new viewpoint image generation unit 60, and the boundary smoothing unit 70 are performed by hardware logic. You may comprise, and may implement | achieve by software using CPU as follows.

  That is, the virtual viewpoint image generation device 1 includes a CPU that executes instructions of a control program that realizes each function, a ROM that stores the program, a RAM that expands the program, a memory that stores the program and various data, and the like. A device (recording medium) is provided. An object of the present invention is a recording in which the program code (execution format program, intermediate code program, source program) of the control program of the virtual viewpoint image generation device 1 which is software for realizing the above-described functions is recorded so as to be readable by a computer. This can also be achieved by supplying a medium to the virtual viewpoint image generating apparatus 1 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The virtual viewpoint image generation device 1 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Further, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention can be used for an apparatus that generates a virtual viewpoint image from an input parallax image. Moreover, it can utilize for the apparatus which performs a three-dimensional display using a multi-viewpoint image. In particular, it can be suitably used for 3D televisions, 3D cameras, smartphones, tablet terminals, PDAs, and the like that perform stereoscopic display using multi-viewpoint images.

DESCRIPTION OF SYMBOLS 1 Virtual viewpoint image generation apparatus 2 Glasses type | mold stereoscopic display apparatus (stereoscopic display apparatus)
3 Multi-viewpoint autostereoscopic display device (stereoscopic display device)
10 Image acquisition unit (acquisition means)
50 Interpolation vector calculation part (movement amount calculation means)
60 New viewpoint image generation unit (new viewpoint image generation means)
70 Boundary smoothing unit (pixel smoothing means)

Claims (10)

A virtual viewpoint image generation device that generates a new viewpoint image that is an image according to a virtual viewpoint position,
An acquisition unit for acquiring two or more images at different viewpoint positions;
A movement that selects one image as a reference image from among the images acquired by the acquisition unit, and calculates a movement amount of each pixel of the reference image to generate a new viewpoint image from the acquired plurality of images. A quantity calculator,
A new viewpoint image generation unit that generates the new viewpoint image by moving each pixel of the reference image based on the movement amount calculated by the movement amount calculation unit;
The target pixel of the new viewpoint image generated by the new viewpoint image generation unit or the target block that is a set of a plurality of pixels is identified, and the movement amount variation in each pixel in the predetermined region including the target pixel or the target block And a smoothing processing unit that executes a smoothing process on the predetermined region according to the degree of variation, and replaces the value of the target pixel or the target block with a smooth value obtained by the smoothing process. A virtual viewpoint image generation device characterized by comprising: The virtual viewpoint image generation apparatus according to claim 1, wherein the smoothing processing unit determines a smoothing strength in accordance with the degree of variation, and performs smoothing in accordance with the strength. 3. The virtual viewpoint image generation apparatus according to claim 1, wherein the smoothing processing unit obtains an absolute value of a difference between the maximum value and the minimum value of the movement amount as the degree of variation.   A control program for causing a computer to function as the virtual viewpoint image generation device according to claim 1, wherein the computer causes the computer to function as each unit.   The computer-readable recording medium which recorded the control program of Claim 4. A virtual viewpoint image generation method by a virtual viewpoint image generation device that generates a new viewpoint image that is an image according to a virtual viewpoint position,
An acquisition step of acquiring two or more images at different viewpoint positions;
From the images acquired in the acquisition step, one image is selected as a reference image, and a movement amount of each pixel of the reference image is calculated from the acquired images to generate a new viewpoint image. A movement amount calculating step;
A new viewpoint image generation step of generating the new viewpoint image by moving each pixel of the reference image based on the movement amount calculated in the movement amount calculation step;
A target block that is a target pixel or a set of a plurality of pixels of the new viewpoint image generated in the new viewpoint image generation step is specified, and in a predetermined region including the target pixel or the target block, a movement amount of each pixel A smoothing process step of obtaining a variation, executing a smoothing process on the predetermined region according to a degree of the variation, and replacing a value of the target pixel or the target block with a smooth value obtained by the smoothing process; A virtual viewpoint image generation method comprising: The virtual viewpoint image generation device according to any one of claims 1 to 3,
A stereoscopic display device that performs stereoscopic display using a virtual viewpoint image generated by the virtual viewpoint image generation device. A new viewpoint position detector that detects the positions of the left and right eyes of the dedicated glasses for stereoscopic viewing as two new viewpoint positions;
The virtual viewpoint image generation device generates two virtual viewpoint images corresponding to the two new viewpoint positions detected by the new viewpoint position detection unit,
The stereoscopic display device according to claim 7, wherein two virtual viewpoint images generated by the virtual viewpoint image generation device are displayed. Dedicated glasses for stereoscopic viewing, with dedicated glasses that detect and transmit your left and right eye positions as two new viewpoint positions;
A new viewpoint position receiving unit that receives two new viewpoint positions detected by the dedicated glasses,
The virtual viewpoint image generation device generates two virtual viewpoint images corresponding to the two new viewpoint positions received by the new viewpoint position receiving unit,
The stereoscopic display device according to claim 7, wherein two virtual viewpoint images generated by the virtual viewpoint image generation device are displayed. The virtual viewpoint image generation device generates a plurality of virtual viewpoint images corresponding to a plurality of predetermined new viewpoint positions,
The three-dimensional autostereoscopic display according to claim 7, wherein the multi-viewpoint autostereoscopic display is performed by displaying the virtual viewpoint image generated by the virtual viewpoint image generation device in a direction corresponding to each viewpoint position. Display device.

Images