JP5210416B2 - Stereoscopic image generating apparatus, stereoscopic image generating method, program, and recording medium - Google Patents

Abstract

When the position of a vanishing point can be estimated, a depth model of an image is generated on the basis of the vanishing point, and when the position of the vanishing point cannot be estimated, the depth model of the image is generated on the basis of the degree of prominence, whereby a three-dimensional image can be generated with a more natural feeling of depth. An apparatus for generating a three-dimensional image (1) is provided with: a vanishing point estimation unit (20) for estimating a vanishing point from an image to be processed; a depth model generator (30) for generating a different depth model on the basis of whether it was possible or impossible for the vanishing point estimation unit (20) to estimate the vanishing point; and a viewpoint image generator (40) for generating the image presented to the right eye and the image presented to the left eye on the basis of the depth model generated by the depth model generator (30), the image to be processed, and assumed viewing condition information. If it was possible for the vanishing point estimation unit (20) to estimate the vanishing point, the depth model generator (30) generates the depth model on the basis of the vanishing point, and if it was impossible for the vanishing point estimation unit (20) to estimate the vanishing point, the depth model is generated on the basis of the degree of prominence of each pixel within the image to be processed.

Description

  The present invention relates to a stereoscopic image generation apparatus, a stereoscopic image generation method, a program, and a recording medium that add binocular stereoscopic information to a 2D image and generate a 3D image.

  In recent years, with the spread of 3D TV (3D Television) and the start of 3D digital broadcasting, the environment for viewing 3D video at home is being prepared. However, with the improvement of the 3D video playback environment, it has been pointed out that there is a shortage of 3D video content. As an approach to resolving such a shortage of content, attention has been focused on 2D / 3D conversion (2D to 3D conversion) in which binocular stereoscopic information is artificially added to a 2D image to generate a 3D image.

  As a technique for realizing 2D / 3D conversion, for example, a technique disclosed in Patent Document 1 is known. This patent document 1 is provided with a basic depth model indicating depth values of three basic images, and the input image depth model is changed by changing the composition ratio of the three basic depth models according to the pattern of the input image. A stereoscopic image generation apparatus that generates an image to be presented to the left eye / right eye from the generated depth model and an input image is disclosed.

Japanese Patent No. 4214976 (Japanese Patent Laid-Open No. 2005-151534)

J. Shi and C. Tomasi, “Good Features to Track,” 9th IEEE Conference on Computer Vision and Pattern Recognition, June 1994 B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” Proceedings of the 1981 DARPA Imaging Understanding Workshop (pp.121-130), 1981 CG-ARTS Association, Digital Image Processing 2nd Edition, 2009 Ota Noboru, Color Engineering 2nd Edition, Tokyo Denki University Press, 2001 L. Itti, C. Koch, E. Niebur, “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No.11, pp.1254-1259, Nov 1998. A. Telea, “An image inpainting technique based on the fast marching method,” Journal of Graphics Tools 9 (2004), pp. 25-36.

  However, in the stereoscopic image generation device disclosed in Patent Literature 1, it is difficult to express a depth model of an image that does not match the assumed basic depth model. For example, when there is a vanishing point outside the screen, there is a problem that it cannot be expressed even if the composition ratio of the three basic depth models is changed. For this reason, it was not possible to generate a stereoscopic image with a natural sense of depth.

  The present invention was made to solve the above-described problems, and when the vanishing point position can be estimated by a geometric depth cue, a depth model of the image is generated based on the vanishing point, If the vanishing point position cannot be estimated due to a geometric depth cue, a more natural depth sensation can be obtained by generating a depth model of the image based on the degree of saliency representing the attractiveness in the image based on human visual characteristics. It is an object of the present invention to provide a stereoscopic image generating apparatus, a stereoscopic image generating method, a program, and a recording medium that can generate a certain stereoscopic image.

In order to solve the above-described problem, a first technical means of the present invention is a stereoscopic image generation apparatus that generates binocular stereoscopic information by adding binocular stereoscopic information to a 2D image, and estimates a vanishing point from the processing target image. Vanishing point estimating means, depth model generating means for generating a different depth model based on whether the vanishing point can be estimated by the vanishing point estimating means, the depth model generated by the depth model generating means, and the processing target Based on the image and the assumed viewing condition information, a viewpoint image generation unit that generates a right eye presentation image and a left eye presentation image is provided, and the depth model generation unit can estimate the vanishing point by the vanishing point estimation unit. A depth model is generated based on the vanishing point, and if the vanishing point cannot be estimated by the vanishing point estimating means, the depth model is based on the saliency of each pixel in the processing target image. Generated, the vanishing point estimation means, wherein the frame vanishing point estimation means from the line information for estimating the vanishing point of the process target image in the process target image, than the processing target image and the processing target image in the past An inter-frame erasure that obtains a projective transformation matrix from the correspondence of feature points with the comparison target image and estimates the vanishing point of the processing target image based on the obtained projective transformation matrix and the position of the vanishing point in the comparison target image Point estimation means, and the stereoscopic image generation apparatus further comprises scene change detection means for detecting whether or not a scene change has occurred between the processing target image and the comparison target image. If a change is detected, select the intra-frame vanishing point estimating means, and if no scene change is detected by the scene change detecting means, That you select the vanishing point estimation means between frames is obtained by the features.

A second technical means is the first technical means, wherein the saliency of each pixel in the processing target image is a local color difference obtained by multiplying a color difference between the target pixel and its surrounding pixels by a predetermined weighting factor , color Now Rui between the target pixel and the entire image is calculated by the linear sum of the global color difference multiplied by a predetermined weighting factor to the color difference between the local region and its peripheral region including the attention pixel, the color difference, lightness, red - green perception chromaticity and yellow - based on the difference between the perception chromaticity of blue is defined is obtained by said Rukoto.

According to a third technical means, in the second technical means, when the vanishing point is estimated by the vanishing point estimating means, the depth model generating means calculates the position of the vanishing point and each pixel in the processing target image. Based on the distance determined from the coordinates and the region division information of the processing target image, an average value of the distance in each region indicated by the region division information is obtained, and the depth value of each pixel in the processing target image is An average value is set as a sum of a value normalized by a first weighting factor and a predetermined depth value, and the first weighting factor sets the difference between the upper limit value and the lower limit value of the depth value as the distance. When the vanishing point is not estimated by the vanishing point estimating means, the saliency of each pixel in the processing target image and the region division information of the processing target image are divided by the difference between the upper limit value and the lower limit value And each area indicated by the area division information based on And calculating a depth value of each pixel in the processing target image as a sum of a value obtained by normalizing the average value by a second weighting factor and a predetermined depth value, The second weighting factor is characterized in that the difference between the upper limit value and the lower limit value of the depth value is divided by the difference between the upper limit value and the lower limit value of the saliency .

The fourth technical means includes storage means for storing vanishing point information including the position of the vanishing point of the comparison target image in any one of the first to third technical means, and the storage means includes the comparison target. When the vanishing point information of the image is stored, the inter-frame vanishing point estimation unit is selected, and when the vanishing point information of the comparison target image is not stored in the storage unit, the intra-frame vanishing point estimation unit It is characterized by being selected.

According to a fifth technical means, in the fourth technical means, the comparison target image is an image immediately before the processing target image.

A sixth technical means is a stereoscopic image generation method by a stereoscopic image generation apparatus that adds binocular stereoscopic information to a 2D image and generates a 3D image, and the stereoscopic image generation apparatus detects a vanishing point from a processing target image. A vanishing point estimating step, a depth model generating step for generating a different depth model based on whether or not the vanishing point can be estimated in the vanishing point estimating step, and a depth model generated in the depth model generating step; A viewpoint image generation step of generating a right eye presentation image and a left eye presentation image based on the processing target image and the assumed viewing condition information, and the depth model generation step includes a vanishing point in the vanishing point estimation step. Can be estimated, a depth model is generated based on the vanishing point, and if the vanishing point cannot be estimated in the vanishing point estimation step, To generate a depth model based on the saliency of each pixel, the vanishing point estimation step, the frame vanishing point estimation step of estimating a vanishing point of the processed image from the line information in the processed image, the processing A projection transformation matrix is obtained from a correspondence relationship between feature points between the target image and a comparison target image that is earlier than the processing target image, and the processing is performed based on the obtained projection transformation matrix and the position of the vanishing point in the comparison target image. An inter-frame vanishing point estimating step for estimating a vanishing point of the target image, and the stereoscopic image generating device further detects whether or not a scene change has occurred between the processing target image and the comparison target image A change detection step, and when a scene change is detected in the scene change detection step, the in-frame vanishing point estimation step is selected, If the scene change is not detected at Nchenji detection step is obtained by said you to select the vanishing point estimation step between the frames.

The seventh technical means is a program for causing a computer to execute the stereoscopic image generating method in the sixth technical means.

The eighth technical means is a computer-readable recording medium on which the program in the seventh technical means is recorded.

According to the present invention, when the vanishing point position can be estimated by the geometric depth cue, a stereoscopic image in which the depth feeling by the geometric depth cue is emphasized is generated by generating the image depth model based on the vanishing point. Can be generated.
Further, according to the present invention, when the vanishing point position cannot be estimated by the geometric depth cue, by generating the image depth model from the saliency representing the attractiveness in the image based on the human visual characteristics, It is possible to generate a stereoscopic image in which the sense of depth of a portion that is noticed by a person is emphasized.

It is a block diagram which shows the structural example of the stereo image production | generation apparatus which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the frame unit of the stereo image production | generation apparatus concerning embodiment of this invention. It is the schematic of the scene change detection based on a brightness | luminance histogram. It is a block diagram which shows the structural example of the scene change detection part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the scene change detection part which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the vanishing point estimation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the vanishing point estimation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the vanishing point estimation part in a flame | frame which concerns on embodiment of this invention. It is a figure which shows an example of the image corresponding to the flow of FIG. It is the schematic for demonstrating the straight line detection by Hough transform. It is a flowchart for demonstrating the operation example of the vanishing point estimation part between frames which concerns on embodiment of this invention. It is a figure which shows an example of the image corresponding to the flow of FIG. It is a figure which shows an example of the range to which an intra-frame vanishing point estimation means and an inter-frame vanishing point estimation means are applied in the same scene. It is a block diagram which shows the structural example of the depth model production | generation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the depth model production | generation part which concerns on embodiment of this invention. It is a figure which shows an example of a basic depth model in case there exists a vanishing point in the screen which concerns on embodiment of this invention. It is a figure which shows an example of the basic depth model in case there exists a vanishing point outside the screen which concerns on embodiment of this invention. It is a figure which shows an example of the process which calculates | requires the depth model based on the vanishing point which concerns on embodiment of this invention. It is a figure which shows an example of the process which calculates | requires the depth model based on the saliency which concerns on embodiment of this invention. It is a bird's-eye view of camera (viewpoint) arrangement for generating a viewpoint image. It is a block diagram which shows the structural example of the viewpoint image generation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the viewpoint image generation part which concerns on embodiment of this invention. It is a figure which shows an example of the process which produces | generates the viewpoint image which concerns on embodiment of this invention. It is a figure which shows the parallax vector of a cross direction and a spreading | diffusion direction. It is a block diagram which shows the modification of the depth model production | generation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the depth model production | generation means based on the saliency in the modification of the depth model production | generation part which concerns on embodiment of this invention. It is a figure which shows an example of the process which calculates | requires the depth model (modification) based on the saliency which concerns on embodiment of this invention. It is a figure which shows an example of the depth model corresponding to an example of the modification of the depth model used as a reference | standard in the depth model (modification) based on the saliency which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the stereo image production | generation apparatus (1st modification) which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the frame unit of the stereo image production | generation apparatus (1st modification) which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the stereo image production | generation apparatus (2nd modification) which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the frame unit of the stereo image production | generation apparatus (2nd modification) which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the spatio-temporal direction smoothing part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the spatiotemporal direction smoothing part which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, portions having the same function are denoted by the same reference numerals, and repeated description is omitted.
FIG. 1 is a block diagram illustrating a schematic configuration example of a stereoscopic image generating apparatus according to the present invention. In the figure, reference numeral 1 denotes a stereoscopic image generating apparatus. The stereoscopic image generation apparatus 1 includes a scene change detection unit 10, a vanishing point estimation unit 20, a depth model generation unit 30, and a viewpoint image generation unit 40. FIG. 2 is a flowchart for explaining an operation example in units of frames of the stereoscopic image generating apparatus 1 according to the present invention.

  2, first, the stereoscopic image generation apparatus 1 in FIG. 1 uses an input image at time t (hereinafter also referred to as a processing target image F (t)) as a scene change detection unit 10, a vanishing point estimation unit 20, a depth. It outputs to the model production | generation part 30 and the viewpoint image production | generation part 40 (step S11 of FIG. 2).

(About the scene change detection unit 10)
The scene change detection unit 10 in FIG. 1 corresponds to the scene change detection means of the present invention, and the input processing target image F (t) and the image input immediately before the processing target image F (t) ( Hereinafter, a predetermined image feature amount is calculated from the comparison target image F (t-1)), the similarity of the calculated image feature amount is compared, and segmentation points (scene changes) of images that are continuous in time series , And scene change information S (t) indicating the presence or absence of a scene change in the processing target image F (t) is output to the vanishing point estimation unit 20 and the depth model generation unit 30 (step S12 in FIG. 2). Here, as an example of the predetermined image feature amount, scene change detection based on a luminance histogram representing the appearance frequency of the luminance value of an image will be described with reference to FIGS.

As shown in FIG. 3, the scene change detection based on the luminance histogram is performed based on the luminance histograms H _L (t) and H _L (t−1) from the processing target image F (t) and the comparison target image F (t−1). ) And the calculated similarity d (H _L (t), H _L (t−1)) of the luminance histogram is compared with a predetermined threshold value, and whether or not there is a scene change in the processing target image F (t) Is to determine.

As shown in FIG. 4, the scene change detection unit 10 includes a luminance histogram generation unit 101, a buffer 102, a histogram similarity calculation unit 103, and a scene change determination unit 104. FIG. 5 is a flowchart for explaining an operation example of the scene change detection unit 10. In FIG. 5, the luminance histogram generation unit 101 in FIG. 4 acquires luminance information from the input processing target image F (t), and a luminance histogram H _L (t) that represents the appearance frequency of the luminance value from the acquired luminance information. And the calculation result (luminance histogram H _L (t)) is output to the buffer 102 and the histogram similarity calculation unit 103 (step S21 in FIG. 5).

The buffer 102 in FIG. 4 stores the luminance histogram H _L (t) of the processing target image F (t) in order to detect a scene change in the image F (t + 1) immediately after the processing target image F (t). (Step S22 in FIG. 5). The histogram similarity calculation unit 103 in FIG. 4 inputs the luminance histogram H _L (t) of the input processing target image F (t) and the luminance histogram H of the comparison target image F (t−1) read from the buffer 102. _The similarity d (H _L (t), H _L (t−1)) is calculated from _L (t−1) by the equation (1), and the calculation result is output to the scene change determination unit 104 (FIG. 5). Step S23).

Here, in Expression (1), W represents the number of pixels per line of the image, H represents the number of lines of the image, v represents the luminance value, V represents the number of gradations of the luminance value, and H _L (v | t) represents the appearance frequency of the luminance value v on the image F (t) at time t. In Expression (1), the range of values taken by the similarity d (H _L (t), H _L (t−1)) is 0 to 2, and the closer the value is to 0, the more similar the shape of the histogram is. The closer the value is to 2, the more different the shape of the histogram is.

The scene change determination unit 104 in FIG. 4 performs threshold determination based on the similarity d (H _L (t), H _L (t−1)) of the input histogram and a predetermined threshold d _th, and formula (2) ), Scene change information S (t) indicating the presence / absence of a scene change in the processing target image F (t) is set and output to the outside (step S24 in FIG. 5).

That is, when the similarity d (H _L (t), H _L (t−1)) is smaller than the threshold value d _th , the scene change determination unit 104 determines that there is no scene change, and the scene change information S (t) Set “0” to. In other cases, it is determined that there is a scene change, and “1” is set in the scene change information S (t).

  As described above, according to the scene change detection unit 10, the predetermined image feature amount is calculated from the processing target image F (t) and the comparison target image F (t-1), and the similarities of the calculated image feature amounts are compared. Thus, it is possible to detect segment points (scene changes) of images that are continuous in time series.

(About the vanishing point estimation unit 20)
Returning to FIG. 1, the vanishing point estimation unit 20 corresponds to the vanishing point estimation unit of the present invention, and the input scene change information S (t) of the processing target image F (t) and the vanishing point estimation unit 20. Vanishing point estimating means (intra-frame vanishing point estimating means for estimating vanishing point position from straight line in image, feature between images based on previous vanishing point information VP (t−1) stored internally An inter-frame vanishing point estimating means for estimating the vanishing point position in the current frame from the point correspondence and the vanishing point position of the previous frame) is selected, and the processing target image F (t) input by the selected vanishing point estimating means is selected. The vanishing point position is estimated, and vanishing point information VP (t) describing the result is output to the depth model generating unit 30 (step S13 in FIG. 2). Here, the “vanishing point” refers to a point that, when two parallel lines in a three-dimensional space are projected (projected) onto a plane, these lines always converge to one point.

  Then, the vanishing point estimation part 20 in this embodiment is demonstrated in detail. As illustrated in FIG. 6, the vanishing point estimation unit 20 includes a switching unit 201, a switching unit 202, an intraframe vanishing point estimation unit 21, an interframe vanishing point estimation unit 22, a buffer 203, and a buffer 204. The intra-frame vanishing point estimation unit 21 in FIG. 6 includes an edge detection unit 211, a straight line detection unit 212, and a vanishing point identification unit 213. This intra-frame vanishing point estimation unit 21 corresponds to the intra-frame vanishing point estimation means of the present invention, and estimates the vanishing point of the processing target image F (t) from the straight line information in the processing target image F (t). 6 includes a feature point detection unit 221, a corresponding point calculation unit 222, a transformation matrix calculation unit 223, and a vanishing point position calculation unit 224. The inter-frame vanishing point estimation unit 22 corresponds to the inter-frame vanishing point estimation means of the present invention, and is compared with the processing target image F (t) and the processing target image F (t) in the past comparison target image F (t−1). ) And the position of the vanishing point in the comparison target image F (t−1), the vanishing point of the processing target image F (t) is estimated. FIG. 7 is a flowchart for explaining an operation example of the vanishing point estimation unit 20.

  In FIG. 7, the vanishing point estimation unit 20 of FIG. 6 estimates vanishing points based on the input scene change information S (t) and the previous vanishing point information VP (t−1) read from the buffer 204. A means is selected (step S31 in FIG. 7). Specifically, when there is a scene change (“S (t) = 1”), or when the vanishing point information VP (t−1) indicates that there is no vanishing point in the previous frame, that is, the vanishing point. When the information VP (t−1) is “vp_num = 0” (Yes in step S31 in FIG. 7), the switching unit 201 in FIG. 6 outputs an image input destination, and the switching unit 202 in FIG. 6 outputs vanishing point information. The original is switched to the intra-frame vanishing point estimation unit 21 (step S32 in FIG. 7), and then the intra-frame vanishing point estimation unit 21 estimates the vanishing point position from the straight line in the image, and the result (vanishing point information) VP (t)) is output (step S33 in FIG. 7).

(Intraframe vanishing point estimation unit 21)
Here, the intra-frame vanishing point estimation unit 21 will be described in detail. FIG. 8 is a flowchart for explaining an operation example of the intra-frame vanishing point estimation unit 21. FIG. 9 is a diagram illustrating an image example corresponding to the flow of FIG. In step S331 in FIG. 8, the edge detection unit 211 in FIG. 6 calculates edge point information Edge (t) used for straight line detection from the input processing target image F (t) (see FIG. 9A). . Specifically, first, a differential operator is applied to each color component (for example, RGB (Red (red), Green (green), Blue (blue))), and a gradient vector of each color component i in the x direction and the y direction. G _i (x, y | t) = (ΔG _ix (t), ΔG _iy (t)) (i = 1, 2, 3) is calculated. Note that i = 1, 2, and 3 are an R component, a G component, and a B component, respectively.

  Subsequently, the edge detection unit 211 calculates the edge strength E (x, y | t) by performing the calculation of Expression (3) for each pixel of the coordinates (x, y).

  Subsequently, the edge detection unit 211 performs the calculation of Expression (4) for each pixel of the coordinates (x, y), so that the edge strength is locally maximum from the edge strength E (x, y | t). Are extracted as edge points, and edge point information Edge (t) describing the result is output to the straight line detection unit 212.

  That is, the edge detection unit 211 has an edge when the edge intensity E (x, y | t) is a maximum value within the window size W1 × W2 centered on the coordinates (x, y) (Local Maxima). “1” is set to the point Edge (x, y | t), and “0” is set to the edge point Edge (x, y | t) otherwise. Here, W1 represents the size of the window in the x direction, and W2 represents the size of the window in the y direction.

  Proceeding to step S332 in FIG. 8, the straight line detection unit 212 in FIG. 6 applies straight line information L (t) to the input edge point information Edge (t) by applying Hough transform (see FIG. 9B). To get. Here, the straight line detection by the Hough transform will be described with reference to FIG. In FIG. 10A, feature points A, B, and C on a certain straight line L are edge points that become “Edge (x, y | t) = 1” obtained by the edge detection unit 211. . First, in the Hough transform, a straight line L on the image space shown in FIG. 10A is expressed using polar coordinate expressions (ρ, θ). ρ represents the distance of a perpendicular drawn from the origin of the image space to the straight line L, and θ represents the angle formed by the perpendicular to the x axis of the image space. The range of ρ is ρ ≧ 0, and the range of θ is 0 ≦ θ <2π.

A straight line L that passes through the feature points A, B, and C on the image space of FIG. 10A is expressed by Expression (5) using parameters (ρ ₀ , θ ₀ ).

Further, when a straight line group passing through each of the feature points A, B, and C is mapped to the parameter space, it is represented as a curve a, a curve b, and a curve c on the parameter space in FIG. 10B. That is, the intersection (ρ ₀ , θ ₀ ) of the curve a, the curve b, and the curve c is detected as a straight line L passing through the feature points A, B, and C on the parameter space.

As described above, the Hough transform is applied to the edge point where “Edge (x, y | t) = 1”, and the curves having a predetermined threshold value or more intersect on the parameter space, and the number of intersections increases. NL polar coordinates (ρs, θs) (s = 1,..., NL) are extracted as a straight line Ls, and data describing the polar coordinates (ρs, θs) is set as straight line information L (t). The range of NL is 0 ≦ NL ≦ NL _max . Here, NL _max is a predetermined constant representing the upper limit value of the number of straight lines to be extracted. If the straight line extraction condition is not satisfied and no straight line is detected, NL = 0.

Returning to FIG. 8 again, the vanishing point identifying unit 213 in FIG. 6 obtains the number of straight lines NL from the input straight line information L (t), compares the number of straight lines NL with a predetermined threshold value, and compares the magnitude of the step 334. ~ It is determined whether or not the position of the vanishing point is to be estimated by the process of step S335 (step S333). When the number of straight lines NL is smaller than the threshold ThL (≧ 2) (or in the following case) (No in step S333), it is determined that there is no geometric depth clue sufficient to estimate the vanishing point, and the process proceeds to step S336. When the number of straight lines NL is greater than or equal to the threshold ThL (≧ 2) (or larger) (Yes in step S333), the vanishing point identifying unit 213 determines that there is a sufficient geometric depth clue for vanishing point estimation. From the input straight line information L (t), the straight line Li (i = 1,..., NL) and the straight line Lj (j) satisfying the conditions of the expressions (6) and (7) with respect to the angle θ representing the straight line. = 1,..., NL), the intersection P _ij (i ≠ j) is calculated by the matrix operation of the equation (8), and the intersection information is acquired (step S334). It should be noted that when the intersection of the straight lines is obtained, the overlap calculation between the straight line Li and the straight line Lj once selected is not performed. Note that the condition of Expression (6) represents that the selected two straight lines are not parallel, and the condition of Expression (7) is the vicinity in the horizontal direction (| θ−π | ≈π / 2) and the vertical direction (| θ−π). | ≈0 or | θ−π | ≈π).

Subsequently, the vanishing point identifying unit 213 assumes that the distribution model of the acquired intersection point P _ij of the straight line is expressed using a mixed model GMM (Gaussian Mixture Model) of Kc Gaussian distributions shown in Expression (9). Then, the parameters (wi, μi, Σi) (i = 1,..., Kc) of the distribution model are acquired by the EM (Expectation-Maximization) algorithm, and the position of the vanishing point is determined (step S335).

  In Equation (9), P (x) represents the probability that the vector x (coordinates of the intersection Pij) will appear. Kc represents the number of classes (number of Gaussian distributions), wi represents a weighting coefficient of class i Gaussian distribution, and the sum of the weighting coefficients is 1. Μi represents a class i average vector (class i barycentric coordinates), Σi represents a class i covariance matrix, and D represents the number of dimensions of the vector x. N (x | μi, Σi) in Expression (9) represents a Gaussian distribution (normal distribution) of class i, and is expressed using an average vector μi and a covariance matrix Σi. That is, the vanishing point identifying unit 213 determines the average vector μi (center of gravity coordinates) of the distribution of the top N (≧ 1) classes having a large weighting coefficient wi as the vanishing point position. Hereinafter, for the sake of simplicity, the number of vanishing points will be described as N = 1, but the present invention is not limited to this.

  Subsequently, the vanishing point identifying unit 213 sets vanishing point information VP (t) as shown in FIG. 9C based on the result of Step S333 or Step S335 (Step S336). The vanishing point information VP (t) is expressed as data in Table 1, for example.

  In Table 1, vanishing point information VP (t) is “vp_time” indicating time t (or a number (frame number) assigned to a frame of an image), “vp_num” indicating the number of detected vanishing points, And a list representing the position “vp_pos [n]” of the n vanishing points detected.

  Returning to step S31 in FIG. 7 again, when there is no scene change (“S (t) = 0”) and the vanishing point information VP (t−1) indicates that there is a vanishing point in the previous frame ( (Vp_num> 0 ”of vanishing point information VP (t−1)) (No in step S31), the switching unit 201 in FIG. 6 is the input destination of the image, and the switching unit 202 in FIG. 6 is the output source of the vanishing point information. Then, switching to the inter-frame vanishing point estimation unit 22 is performed (step S34), and then the inter-frame vanishing point estimation unit 22 and the input processing target image F (t) and the previous image F (t stored in the buffer 203 ( From t-1), a correspondence relationship between feature points between images is obtained, and the vanishing point position in the processing target image F (t) is estimated from the correspondence relationship and the previous vanishing point information VP (t-1). As a result, vanishing point information VP (t) is output (step). S35). That is, the vanishing point estimation unit 20 includes storage means (buffer 204 in FIG. 6) that stores vanishing point information VP (t−1) including the position of the vanishing point of the comparison target image F (t−1). Whether or not the frame has a vanishing point is determined by storing the vanishing point information VP (t−1) of the previous frame in this storage means, and the vanishing point information VP (t−1) is “vp_num> 0”. It is determined by whether or not there is.

(About the inter-frame vanishing point estimation unit 22)
Here, details of the inter-frame vanishing point estimation unit 22 will be described. FIG. 11 is a flowchart for explaining an operation example of the inter-frame vanishing point estimation unit 22. FIG. 12 is a diagram illustrating an example of an image corresponding to the flow of FIG. In step S351 in FIG. 11, the feature point detection unit 221 in FIG. 6 receives the input processing target image F (t), the previous image F (t−1), as shown in FIG. NK feature points Ks (s = 1,..., NK) used for obtaining the correspondence between the images are detected, and the coordinates (x _{Ks, t} , y _{Ks, t} ) of the feature points Ks are detected. The described feature point information K (t) is output to the corresponding point calculation unit 222 in FIG. 6 (step S351).

  A feature point is a point extracted as a part or vertex of a subject's edge based on a change in color or luminance between pixels. For example, a secondary expressed using a gradient vector Gi (x, y) (i = x, y) of luminance in the x and y directions within the range of the local region S with the pixel (x, y) as the center. A first eigenvalue λ1 and a second eigenvalue λ2 of the moment matrix A (Expression (10)) are obtained, and a pixel (x, y) that satisfies the condition shown in Expression (11) is detected as a feature point.

  That is, a characteristic point is obtained when the smaller eigenvalue of the first eigenvalue λ1 and the second eigenvalue λ2 of the second moment matrix A is greater than (or greater than) the predetermined threshold λth (for example, non-patent literature). 1). In Equation (10), the coefficient w (u, v) represents a weighting coefficient for a pixel (x + u, y + v) that is separated from the pixel (x, y) by u in the x direction and v in the y direction. A value obtained by normalizing the value of the secondary Gaussian distribution within the range of the local region S determined so as to satisfy the condition (12) is used.

Proceeding to step S352 in FIG. 11, the corresponding point calculation unit 222 in FIG. 6, as shown in FIG. 12B, inputs the processing target image F (t) and the previous one read from the buffer 203. Based on the image F (t−1) and the feature point information K (t) of the processing target image F (t) acquired in step S351, each feature point Ks (s = 1) of the processing target image F (t). ,..., NK) is calculated by optical flow at a position (x _{Ks, t −1} , y _{Ks, t −1} ) where the previous image F (t−1) is on the image F (t−1). Corresponding point information Q (t, t−1) describing the positions at time t and time t−1 is output to the transformation matrix calculation unit 223 in FIG. 6 (step S352).

Note that the position (x _{Ks, t−1} , y _{Ks, t−1} ) where each feature point Ks of the processing target image F (t) is on the previous image F ( _t−1 ) is, for example, an expression It can be acquired by solving the constraint condition of the optical flow by the gradient method shown in (13) with respect to (x _{Ks, t−1} , y _{Ks, t−1} ) (for example, see Non-Patent Document 2).

  Here, in Expression (13), Gi (x, y | t) (i = x, y, t) is a gradient vector in the x direction, y direction, and t direction (time direction) related to the luminance of the image F (t). S represents a local area of a predetermined size centered on the feature point Ks.

  Proceeding to step S353 in FIG. 11, the transformation matrix calculation unit 223 in FIG. 6 uses the feature point Ks (s = 1,..., NK) from the corresponding point information Q (t, t−1) acquired in step S352. ) Is calculated from the position on the previous image F (t-1) to the position on the processing target image F (t), and information describing the conversion matrix H is calculated as shown in FIG. The data is output to the vanishing point position calculation unit 224 (step S353). Note that the correspondence between two images (F (t), F (t−1)) can be expressed by Expression (14) using the transformation matrix H (see, for example, Non-Patent Document 3). . In this equation (14), the symbol “˜” represents an equivalence relationship, which means that they are equal by allowing a constant multiple difference.

  The transformation matrix H is called projective transformation because it can express general transformation. Here, if it is assumed that the correspondence between images can be expressed as parallel movement, Expression (14) is expressed as Expression (15).

  Coefficients tx and ty in equation (15) represent the amounts of movement in the x and y directions, respectively. Assuming that the correspondence between images can be expressed as affine transformation including translation, rotation, and enlargement / reduction, Expression (14) is expressed as Expression (16).

Coefficients a, b, c, and d in Expression (16) represent enlargement / reduction and rotation, and coefficients tx and ty are the same as those in Expression (15). The coefficients h _ij (i, j = 1, 2, 3) of the transformation matrix H in the equations (14), (15), and (16) are the constraint conditions and corresponding point information Q ( It is calculated by solving simultaneous equations derived from t, t-1) by the method of least squares. If there is not a sufficient number of corresponding points and the conversion matrix H cannot be calculated, a predetermined conversion matrix H ₀ is used.

  Proceeding to step S354 in FIG. 11, the vanishing point position calculation unit 224 in FIG. 6 reads “The vanishing point position at time t is the vanishing point position on the previous image F (t−1). Using the transformation matrix H calculated by the transformation matrix calculation unit 223, the vanishing point position at the time t is calculated (step 354), assuming that the position is projected onto the processing target image F (t). Based on the result, vanishing point information VP (t) is set (step S355). FIG. 12C shows an image example when the vanishing point on the image F (t−1) is projected onto the image F (t) by the transformation matrix H.

  Returning to step S36 in FIG. 7 again, the buffer 203 in FIG. 6 deletes the previous image F (t−1) and stores the input processing target image F (t). Also, the buffer 204 in FIG. 6 deletes the previous vanishing point information VP (t−1), and the vanishing point information input from the intra-frame vanishing point estimation unit 21 or the inter-frame vanishing point estimation unit 22. VP (t) is stored, and the vanishing point estimation process for the processing target image F (t) is terminated (step S36).

As described above, according to the vanishing point estimation unit 20 of the present embodiment, as shown in FIG. 13, in the same scene (a time series image group correlated in the spatial direction and the time direction), the same from the first frame F (t ₀ ). Until the frame F (t ₀ + k−1) in which the vanishing point is first detected in the scene, the vanishing point is estimated by the intra-frame vanishing point estimating means (intra-frame vanishing point estimating unit 21). From the next frame F (t ₀ + k) of the frame F (t ₀ + k−1) in which the vanishing point is detected to the last frame F (t ₀ + N) in the same scene, the inter-frame vanishing point estimation means (inter-frame Since the vanishing point is estimated by the vanishing point estimation unit 22), the vanishing point is more robust to the camera work than the case where the vanishing point is estimated by the intra-frame vanishing point estimation unit, and the vanishing point fluctuation is suppressed and stabilized. The vanishing point can be estimated.

(About the depth model generation unit 30)
Returning to FIG. 2 again, the depth model generation unit 30 in FIG. 1 corresponds to the depth model generation means of the present invention, and generates different depth models based on whether or not the vanishing point can be estimated by the vanishing point estimation unit 20. To do. That is, the depth model generation unit 30 is based on the vanishing point information VP (t), based on the visual model of the depth model creating means (first depth model creating means for creating the depth model from the vanishing point position, human visual characteristics). Second depth model creating means for creating a depth model from the saliency representing the attractiveness in the image), and the depth value of each pixel in the processing target image F (t) is set by the selected depth model creating means. Then, the depth model D (t) representing the depth value of each pixel is output to the viewpoint image generation unit 40 (step S14 in FIG. 2).

  Next, the depth model generation unit 30 in the present embodiment will be described in detail. As illustrated in FIG. 14, the depth model generation unit 30 includes a switching unit 301, a switching unit 302, an area dividing unit 303, a distance calculating unit 304, a saliency calculating unit 305, and a depth value setting unit 306. FIG. 15 is a flowchart for explaining an operation example of the depth model generation unit 30.

  In FIG. 15, the depth model generation unit 30 in FIG. 14 selects the depth model creation means based on the input vanishing point information VP (t) (step S41). That is, when there is a vanishing point in the current frame (“vp_num> 0” of vanishing point information VP (t)) (Yes in step S41), the switching unit 301 in FIG. 14 transfers the image input destination to the region dividing unit 303. The switching unit 302 in FIG. 14 switches the output source of the data input to the depth value setting unit 306 to the distance calculation unit 304, and the first depth model creation means based on the vanishing point is selected (step S42).

  In step S43, the distance calculation unit 304 in FIG. 14 calculates the distance Dist (x, y) between the coordinates of the vanishing point of the vanishing point information VP (t) and the coordinates of each pixel, and describes the result. The distance information Dist (t) is output to the depth value setting unit 306 (step S43). Specifically, the distance Dist (x, y) between the coordinates of the vanishing point and the coordinates of each pixel is calculated based on any one of Expression (17), Expression (18), and Expression (19). In Expressions (17), (18), and (19), Δx and Δy represent the distance in the x direction and the distance in the y direction between each pixel and the vanishing point, respectively.

  Here, FIG. 16 shows an example of the distance information Dist (t) based on the equations (17), (18), and (19) when the vanishing point VP is present in the screen. FIG. 16A shows an example in which the vanishing point VP is in the screen. BD1a in FIG. 16B represents equation (17), BD1b in FIG. 16C represents equation (18), and BD1c in FIG. 16D represents distance information Dist (t) based on equation (19). . FIG. 17 shows an example of the distance information Dist (t) based on the equations (17), (18), and (19) when the vanishing point VP exists outside the screen. FIG. 17A shows an example in which the vanishing point VP is outside the screen. BD2a in FIG. 17B represents equation (17), BD2b in FIG. 17C represents equation (18), and BD2c in FIG. 17D represents distance information Dist (t) based on equation (19). . In FIG. 16 and FIG. It is assumed that the white part is closest and becomes farther away as it becomes blacker.

  Proceeding to step S44, the region dividing unit 303 in FIG. 14 has a plurality of similar feature amounts (feature value values are within a predetermined range) by region division (clustering) of the processing target image F (t). Into a set of pixels (region; class). For example, the region dividing unit 303 divides the image into a plurality of regions by clustering in the feature amount space. The clustering by the feature amount space means that each pixel of the image space is mapped to the feature amount space (for example, color, edge, motion vector), and the K-means method, the Mean-Shift method, or the K nearest neighbor in the feature amount space. Clustering is performed by a technique such as a search method (approximate K nearest neighbor search method). After the clustering processing in the feature amount space is finished, the pixel value in the class is replaced with a pixel value (for example, an average value) that is a representative value of each region, and the pixel value in the original image space is replaced with each region Is assigned to all pixels in each region, and region information R (t) describing the result is output to the depth value setting unit 306 (step S44).

Proceeding to step S45, the depth value setting unit 306 in FIG. 14 sets the depth value of each pixel based on the input distance information Dist (t) and region information R (t). Specifically, as shown in Expression (20), the average value of the distances Dist (x, y) of the pixels in each area indicated by the area information R (t) is scaled, and the depth value D _{base serving} as a reference is scaled. A value shifted by (x, y) is set as the depth value D (x, y) of each pixel (step S45).

In Expression (20), D _max is the upper limit value of the depth value, D _min is the lower limit value of the depth value, Dist _max is the maximum value of the distance information Dist (t), and Dist _min is the minimum value of the distance information Dist (t). The value D _base (x, y) is a predetermined constant for adjusting the reference value of the depth value of each pixel (the depth value as the farthest view). Here, an example of the depth model based on the vanishing point is shown in FIG. In FIG. 18, an image A represents an example of the processing target image F (t), and an image B is a region division result (region division information R (t)) of the processing target image F (t) obtained by the region division unit 303. For example, the image C represents an example of the vanishing point VP of the processing target image F (t), and the image D represents an example of the distance information Dist (t) of the processing target image F (t) obtained by the distance calculation unit 304. The image E is an example of the depth model obtained by the depth value setting unit 306 based on the region division information R (t) of the image B and the distance information Dist (t) of the image D. In the image E of FIG. 18, the bright part represents the near side, and the dark part represents the back.

  Returning to step S41 in FIG. 15 again, when there is no vanishing point in the current frame (“vp_num = 0” in vanishing point information VP (t)) (No in step S41), the switching unit 301 in FIG. The switching unit 302 in FIG. 14 switches the output source of the data input to the saliency calculating unit 306 to the saliency calculating unit 306, and the depth model creating means based on the saliency is selected. (Step S46).

  The saliency calculating unit 305 in FIG. 14 calculates the saliency M (t) representing the attractiveness in the image based on the human visual characteristics from the input processing target image F (t) (step S47). Examples of portions that are easy for people to pay attention to are locations where the color difference between the pixel of interest and its surrounding pixels is large (local color difference), locations where the color difference between the pixel of interest and the entire image is large, or a local region including the pixel of interest There is a portion (global color difference) where the color difference between the image and the surrounding area is large. The color difference is a quantitative representation of a perceptual difference in color. As a color space for evaluating the color difference, a CIELAB color space (CIE 1976 L * a * b *) which is a uniform color space is used. Also referred to as space). Based on the visual characteristics of the person, the saliency M (x, y) of each pixel is calculated by Equation (21).

Here, in Expression (21), ΔE _local represents a local color difference, ΔE _global represents a _global color difference, and coefficients α and β represent predetermined weight coefficients. That is, Expression (21) expresses the saliency by a linear sum of the local color difference and the global color difference. The local color difference ΔE _local is calculated by equation (22), and the global color difference ΔE _global is calculated by equation (23). In Expressions (22) and (23), L * represents a lightness index, a * represents red-green perceptual chromaticity, and b * represents yellow-blue perceived chromaticity. Note that a CIELV color space (also referred to as CIE 1976 L * u * v * color space) may be used as the color space for evaluating the color difference. Note that the coefficient w (u, v) in the equation (22) is the same as that in the equation (12), and thus the description thereof is omitted.

  Proceeding to step S48, the depth value setting unit 306 in FIG. 14 sets the depth value of each pixel by the calculation of Expression (24) based on the input saliency M (t), and describes the result. The depth model D (t) is output (step S48).

In Formula (24), D _max is the upper limit value of the depth value, D _min is the lower limit value of the depth value, M _max is the maximum value of the saliency M (t), M _min is the minimum value of the saliency M (t), D _base (x, y) is a predetermined constant for adjusting the reference value of the depth value of each pixel (the depth value as the farthest view). That is, the saliency M (x, y) of each pixel is scaled by the equation (24), and the value shifted by the reference depth value D _base (x, y) is the depth value D (x, y) of each pixel. Set as. Here, an example of the depth model based on the saliency is shown in FIG. In FIG. 19, an image A represents an example of the processing target image F (t), an image B represents an example of the saliency M (t) of the processing target image F (t) obtained by the saliency calculating unit 305, and the image C represents an example of a reference depth model (D _base ), and the image D is created by combining the depth model (D _base ) of the image C with the saliency M (t) of the image B in the depth value setting unit 306. It is an example of a depth model.

In the image B of FIG. 19, a bright part (white) represents a part that is easily noticed by a person (high attractiveness), and a dark part (black) represents a part that is difficult for human attention (low attractiveness). Moreover, in the image D of FIG. 19, it represents that a bright part is a near side, and a dark part represents that it is the back. As shown in the image D of FIG. 19, the depth model creation means based on the saliency superimposes the scaled saliency on the reference depth surface (D _base ), and the region of interest and its surrounding region By emphasizing the difference in relative depth, a pseudo depth feeling is perceived.

  When the depth model is generated based on the saliency, the depth of the portion with high saliency (high attraction) is on the near side on the reference depth surface, and the saliency is low (low attraction) ) Set the depth of the part to the back side. Thereby, the relative depth difference between the region of interest and the surrounding region is emphasized, and a pseudo depth sensation can be perceived. In other words, when the saliency of the attention area is higher than the saliency of the surrounding area, the depth is set to be relatively closer to the surrounding area. Further, when the saliency of the attention area is equal to the saliency of the surrounding area, the depth is set to be relatively the same. Further, when the saliency of the attention area is lower than the saliency of the peripheral area, the depth is set to be relatively deep with respect to the peripheral area.

  As described above, according to the depth model generation unit 30, when the vanishing point position can be estimated by the geometric depth clue, the depth model of the image is created based on the vanishing point, thereby generating the geometric depth clue. It is possible to create a depth model that emphasizes the sense of depth. In addition, when the vanishing point position cannot be estimated due to the geometric depth cue, the depth model of the image is created from the saliency representing the attractiveness in the image based on the visual characteristics of the person, so that It is possible to create a depth model that emphasizes the sense of depth.

(About the viewpoint image generation unit 40)
Returning to FIG. 2 again, the viewpoint image generation unit 40 of FIG. 1 corresponds to the viewpoint image generation means of the present invention, and each pixel represented by the depth model D (t) based on the assumed viewing condition information set in advance. From the depth value of each pixel, each pixel on the reference image F (t) (input image; processing target image) and the viewpoint image Fi (t) (i = 1, r; Fr: right eye presentation image, Fl: left eye presentation image) ) Calculates a disparity vector (shift amount) representing a shift amount to the corresponding pixel on the above, and based on the pixel on the reference image F (t) and the corresponding calculated disparity vector, each viewpoint image Fi (t) (I = 1, r) is generated (step S15).

  Here, “assumed viewing condition information” is information for generating a stereoscopic image (left-eye presentation image, right-eye presentation image) to be presented to the viewer, and the pixel pitch (pixel) of the display that displays the stereoscopic image. Distance) μ, display image size, distance between viewer and display for displaying stereoscopic image (assumed viewing distance) f, parallax range (range of parallax vector) representing depth of stereoscopic image, baseline length t (viewpoint) This represents the distance between the virtual right viewpoint Cr of the image Fr (t) and the virtual left viewpoint Cl of the viewpoint image Fl (t).

  An overhead view of an example of a camera (viewpoint) arrangement for generating a viewpoint image based on this assumed viewing condition information is shown in FIG. In the example of FIG. 20, assuming that a parallel image is captured, a camera on the virtual right viewpoint Cr and a camera on the virtual left viewpoint Cl are arranged in parallel with the camera on the reference viewpoint Cc in the x-axis direction, respectively. Is observing a point of interest P in a three-dimensional space. Further, the position of the attention point P projected on the image plane Ic of the reference viewpoint Cc is Xc, the position of the attention point P projected on the image plane Il of the virtual left viewpoint Cl is X1, and the image plane Ir of the virtual right viewpoint Cr. Let Xr be the position of the point of interest P projected above. In FIG. 20, the distance (focal length or viewing distance) f to the image plane corresponding to each viewpoint, the distance Z in the z direction from the viewpoint to the point of interest P, the reference viewpoint Cc and each virtual viewpoint (Cr, Cl). The geometrical relationships between the positions Xl and Xc, Xr and Xc of the target point P projected on each image plane using the distance t / 2 in the x direction are expressed by equations (25) and (26), respectively. The

  From the above, the disparity vector (shift amount) di (i = 1, r) representing the amount of deviation between the pixel on the reference image F (t) and the corresponding pixel on the viewpoint image Fi (t) (i = 1, r). ) Is derived from equations (27) and (28) obtained by modifying equations (25) and (26).

  Note that the variable μ in the equations (27) and (28) represents the pixel pitch. That is, a reference image F (t), a depth model D (t) that is a relative depth value, and a function (Z = z (D (t))) that converts the depth model D (t) into an absolute depth value Z are given. For example, each viewpoint image Fi (t) (i = 1, r) can be generated based on Expression (27) and Expression (28).

  Below, the viewpoint image generation part 40 is demonstrated based on the said view. FIG. 21 is a block diagram illustrating a configuration example of the viewpoint image generation unit 40. FIG. 22 is a flowchart for explaining an operation example of the viewpoint image generation unit 40. FIG. 23 is a diagram for explaining a viewpoint image generation example in the viewpoint image generation unit 40. First, as illustrated in FIG. 21, the viewpoint image generation unit 40 includes a disparity vector calculation unit 401, a texture shift unit 402, a gap filling unit (also referred to as an occlusion compensation unit) 403, and a floating window superimposing unit 404. .

  In FIG. 22, the disparity vector calculation unit 401 in FIG. 21 converts the input assumed viewing condition information, the depth model D (t), and the depth model D (t) into an absolute depth value (Z = z (D ( t))) and the disparity vectors di (i = i = i) from the equations (27) and (28) to the respective pixels on the reference image F (t) and the corresponding pixels on each viewpoint image Fi (t). l, r) is calculated, and the result is output to the texture shift unit 402 (step S51 in FIG. 22). In addition to calculating the disparity vector of each pixel based on Expressions (27) and (28), the disparity vector calculation method is based on the assumed viewing condition information in advance as shown in the LUT of FIG. You may calculate using the lookup table which derives a parallax vector from the depth value (relative depth value) set based on. Here, the spreading direction and the crossing direction of the disparity vector (shift amount) in FIG. 23B will be described with reference to FIG. In FIG. 24, let P be a certain point of interest, Pr be the point of interest P projected onto the display surface when viewed from the right eye, and Pl be the point of interest P projected onto the display surface when viewed from the left eye. At this time, as shown in FIG. 24A, the disparity vector in the spreading direction is such that a certain point of interest P is located behind the display surface, and the disparity vector from Pr to Pl on the display surface, or Pl This is a case where the value of the parallax vector from to Pr becomes positive. Similarly, as shown in FIG. 24B, the disparity vector in the cross direction is such that a certain point of interest P is located in front of the display surface, and the disparity vector from Pr to Pl on the display surface, or Pl to Pr. This is a case where the value of the parallax vector to becomes negative. When the value of the parallax vector is zero, the attention point P is located on the display surface.

  Subsequently, the texture shift unit 402 in FIG. 21 converts each pixel (x, y) of the reference image F (t) to each viewpoint image Fi (t) based on the corresponding disparity vector di (i = 1, r). ) (I = 1, r) and the corresponding pixel value of the pixel (u, v), and the generated viewpoint image is output to the gap filling unit 403 (step S52 in FIG. 22). When the pixel value is set, texture shift is performed from a pixel whose disparity vector value has a value on the spreading direction side (for example, d2 on the LUT in FIG. 23B).

  For example, in FIG. 23A, a case where a virtual left viewpoint viewpoint image Fl (t) (left eye presented image) is generated from the reference image iF and the depth model iD is considered. In the depth model iD, the depth value of the white portion is D1 and the depth value of the black portion is represented by D2. At this time, when texture shift is performed based on the LUT in FIG. 23B, first, each pixel of the layer L2 having the depth value D2 in FIG. 23A is shifted by d2 in the spreading direction. After that, when each pixel of the layer L1 having the depth value D1 in FIG. 23A is shifted by d1 in the intersecting direction, a defective region Gs1 that is not located at the left end / right end of the screen and a defective region Gl1 that is located at the left end / right end of the screen A viewpoint image oF1 having is obtained. Here, the missing area (occlusion area) represents an area in which no pixel value is set because there is no corresponding pixel on the reference image in the viewpoint image oF1 in FIG.

  Subsequently, the gap filling unit 403 in FIG. 21 in the input viewpoint image Fi (t) (i = 1, r), the missing region that is not located at the screen edge (for example, the viewpoint image oF1 in FIG. 23A). The pixel of Gs1) is interpolated from the pixel group located around the defect area, and the interpolated viewpoint image Fi (t) (i = 1, r) is output to the floating window superimposing unit 404 (step S53 in FIG. 22). . Note that, for example, linear interpolation, a median filter, or a known image restoration method (see, for example, Non-Patent Document 6) is used as a method for interpolating a pixel in a defective region.

  Subsequently, the floating window superimposing unit 404 of the input viewpoint images Fi (t) (i = 1, r) has a defect area located at the screen edge (for example, the viewpoint image oF1 in FIG. 23A). In Gl1), the maximum width W1 of the defect region is obtained. Subsequently, a floating window (black band) having a width W2 (= αW1) is inserted into the right end and the left end of each viewpoint image, and the result is output (step S54 in FIG. 22). W2 is a value obtained by scaling W1 with a predetermined constant α. Further, the viewpoint image after the floating window is inserted is, for example, the viewpoint image oF2 of FIG. In the viewpoint image oF2 in FIG. 23A, floating windows fw1 and fw2 are inserted at the left end and the right end of the screen, respectively. The reason for inserting the floating window is that the position or shape of a certain target is extremely different in the image presented to the left eye / right eye (for example, the generated viewpoint image is located at the left end / right end of the screen). This is to suppress a visual field struggle that alternately perceives left and right retinal images caused by the inability to see both eyes as one object.

  As described above, according to the present embodiment, when the vanishing point position can be estimated by the geometric depth cue, the depth model of the image is generated based on the vanishing point, so that the depth sensation by the geometric depth cue can be obtained. An enhanced stereoscopic image can be generated. In addition, when the vanishing point position cannot be estimated due to the geometric depth cue, the depth model of the image is generated from the saliency representing the attractiveness in the image based on the human visual characteristics, thereby A stereoscopic image in which the sense of depth is emphasized can be generated.

(Modification of depth model generation unit 30 (depth model generation unit 30a))
In the embodiment described above, the depth model generation unit 30 scales the saliency M (x, y) of each pixel according to the equation (24) as an example of the depth model creation unit based on the saliency, and becomes a reference depth value D. _{Although the case} where the value shifted by _base (x, y) is set as the depth value D (x, y) of each pixel has been described, the present invention is not limited to this. For example, the configuration of the depth model generation unit 30 may be changed so that the switching unit 301 is removed as illustrated in FIG. 25 and the image F (t) is input to the region division unit 303 and the saliency calculation unit 305. . That is, the depth model generation unit 30a includes a switching unit 302, an area dividing unit 303, a distance calculating unit 304, a saliency calculating unit 305, and a depth value setting unit 306.

  An example of the operation of the depth model creation means based on the saliency in this case will be described with reference to FIG. Note that the operation of the depth model creation means based on the vanishing point is the same processing as steps S42 to S45 in FIG.

  First, the saliency calculating unit 305 in FIG. 25 calculates the saliency M (t) from the processing target image F (t) by the same processing as step S47 in FIG. 15, and the result is sent to the depth value setting unit 306. This is output (step S47 ′ in FIG. 26). Subsequently, the area dividing unit 303 in FIG. 25 divides the processing target image F (t) into areas by processing similar to that in step S44 in FIG. 15, and sets area information R (t) describing the result to the depth value setting. It outputs to the part 306 (step S48 'of FIG. 26).

Thereafter, the depth value setting unit 306 in FIG. 25, based on the input saliency M (t) and the area information R (t), each area information R (t) indicates, as shown in Expression (29). The average value of the saliency M (x, y) of the pixels in the region is scaled, and the value shifted by the reference depth value D _base (x, y) is used as the depth value D (x, y) of each pixel. Setting is made (step S49 'in FIG. 26).

In Expression (29), D _max is the upper limit value of the depth value, D _min is the lower limit value of the depth value, M _max is the maximum value of the saliency M (t), and M _min is the minimum value of the saliency M (t). The value D _base (x, y) is a predetermined constant for adjusting the reference value of the depth value of each pixel (the depth value as the farthest view). Here, an example of the depth model based on the saliency in the modification is shown in FIG. In FIG. 27, image A represents an example of the processing target image F (t), and image B is an example of the region division result (region information M (t)) of the processing target image F (t) obtained by the region dividing unit 303. The image C represents the saliency M (t) of the processing target image F (t) obtained by the saliency calculating unit 305, the image D represents an example of a reference depth model (D _base ), and the image E Is an example of the depth model obtained by the depth value setting unit 306 based on the region information R (t) of the image B, the saliency M (t) of the image C, and the depth model (D _base ) as a reference of the image D. is there. In the image C of FIG. 27, a bright part (white) represents a part that is easily noticed by a person (high attraction), and a dark part (black) represents a part that is difficult for a person to pay attention (low attraction). In the image E of FIG. 27, the bright part represents the near side, and the dark part represents the back. In addition, regarding the reference depth model (D _base ), in the image D of FIG. 27, a plane having the same depth value is given as an example, but the present invention is not limited to this. For example, the plane equation shown in the following equation (30) may be determined in advance, and the reference depth value D _base (x, y) may be set by the coordinates (x, y) of each pixel. An example of the depth model serving as a reference expressed by the equation (30) is shown in an image A of FIG. Image A in FIG. 28 is a depth model that serves as a reference in which the coefficients a, b, and c in Expression (30) are set so that the depth is closer to the lower end and the depth is closer to the upper end. The result of the depth model D (t) created when the image A in FIG. 28 is input instead of the image D in FIG. 25 is shown in an image B in FIG.

  As described above, when the vanishing point position cannot be estimated due to the geometric depth cue, the depth model generation unit 30a and the saliency representing the attractiveness in the image based on the visual characteristics of the person and the image segmentation result ( By setting a uniform depth value for each region based on (region information), it is possible to generate a depth model that suppresses errors in the depth context.

(Modification of scene change detection unit 10)
In the above embodiment, the scene change detection unit 10 has been described using the luminance histogram as the image feature amount used for scene change detection, but the present invention is not limited to this. For example, instead of the luminance histogram, a color histogram representing the appearance frequency of each color component, an average error of inter-frame differences, and a motion vector distribution may be used as the image feature amount.

(Modification of the edge detection unit 211)
In the above embodiment, the edge detection unit 211 has described edge detection in which a point (Local Maxima) where the edge intensity is locally maximum in the image space is extracted as an edge point, but the present invention is not limited to this. For example, a known edge detection method such as Canny Edge detection may be used. Further, as a differential operator (edge detector), a known method such as a Sobel filter, a Prewitt filter, a LoG filter (Laplacian of Gaussian), or a DoG filter (Difference of Gaussian) is used. May be.

(Modification of Vanishing Point Identification Unit 213)
In the said embodiment, although the vanishing point identification part 213 demonstrated the case where Gaussian distribution was used as a distribution model used for a mixed model, this invention is not limited to this. For example, an exponential distribution family (Laplace distribution, beta distribution, Bernoulli distribution, etc.) may be used for the distribution model. The vanishing point identifying unit 213 may determine the number of classes Kc used in the mixed model as a predetermined value and determine the value as in the following example. The vanishing point identifying unit 213 sets a predetermined class number Kc ′ as the class number Kc, and performs clustering by the K-means method. Thereafter, when there is a class Ci and a class Cj where the distance between classes satisfies a predetermined threshold value or less (or less), the vanishing point identifying unit 213 merges the class Ci and the class Cj to form a new class Ck ′. Process. The vanishing point identifying unit 213 determines the number of classes Kc (≦ Kc ′) by repeating this process until the number of classes converges to a constant value. Note that the method used by the vanishing point identifying unit 213 to estimate the distribution model of the intersection is not limited to a parametric estimation method such as a mixed model, and the Mean-shift method, the K-means method, the K nearest neighbor search method (approximate K) A non-parametric estimation method such as a nearest neighbor search method may be used.

(Modification of the area dividing unit 303)
In the above embodiment, the case in which the region dividing unit 303 performs clustering in the feature amount space has been described, but the present invention is not limited to this, and clustering in the image space may be performed. Clustering in the image space is a method for performing region division based on the similarity between pixels or pixel groups (regions) constituting the region in the original image space without mapping to the feature amount space. is there. For example, the region dividing unit 303 uses (a) pixel combination method, (b) region growth method (also referred to as Region Growing method), and (c) region division integration method (also referred to as Split & Merge method) in the image space. Clustering may be performed.

(Modification of saliency calculation unit 305)
In the above embodiment, the case where the saliency calculating unit 305 calculates the saliency based on the local color difference and the global color difference has been described. However, the present invention is not limited to this, and the local color difference (expression ( The saliency may be calculated based on one of the indicators of the first term ΔE _local ) in 21) or the global color difference (second term ΔE _{global in} equation (21)). Alternatively, the color difference may be calculated using only the L * lightness index without using the red-green perceptual chromaticity a * and the yellow-blue perceptual chromaticity b *. In this case, in a human visual characteristic, a portion having a large brightness contrast (contrast difference) represents high attractiveness. Further, the local color difference ΔE _local and the global color difference ΔE _global are expressed by the following equation (31), using the lightness difference ΔL *, chroma difference ΔC *, and hue difference ΔH * based on the CIE method, respectively. You may obtain | require by Formula (32) (for example, refer nonpatent literature 4).

  Note that the coefficient w (u, v) in equation (31) is the same as in equation (12). Further, the coefficients l, c, h in the expressions (31) and (32) are predetermined weighting coefficients. The method of obtaining the saliency is not limited to the color difference, and the saliency may be calculated based on a plurality of image feature amounts such as a color difference, an edge gradient, and a motion vector (for example, see Non-Patent Document 5).

(First Modification of Stereoscopic Image Generation Device 1)
In the above embodiment, the input / output image size of the scene change detection unit 10, the vanishing point estimation unit 20, and the depth model generation unit 30 of the stereoscopic image generation device 1 is assumed to be the same as the input image F (t). However, the present invention is not limited to this. For example, in order to reduce the amount of computation and the memory size, images input to the scene change detection unit 10, the vanishing point estimation unit 20, and the depth model generation unit 30 are reduced in advance to a predetermined image size, and the depth You may add and implement the process which expands the depth model output from the model production | generation part 30 to input image size. That is, the first modification (stereoscopic image generating apparatus 2) of the stereoscopic image generating apparatus 1 includes a reduction processing unit 50, a scene change detecting unit 10, a vanishing point estimating unit 20, and a depth model generating unit as shown in FIG. 30, an enlargement processing unit 60, and a viewpoint image generation unit 40. The reduction processing unit 50 corresponds to reduced image generation means of the present invention, and generates a reduced image having a predetermined image size from the input image F (t). Then, the generated reduced image is input to the vanishing point estimation unit 20 and the depth model generation unit 30. The enlargement processing unit 60 corresponds to the enlarged depth model generation unit of the present invention, and generates an enlarged depth model having the same image size as the input image F (t) from the depth model of the reduced image generated by the depth model generation unit 30. .

  An example of the operation of the stereoscopic image generating apparatus 2 will be described with reference to FIG. Each operation (step S63, step S64, step S65, step S67 in FIG. 30) of the scene change detection unit 10, the vanishing point estimation unit 20, the depth model generation unit 30, and the viewpoint image generation unit 40 in FIG. Each operation of the scene change detection unit 10, the vanishing point estimation unit 20, the depth model estimation unit 30, and the viewpoint image generation unit 40 of the stereoscopic image generation apparatus 1 shown in FIG. 1 (step S12 in FIG. Since it is the same as S13, step S14, and step S15), description thereof is omitted.

  In FIG. 30, first, the stereoscopic image generating apparatus 2 in FIG. 29 outputs the input image at time t to the reduction processing unit 50 and the viewpoint image generating unit 40 (step S61 in FIG. 30).

  29 reduces the input processing target image F (t) to a predetermined image size, converts the reduced image Fd (t) to the scene change detection unit 10, the vanishing point estimation unit 20, and It outputs to the depth model production | generation part 30 (step S62 of FIG. 30). Note that image reduction is performed using, for example, a nearest neighbor method, a bilinear method, or a bicubic method.

  29 enlarges the input depth model D (t) to the image size of the input image F (t), and outputs the enlarged depth model Du (t) to the viewpoint image generation unit 40 (FIG. 30 step S66). The depth model is enlarged using, for example, the nearest neighbor method, the bilinear method, or the bicubic method.

  According to the stereoscopic image generating apparatus 2, scene change detection processing, vanishing point estimation processing, and depth model generation processing are performed using a reduced image having an image size smaller than the input image, and therefore, compared with the stereoscopic image generating apparatus 1 of FIG. 1. Thus, the memory size and the amount of calculation can be reduced.

(Second Modification of Stereoscopic Image Generation Device 1)
In the above embodiment, when the vanishing point position can be estimated by the geometric depth cue, the stereoscopic image generation apparatus 1 generates a depth model of the image based on the vanishing point and estimates the vanishing point position by the geometric depth cue. When it is not possible, the depth model of the image is generated based on the saliency representing the attractiveness in the image based on the visual characteristics of the person. For this reason, in the frames before and after the depth model generation unit is switched, the depth model is different in the time direction, so that the change in parallax (depth) is considered to be large. Similarly, in the frames before and after the scene change occurs, since the depth model is different in the time direction, it is considered that the change in parallax (depth) increases. Therefore, in the second modification of the stereoscopic image generating device 1 (stereoscopic image generating device 3 in FIG. 31), the spatio-temporal direction in which the depth model is smoothed in the spatiotemporal direction in order to reduce the change in parallax in the temporal direction. A smoothing unit 70 is provided between the depth model generation unit 30 and the viewpoint image generation unit 40. That is, the stereoscopic image generation device 3 includes a scene change detection unit 10, a vanishing point estimation unit 20, a depth model generation unit 30, a spatio-temporal direction smoothing unit 70, and a viewpoint image generation unit 40 as illustrated in FIG. Is done. The spatio-temporal direction smoothing unit 70 corresponds to the spatio-temporal direction smoothing means of the present invention, and includes a spatial direction smoothing unit 701, a time direction smoothing unit 702, and a buffer 703 as shown in FIG. The The spatio-temporal direction smoothing unit 70 smoothes the depth model D (t) of the processing target image F (t) generated by the depth model generation unit 30 in the spatial direction, and the image F (t) smoothed in the spatial direction. Based on the depth model Ds (t) and the depth model Dt (t−1) smoothed in the spatio-temporal direction of the comparison target image F (t−1) in the past from the image F (t), the image F The depth model Ds (t) of (t) is smoothed in the time direction, and the depth model Dt (t) smoothed in the spatio-temporal direction of the image F (t) is generated.

  An operation example of the stereoscopic image generation device 3 will be described with reference to FIGS. 33 and 34. FIG. In addition, each operation | movement (step S72, step S73, step S74, step S76 of FIG. 33) of the scene change detection part 10, the vanishing point estimation part 20, the depth model generation part 30, and the viewpoint image generation part 40 of FIG. Each operation of the scene change detection unit 10, the vanishing point estimation unit 20, the depth model estimation unit 30, and the viewpoint image generation unit 40 of the stereoscopic image generation apparatus 1 shown in FIG. 1 (step S12 in FIG. Since it is the same as S13, step S14, and step S15), description thereof is omitted.

(About the spatio-temporal direction smoothing unit 70)
The spatiotemporal direction smoothing unit 70 in FIG. 31 performs a smoothing process in the spatiotemporal direction on the depth model D (t) of the input processing target image F (t), and the result (smoothed depth model Dt ( t) is output (step S75 in FIG. 33).

  Specifically, the spatial direction smoothing unit 701 in FIG. 32 smoothes the depth model D (t) in the spatial direction using a one-dimensional smoothing filter in the order of the horizontal direction, the vertical direction, or the vertical direction and the horizontal direction. The result (depth model Ds (t)) is output to the time direction smoothing unit 702 (step S81 in FIG. 34). As the one-dimensional smoothing filter, for example, a one-dimensional Gaussian filter is used.

  The time direction smoothing unit 702 in FIG. 32 is based on the input depth model Ds (t) smoothed in the spatial direction and the smoothed depth model Dt (t−1) of the previous frame stored in the buffer 703. Then, the smoothed depth model Dt (t) is generated by the following equation (33), and the result is output to the buffer 703 and the outside (step S82 in FIG. 34). In addition, coefficient (alpha) in Formula (33) is a predetermined value between 0-1.

  The buffer 703 in FIG. 32 deletes the smoothed depth model Dt (t−1) of the previous frame and stores the input smoothed depth model Dt (t) of the current frame (step S83 in FIG. 34).

  According to the stereoscopic image generating device 3, by changing the depth model in the spatio-temporal direction, the change in parallax (depth) in the frames before and after the depth model generating means is switched and in the frames before and after the scene change occurs. Can be reduced.

  As mentioned above, although it demonstrated centering on each embodiment of the stereo image production | generation apparatus which concerns on this invention, this invention can also be made into the form of the stereo image production | generation method by the stereo image production | generation apparatus 1. FIG. Moreover, it is good also as a form of the program for making a computer perform this stereo image production | generation method.

  That is, you may make it implement | achieve a part of the stereo image production | generation apparatus 1 in embodiment mentioned above with a computer. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. Here, the “computer system” is a computer system built in the stereoscopic image generation apparatus 1 and includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In such a case, a volatile memory inside a computer system serving as a server or a client may be included and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  Moreover, you may implement | achieve part or all of the stereo image production | generation apparatus 1 in embodiment mentioned above as integrated circuits, such as LSI (Large Scale Integration). Each functional block of the stereoscopic image generating apparatus 1 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, in the case where an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology may be used.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

1, 2, 3, 3D image generation device, 10, scene change detection unit, 20, vanishing point estimation unit, 21, intra-frame vanishing point estimation unit, 22, interframe vanishing point estimation unit, 30, depth model generation unit, 40: viewpoint image generation unit, 50 ... reduction processing unit, 60 ... enlargement processing unit, 70 ... spatiotemporal direction smoothing unit, 101 ... luminance histogram generation unit, 102, 203, 204, 703 ... buffer, 103 ... histogram similarity Calculation unit 104... Scene change determination unit 201, 202, 301, 302... Switching unit 211 211 Edge detection unit 212 Straight line detection unit 213 Vanishing point identification unit 221 Feature point detection unit 222 Point calculation unit, 223 ... transformation matrix calculation unit, 224 ... vanishing point position calculation unit, 303 ... area division unit, 304 ... distance calculation unit, 305 ... saliency calculation unit, 306 ... depth value setting unit, 01 ... disparity vector calculation unit, 402 ... texture shift unit, 403 ... gap filling section, 404 ... floating window superimposing unit, 701 ... spatial direction smoothing unit, 702 ... time direction smoothing unit.

Claims (8)

A stereoscopic image generating apparatus that adds binocular stereoscopic information to a 2D image and generates a 3D image,
Vanishing point estimating means for estimating vanishing points from the processing target image;
A depth model generating means for generating a different depth model based on whether or not the vanishing point can be estimated by the vanishing point estimating means;
A viewpoint image generation unit that generates a right eye presentation image and a left eye presentation image based on the depth model generated by the depth model generation unit, the processing target image, and the assumed viewing condition information;
When the vanishing point can be estimated by the vanishing point estimating unit, the depth model generating unit generates a depth model based on the vanishing point, and when the vanishing point cannot be estimated by the vanishing point estimating unit, Generate a depth model based on the saliency of each pixel in the processing target image ,
The vanishing point estimating means includes an intra-frame vanishing point estimating means for estimating a vanishing point of the processing target image from straight line information in the processing target image, and a comparison target image that is earlier than the processing target image and the processing target image. An inter-frame vanishing point estimating unit that obtains a projective transformation matrix from a correspondence relationship between feature points between the image and a vanishing point of the processing target image based on the obtained projective transformation matrix and a position of the vanishing point in the comparison target image And
The stereoscopic image generating apparatus includes a scene change detection unit that detects whether or not a scene change has occurred between the processing target image and the comparison target image, and when a scene change is detected by the scene change detection unit, the select frame vanishing point estimation means, wherein the scene change when the scene change is not detected by the detecting means, the stereoscopic image generating apparatus according to claim you to select the vanishing point estimation means between said frame. Saliency of each pixel in the processed image, the pixel of interest and the local color difference multiplied by a predetermined weighting factor to the color difference with surrounding pixels, color Saha Rui the entire target pixel and the image is attention Calculated by the linear sum of the color difference between the local area including the pixel and the surrounding area multiplied by a predetermined weighting factor ,
The color difference, brightness index, red - green perception chromaticity and yellow - blue stereoscopic image generation apparatus according to claim 1, based on each difference of the perceptual chromaticity defined characterized Rukoto of. The depth model generation means, if the vanishing point is estimated by the vanishing point estimating means, and the distance determined from the coordinates of each pixel in position and the processing target image of the vanishing point, the area division information of the processing target image Based on the above, the average value of the distance in each region indicated by the region division information is obtained, and the depth value of each pixel in the processing target image is a value obtained by normalizing the average value by the first weighting factor. And the sum of the specified depth value,
The first weighting factor is obtained by dividing the difference between the upper limit value and the lower limit value of the depth value by the difference between the upper limit value and the lower limit value of the distance,
When the vanishing point is not estimated by the vanishing point estimation unit, the saliency in each region indicated by the region division information is based on the saliency of each pixel in the processing target image and the region division information of the processing target image. Obtaining an average value of the degree, and setting a depth value of each pixel in the processing target image as a sum of a value obtained by normalizing the average value by a second weighting factor and a predetermined depth value;
The second weighting coefficient, the difference between the upper limit and the lower limit of the depth values, according to claim 2, wherein said is obtained by dividing the difference between the upper limit and the lower limit of the saliency Stereoscopic image generation device. A storage unit that stores vanishing point information including the position of the vanishing point of the comparison target image, and the vanishing point information of the comparison target image is stored in the storage unit; is, if the vanishing point information of the comparison target image in the storage means is not stored, according to claim 1, wherein the frame vanishing point estimation means is selected Stereo image generation device. The stereoscopic image generation apparatus according to claim 4 , wherein the comparison target image is an image immediately before the processing target image. A stereoscopic image generation method by a stereoscopic image generation apparatus that adds binocular stereoscopic information to a 2D image and generates a 3D image,
The three-dimensional image generation device, a vanishing point estimation step for estimating a vanishing point from the processing target image,
A depth model generation step for generating a different depth model based on whether or not the vanishing point could be estimated in the vanishing point estimation step;
A viewpoint image generation step of generating a right eye presentation image and a left eye presentation image based on the depth model generated in the depth model generation step, the processing target image, and the assumed viewing condition information;
In the depth model generation step, when the vanishing point can be estimated in the vanishing point estimation step, a depth model is generated based on the vanishing point, and the vanishing point cannot be estimated in the vanishing point estimation step. A depth model is generated based on the saliency of each pixel in the processing target image ,
The vanishing point estimating step includes an intra-frame vanishing point estimating step of estimating a vanishing point of the processing target image from straight line information in the processing target image;
A projection transformation matrix is obtained from the correspondence relationship of the feature points between the processing target image and a comparison target image that is earlier than the processing target image, and the calculated projection transformation matrix and the position of the vanishing point in the comparison target image are obtained. An inter-frame vanishing point estimating step for estimating a vanishing point of the processing target image,
The stereoscopic image generation apparatus further includes a scene change detection step for detecting whether or not a scene change has occurred between the processing target image and the comparison target image, and the scene change is detected in the scene change detection step. was case, selects the frame vanishing point estimation step, when said scene change by the scene change detection step is not detected, the stereoscopic image generation method comprising you to select the vanishing point estimation step between the frames. A program for causing a computer to execute the stereoscopic image generation method according to claim 6 . A computer-readable recording medium on which the program according to claim 7 is recorded.