JP4851563B2 - Video encoding method, video decoding method, video encoding program, video decoding program, and computer-readable recording medium on which these programs are recorded - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology with which a required video image can be decoded without the need for using video images of all cameras in a referencing relation to each other in adopting a configuration of efficiently coding a multi-viewpoint moving picture through the use of parallax direction prediction and temporal direction prediction at the same time. <P>SOLUTION: A coder side codes a reference viewpoint image photographed by a camera at a reference viewpoint, a distance image denoting an estimate distance from the camera for photographing the reference viewpoint image to an object, and a parallax difference image denoting the difference between a coding object image and a parallax compensation image estimated on the basis of a positional relation among the reference viewpoint image, the distance image and the camera. A decoder side decodes the reference viewpoint image, the distance image, and the parallax difference image, estimates a parallax compensation image on the basis of the positional relation among the decoded reference viewpoint image, the decoded distance image, and the camera, and restores coding object images photographed by cameras cross-referenced with viewpoints other than the reference viewpoint on the basis of the estimated parallax compensation image and the decoded parallax difference image. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to a video encoding method for encoding images captured by a plurality of cameras that capture a certain subject, a video encoding program used to realize the video encoding method, and a computer-readable recording of the program And a video decoding method for decoding data encoded by the video encoding method, a video decoding program used for realizing the video decoding method, and a computer-readable recording medium recording the program .

  A multi-view video (multi-view video) refers to a plurality of videos obtained by shooting the same subject and background with a plurality of cameras. Hereinafter, a moving image captured by one camera is referred to as a “two-dimensional moving image”, and a two-dimensional moving image group in which the same subject and background are captured is referred to as a multi-viewpoint moving image.

  The two-dimensional moving image of each camera included in the multi-viewpoint moving image has a strong correlation with respect to the time direction. On the other hand, when the cameras are synchronized, the images of the cameras corresponding to the same time are obtained by photographing the subject and the background in exactly the same state from different positions, and thus there is a strong correlation between the cameras. Hereinafter, the correlation between the cameras is referred to as a spatial correlation.

  In the encoding of moving images, the encoding efficiency is improved by using these correlations.

  First, a conventional technique related to a two-dimensional video encoding technique will be described.

  In the encoding of a two-dimensional moving image, only the difference between an image to be encoded and an image that has already been encoded is used as an encoding target, thereby utilizing the correlation in the time direction and improving the encoding efficiency. .

  H., an international encoding standard. In many conventional two-dimensional video encoding systems such as H.264, MPEG-2, and MPEG-4, encoding efficiency is further improved by using a technique called motion compensation when obtaining a difference. Motion compensation is a technique in which an image is divided into smaller blocks, and blocks that take a difference are switched for each block. As a result, even when the subject moves or the camera moves, the difference can be reduced and the encoding efficiency can be increased.

  Next, a conventional multi-view video encoding method will be described.

  In conventional multi-view video coding, in order to increase the coding efficiency by using the correlation in the time direction and the spatial direction, coding with prediction in the time direction and compensation between cameras is adopted. . As an example, there is a method of Non-Patent Document 1 shown below.

  Compensation between cameras performed by the method of Non-Patent Document 1 is called parallax direction prediction, and performs motion compensation using an image of another camera as a reference image. In this method, considering the coding efficiency, it is possible to select which correlation is used for compensation for each macroblock. Therefore, since the correlation in the time direction and the spatial direction is used for encoding, it is possible to improve the encoding efficiency as compared with the method using the correlation only in the time direction.

Hideaki Kimata and Masaki Kitahara, "Preliminary results on multiple view video coding (3DAV)," document M10976 MPEG Redmond Meeting, July, 2004.

  Certainly, in the method of Non-Patent Document 1, although the encoding efficiency can be improved as compared with the method using the correlation only in the time direction, the method of referring to the images of a plurality of cameras is used. Therefore, when decoding the data, the images of all the cameras having a reference relationship are required.

  For example, when there is a reference relationship as shown in FIG. 19, it is necessary to pass video from all cameras to the decoding side. This means that in the free viewpoint moving image synthesis, which is one of the utilization methods of multi-viewpoint moving images, it is necessary to decode even the video of the camera that is not necessary. Considering that data is communicated over a network, it becomes necessary to send information to the other party that is not necessary.

  The free-viewpoint video synthesis described here is a technique for synthesizing video from a point where a camera is not placed using video from an adjacent camera. Therefore, there is a need for a function for encoding the video from each camera so that it can be taken out independently.

  Also, when considering the synthesis of free-viewpoint video as a method of using multi-viewpoint video, the synthesis processing of free-viewpoint video is a process with a large amount of computation. It can be said that it is a load. Therefore, a function capable of assisting the synthesis process in the encoding / decoding process is also required.

  As an encoding method for realizing the function of independently extracting the video of each camera as necessary, a method of limiting the reference relationship in the technique as in Non-Patent Document 1 so that only the video of a specific camera can be referred to Can be considered. If this method is used, the reference relationship is limited, so that it is possible to reduce the number of camera images necessary for decoding data.

  However, with such a method, camera information can be individually extracted only in units set on the encoding side. Also, it is obviously not efficient to encode each camera video as one or more bitstreams, so it can be used even if there is a strong correlation between certain cameras by limiting the reference relationship. It may disappear. That is, since the correlation cannot be removed, it is difficult to achieve sufficient coding efficiency as a whole.

  Further, according to the following references, there are two methods for synthesizing a free viewpoint video from multi-view video, a method using the geometric information of the subject, and a video without using the geometric information of the subject. There is a method to synthesize.

References: Heung-Yeung Shum, Sing Bing Kang, and Shing-Chow Chan, "Survey of image-based representations and compression techniques," IEEE Tran sactions on Circuits and Systems for Video Technology, vol.13, no. 11, Nov .2003, pp.1020-1037.
In order to synthesize a video having the same quality as that using the geometric information without using the geometric information, more videos from the camera are required as input. Using many camera images means that a great deal of computation is required.

  In this regard, in the method as described in Non-Patent Document 1, information of each pixel is expressed only in the form of a residual signal and either a motion vector from the previous frame or a parallax vector from another camera. Therefore, when a free viewpoint moving image is synthesized using a multi-view moving image encoded by a technique such as that described in Non-Patent Document 1, it is necessary to calculate geometric information, which increases the amount of calculation and takes time to synthesize. Or more video from more cameras.

  The present invention has been made in view of such circumstances, and has a configuration in which the encoding efficiency of a multi-view video can be improved by simultaneously using the parallax direction prediction and the temporal direction prediction. A new video coding and decoding technology that enables the synthesis of free-viewpoint video with a small load by enabling the necessary video to be decoded without using the video of all cameras in a reference relationship. For the purpose of provision.

[1] Basic Configuration of Video Encoding Method of the Present Invention To achieve this object, the video encoding method of the present invention encodes images captured by a plurality of cameras that capture a certain subject. (A) a reference viewpoint image encoding step for encoding a reference viewpoint image captured by a camera serving as a reference viewpoint, and (b) an estimated distance from the camera that captured the reference viewpoint image to the subject. A distance image generating step that generates a distance image to be shown; (c) a distance image encoding step that encodes the generated distance image; and (d) a camera that defines the reference viewpoint image, the distance image, and the installation position and orientation of the camera. Based on the positional relationship, a parallax compensated image (what becomes a predicted image based on parallax) at a viewpoint other than the reference viewpoint is estimated, and at this time, the pixel value cannot be estimated. For a prime, parallax compensation that estimates the motion information of the pixel from the motion information of neighboring pixels and estimates the pixel value of the pixel based on the estimated motion information and the pixel value of the encoded image An image estimation step; (e) a parallax difference image calculation step for calculating a parallax difference image indicating a difference between the estimated parallax compensation image and an encoding target image captured by a camera associated with the estimation target viewpoint; f) a parallax difference image encoding step for encoding the calculated parallax difference image.

  In adopting this basic configuration, the video encoding method of the present invention may further adopt the following configuration.

[1-1]
In the parallax compensation image estimation step, when the decoding side can obtain the positional relationship information of the camera instead of the encoded data, the reference viewpoint image obtained by decoding the encoded data of the reference viewpoint image; A parallax compensation image at a viewpoint other than the reference viewpoint may be estimated based on the distance image obtained by decoding the encoded data of the distance image and the positional relationship of the cameras that are not encoded.

[1-2]
When the decoding side obtains information on the camera positional relationship from the encoded data, in addition to the basic configuration steps described above, (g) acquire the camera positional relationship according to information from the outside, A camera positional relationship setting step for setting the positional relationship of the cameras by estimating the positional relationship of the cameras based on the images of all the cameras; and (h) camera positional relationship information for encoding the information on the positional relationship of the set cameras. Encoding step.

[1-3]
In the parallax compensation image estimation step, when the decoding side obtains information on the positional relationship of the camera from the encoded data, the reference viewpoint image obtained by decoding the encoded data of the reference viewpoint image, and the distance image Based on the distance image obtained by decoding the encoded data and the camera positional relationship obtained by decoding the encoded data of the camera positional relationship information, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated. Sometimes.

[1-4]
In the case of automatically determining the camera to be the reference viewpoint, in addition to the basic configuration steps described above, (i) the camera that captures the space most overlapping with the space captured by the other camera is used as a reference. There may be a step of setting as a camera as a viewpoint.

[1-5]
In the distance image generation step, a distance image may be generated by dividing the image into blocks and estimating the distance for each block.

[1-6]
In the distance image generation step, when a distance image is generated according to a prescribed algorithm, a difference value between the evaluation value of the distance image generated at the current time and the evaluation value of the distance image generated at the previous time is obtained. When the difference value is determined by comparing the difference value with a predetermined threshold value and the difference value is determined to be large, it is determined that the distance image generated at the current time is used as it is, and the difference is determined. When it is determined that the value is small, a distance image may be generated by determining to use the distance image generated at the previous time.

[1-7]
In the parallax compensation image estimation step, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated based on the reference viewpoint image, the distance image, and the positional relationship between the cameras. It is also possible to perform processing so as to estimate the pixel value of the pixel from the pixel values of the surrounding pixels.

[1-8]
Actually, in the parallax compensation image estimation step, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated based on the reference viewpoint image, the distance image, and the positional relationship between the cameras. At this time, the pixel value is estimated. For a pixel that cannot be used, the motion information of the pixel is estimated from the motion information of surrounding pixels, and the pixel value of the pixel is estimated based on the estimated motion information and the pixel value of the encoded image. become.

[1-9]
In the parallax compensation image estimation step, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated based on the reference viewpoint image, the distance image, and the positional relationship between the cameras. The amount of code when using the parallax compensated image estimated according to the method [1-7] is compared with the amount of code when using the parallax compensated image estimated according to the method [1-8]. It is also possible to perform processing so as to estimate the pixel value of the pixel by selecting a method that enables efficient encoding. When this configuration is adopted, information indicating which prediction mode is used is also encoded.

[1-10]
In the distance image encoding step, the distance image may be encoded using a motion vector used when encoding the reference viewpoint image.

[1-11]
In the parallax difference image encoding step, the motion vector estimated when encoding the reference viewpoint image, the motion vector estimated based on the distance image and the positional relationship between the cameras, or the motion estimated from its own reference image Of the vectors, the one with the best coding efficiency may be selected and coded.

[1-12]
In the parallax difference image encoding step, the motion vector estimated when encoding the reference viewpoint image, the motion vector estimated based on the distance image and the positional relationship between the cameras, or the motion estimated from its own reference image The vector with the most efficient encoding may be selected and encoded. When estimating this motion vector, the motion using the distance image obtained by decoding the encoded data of the distance image is used. The vector may be estimated.

  Here, the video encoding method of the present invention configured as described above can also be realized by a computer program. The computer program is provided by being recorded on a suitable computer-readable recording medium or via a network. The present invention is realized by being installed when operating the present invention and operating on a control means such as a CPU.

[2] Basic Configuration of Video Decoding Method of the Present Invention The video decoding method of the present invention includes a plurality of cameras that capture a certain subject by decoding encoded data generated by the video encoding method of the present invention. (A) a reference viewpoint image decoding step for decoding encoded data of a reference viewpoint image captured by a camera serving as a reference viewpoint, and (b) a reference viewpoint. A distance image decoding step of decoding encoded data for a distance image indicating an estimated distance from the camera that has captured the image to the subject; and (c) an encoding target image captured by a camera associated with a viewpoint other than the reference viewpoint; A parallax difference image decoding step for decoding encoded data of the parallax difference image indicating the difference between the decoded reference viewpoint image, the decoded distance image, and the camera. Based on the positional relationship of the camera that defines the installation position and orientation of the camera, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated. At this time, the pixel value cannot be estimated from the motion information of surrounding pixels. A parallax compensation image estimation step of estimating pixel motion information, and estimating the pixel value of the pixel based on the estimated motion information and the pixel value of the decoded image; and (e) the estimated parallax compensation image. And an encoding target image restoration step for restoring an encoding target image captured by a camera associated with a viewpoint other than the reference viewpoint based on the decoded parallax difference image.

  In adopting this basic configuration, the video decoding method of the present invention may further adopt the following configuration.

[2-1]
When the camera positional relationship information is obtained from the encoded data, in addition to the basic configuration steps described above, (f) encoded data regarding the positional relationship information of the camera that captured each image. The camera positional relationship information decoding step for decoding

[2-2]
In the parallax compensation image estimation step, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated based on the decoded reference viewpoint image, the decoded distance image, and the positional relationship between the cameras. At this time, the pixel value is estimated. For pixels that cannot be processed, it is also possible to process so as to estimate the pixel value of the pixel from the pixel values of the surrounding pixels.

[2-3]
Actually, in the parallax compensation image estimation step, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated based on the decoded reference viewpoint image, the decoded distance image, and the positional relationship between the cameras. For a pixel whose pixel value cannot be estimated, the motion information of the pixel is estimated from the motion information of surrounding pixels, and the pixel value of the pixel is calculated based on the estimated motion information and the pixel value of the decoded image. Will be estimated.

[2-4]
In the parallax compensation image estimation step, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated based on the decoded reference viewpoint image, the decoded distance image, and the positional relationship between the cameras. At this time, the pixel value is estimated. For pixels that cannot be processed, the parallax compensation image is selected by selecting either the estimation method [2-2] or the estimation method [2-3] based on the prediction mode information embedded in the encoded data. It is also possible to perform processing so as to estimate the pixel value while switching between the two methods in units of.

  Here, the video decoding method of the present invention configured as described above can also be realized by a computer program, and this computer program is provided by being recorded on an appropriate computer-readable recording medium or via a network. The present invention is realized by being provided and installed on implementing the present invention and operating on a control means such as a CPU.

[3] Regarding the processing of the present invention The video encoding method of the present invention was taken by the camera serving as the reference viewpoint when the decoding side can obtain the positional information of the camera instead of the encoded data. A reference viewpoint image, a distance image indicating an estimated distance from the camera that captured the reference viewpoint image to the subject, a parallax compensation image estimated based on the reference viewpoint image, the distance image, and the positional relationship between the camera and the encoding target image Are encoded three types of images called parallax difference images. When the decoding side obtains information on the positional relationship of the camera from the encoded data, the information on the positional relationship of the camera is encoded in addition to the three types of images.

  Receiving this encoded data, the video decoding method of the present invention is obtained by decoding the reference viewpoint image, the distance image, and the parallax difference image, and the positional relationship between the decoded reference viewpoint image, the decoded distance image, and the camera. Based on the above, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated, and a code captured by a camera associated with a viewpoint other than the reference viewpoint is based on the estimated parallax compensation image and the decoded parallax difference image Restore the target image.

  As described above, according to the present invention, a distance image is created by obtaining a distance from one viewpoint (reference viewpoint) to a subject in a multi-view video to be encoded using the multi-view video. Next, a moving image at another viewpoint is predicted using a moving image at the reference viewpoint and a moving image of depth information (distance moving image). Then, by obtaining the difference between the predicted moving image and the moving image to be encoded, a difference moving image is obtained, and the moving image of the reference viewpoint, the moving image of the depth information at the reference viewpoint (distance moving image), the reference viewpoint The difference moving images at the viewpoints other than are encoded as two-dimensional moving images.

  That is, in the present invention, since only the reference viewpoint is used for reference, it is possible to realize a function of taking out information other than necessary cameras as much as possible. In addition, since the moving image of depth information (distance moving image) is created from information of all viewpoints, the correlation in the spatial direction can be used by predicting using the information.

  Furthermore, because the prediction residual is information that cannot be predicted from the reference viewpoint, it is not included in the image of the reference viewpoint, but information that is included only in that viewpoint remains. Coding efficiency can be improved by using the correlation in the time direction.

  Furthermore, the depth information moving image (distance moving image) not only represents the parallax information between the reference viewpoint and all other viewpoints in one expression, but also encoded it as a two-dimensional moving image. By doing so, it is possible to remove the redundancy of the parallax in the time direction.

  Also, when composing free viewpoint video from multi-view video, the number of images required to create a certain quality video can be reduced due to the existence of the geometric information of the subject, and the compositing process is simplified. However, since the depth moving image (distance moving image) required in the present invention represents a kind of geometric information, the quality of the synthesized image is improved, and the geometric information estimation process is simplified. It will be possible.

  As described above, according to the present invention, it is possible to realize a function of extracting an image of an arbitrary camera required by the receiver while suppressing transmission of unnecessary camera information as much as possible. And since it is encoded using the estimated distance information of the subject from a certain viewpoint, the geometric information of the subject is provided, and there is an advantage that the preprocessing for synthesizing the free viewpoint moving image can be omitted. In addition, by simultaneously using the correlation in the spatial direction and the correlation in the temporal direction, the multi-view video can be efficiently encoded.

  Next, the meaning of each processing function described in the above [1-1] to [1-12] of the video encoding method of the present invention will be described.

  (A) Meaning of the processing function described in [1-1]

  The distance image and the reference viewpoint image that can be obtained by the decoding side are images obtained by decoding the encoded image. Therefore, the encoding side estimates using the original distance image and the reference viewpoint image that are not encoded / decoded. If the parallax compensated image is used, there is an error between the parallax compensated image estimated on the decoding side, and the image other than the reference viewpoint is the error of the parallax difference image, the error of the reference viewpoint image, and the distance image. The error will overlap.

  Therefore, in the present invention, as described in [1-1], when the decoding side can obtain the positional information of the camera instead of the encoded data, the encoding side Based on the reference viewpoint image obtained by decoding the encoded data, the distance image obtained by decoding the encoded data of the distance image, and the positional relationship of the cameras that are not encoded, Processing is performed so as to estimate a parallax compensation image at a viewpoint other than the above.

  In accordance with this processing function, the encoding side uses a distance image including a coding distortion and a reference viewpoint image, so that the coding distortion in an image other than the reference viewpoint is only affected by the coding distortion in the parallax difference image. Will be able to. That is, in encoding of the parallax difference image, it is possible to perform encoding considering the encoding distortion of the distance image and the reference viewpoint image on the decoding side, so that encoding without distortion can be achieved at the maximum. Become.

  (B) Meaning of the processing function described in [1-2]

  When the positional relationship of the camera is not explicitly given, the accuracy of the process of obtaining the distance image from the multi-viewpoint moving image and obtaining the parallax compensation image is significantly reduced. For this reason, there are many residuals in the parallax difference image, resulting in poor coding efficiency.

  Therefore, in the present invention, as described in [1-2], when the decoding side obtains information on the camera positional relationship from the encoded data, and the camera positional relationship is given from the outside. Obtains and encodes it, and if the camera's positional relationship is not given from the outside, estimates the camera's positional relationship based on the images of all cameras and encodes it To do.

  According to this processing function, when it is necessary to encode the positional relationship of the camera, when the positional relationship of the camera is not given from the outside, in order to estimate the positional relationship of the camera from the given multi-view video, Since it is possible to obtain a more accurate distance image and parallax compensation image, it is possible to reduce the signal remaining in the parallax difference image, resulting in poor encoding efficiency even if the positional relationship of the camera is not explicitly given. Can be prevented. In other words, even when the positional relationship of the cameras is not explicitly given, encoding can be performed in a more flexible manner with respect to multi-view video images with various camera arrangements.

  (C) Meaning of the processing function described in [1-3]

  The meaning of the processing function described in [1-3] is basically the same as the meaning of the processing function described in [1-1]. However, since the processing function described in [1-3] assumes that information on the positional relationship of the camera is also encoded, the positional relationship of the camera necessary for estimating the parallax compensation image is also Processing is performed such that data obtained by decoding the encoded data of the camera positional relationship information is used.

  (D) Meaning of the processing function described in [1-4]

  Even though each camera captures almost the same subject and background, it does not shoot from the exact same viewpoint, so the space captured by each camera is different. For this reason, the size of the region where the parallax prediction with high accuracy is different depends on which camera pair performs the parallax prediction. If a camera with a small area where parallax prediction with high accuracy is small is used as a reference viewpoint, the encoding efficiency is deteriorated.

  Therefore, in the present invention, as described in [1-4], processing is performed so that the camera that captures the space most overlapping with the space captured by the other camera is set as the reference viewpoint camera. .

  According to this processing function, the camera that picks up more common parts of the images taken from each camera is selected as the reference viewpoint, so the total of the areas that can be accurately predicted for all cameras is maximized. Therefore, it is possible to prevent the encoding efficiency from deteriorating. That is, since a highly accurate parallax compensation image can be generated for many regions, more efficient encoding can be performed.

  (E) Meaning of the processing function described in [1-5]

  In the present invention, it is necessary to encode a distance image in addition to a multi-view video that is originally an encoding target. Therefore, it is necessary to reduce the amount of codes necessary for encoding the distance image as much as possible. Since the distance image exists to provide the parallax information and create the parallax compensation image, it is only necessary to include information necessary for that purpose. That is, the distance image may be of a size that can provide disparity information in units of pixels. In such an accurate distance estimation, the estimated distances in adjacent pixels are often the same.

  Therefore, in the present invention, as described in [1-5], the image is divided into blocks, and the distance image is generated by estimating the distance for each block.

  According to this processing function, more efficient encoding can be performed in encoding of a distance image while maintaining a certain degree of accuracy.

  (F) Meaning of the processing function described in [1-6]

  When the distance prediction is performed so that the error between the parallax-predicted image and the encoding target image is simply reduced, the correlation in the time direction is not taken into account in the distance prediction. Correlation in the time direction in the parallax compensated image is lost. If there is no temporal correlation in the parallax-predicted parallax-compensated image, the temporal correlation will be lost in the parallax difference image obtained from the difference between it and the encoding target image. In such a case, the encoding efficiency of the parallax difference image becomes very poor.

  Therefore, in the present invention, as described in [1-6], when a distance image is generated according to a prescribed algorithm, the encoding efficiency of the parallax difference image that is calculated based on the distance image is improved. If so, processing is performed so as to determine whether the generated distance image is used as it is or after being changed to one that does not change with time. Here, when the distance image is encoded in units of blocks, this determination is performed in units of the blocks.

  According to this processing function, the distance is estimated in consideration of the correlation in the time direction in the parallax difference image, so that it is possible to prevent the time direction correlation from being lost in the parallax difference image. Thus, more efficient encoding can be performed.

  (G) Meaning of the processing function described in [1-7]

  When the parallax compensation is performed, the reference viewpoint image and the image at another viewpoint do not necessarily correspond one-to-one. Therefore, pixels having no predicted value appear in the parallax compensation image. In this case, the parallax difference image becomes an image having a greatly different value between pixels. Since such an image has a different property from a natural image, the encoding efficiency decreases when a general encoding method is used.

  Therefore, in the present invention, as described in [1-7], for a pixel whose pixel value cannot be estimated when estimating a parallax compensation image, the pixel value of the pixel is estimated from the pixel values of surrounding pixels. Is processed.

  According to this processing function, by generating a predicted value from an adjacent pixel for a pixel having no parallax compensation value, it is possible to generate a parallax difference image whose value does not differ greatly between pixels. It is well known that there is a correlation between adjacent pixel values, and since this is also used in the encoding of a two-dimensional moving image, this prediction is unlikely to deviate greatly. Therefore, the encoding efficiency can be increased by reducing the encoding target information in the parallax difference image.

  (H) Meaning of the processing function described in [1-8]

  In the processing function described in [1-7], a pixel value cannot be estimated when estimating a parallax compensation image, and the pixel value is estimated by performing spatial prediction.

  On the other hand, in the processing function described in [1-8], for a pixel whose pixel value cannot be estimated when estimating a parallax compensation image, the motion information of the pixel is estimated from the motion information of the surrounding pixels. Based on the estimated motion information and the pixel value of the encoded image, processing is performed to estimate the pixel value of the pixel.

  In this processing function, since the correlation of motion vectors in adjacent pixels used in general video encoding is used, this prediction is unlikely to deviate greatly. Therefore, similarly to the processing function described in [1-7], the encoding efficiency can be increased by reducing the encoding target information in the parallax difference image.

  (I) Meanings of the processing functions described in [1-9]

  In the processing function described in [1-7], for a pixel whose pixel value cannot be estimated when estimating a parallax compensation image, the pixel value is estimated by performing spatial prediction, In the processing function described in [8], the pixel value is estimated by performing temporal prediction on a pixel whose pixel value cannot be estimated when the parallax compensation image is estimated.

  It is desirable to select which prediction method to use from the viewpoint of coding efficiency.

  Therefore, in the present invention, as described in [1-9], the code amount in the case of using the parallax compensation image estimated according to the estimation method of [1-7] and the estimation based on the estimation method of [1-8] By comparing the amount of code when using a parallax compensated image and selecting a method that allows efficient coding for each parallax compensated image, processing is performed to estimate the pixel value of that pixel. When this configuration is adopted, information indicating which prediction mode is used is also encoded.

  According to this processing function, a parallax compensation image is estimated by creating a prediction value for a pixel having no parallax compensation value, and thereby encoding efficiency can be increased by reducing encoding target information in the parallax difference image. Therefore, the improvement of the encoding efficiency can be further ensured.

  (J) Meaning of the processing function described in [1-10]

  Since both the reference viewpoint image and the distance image are images from the same viewpoint, it can be said that there is a very strong correlation with respect to motion. Therefore, if both are encoded independently, considerable redundancy remains.

  Therefore, in the present invention, as described in [1-10], processing is performed so that the distance image is encoded using the motion vector used when the reference viewpoint image is encoded.

  According to this processing function, it is possible to improve the encoding efficiency by avoiding redundant encoding of vectors representing the same motion. In addition, when decoding an image of a viewpoint other than the reference viewpoint, both the reference viewpoint and the distance image are always required. Therefore, even if there is a reference relationship between them, information other than the necessary camera realized by the present invention can be obtained. It does not impair the function of taking out as much as possible. Therefore, it is possible to increase the coding efficiency of the reference viewpoint image and the distance image by using the correlation between the reference viewpoint image and the distance image.

  (K) Meaning of the processing function described in [1-11]

  The fact that the same subject is being photographed means that the actual three-dimensional movement of the subject is one. Therefore, it is possible to predict the motion at another viewpoint to some extent from the positional relationship of the camera, the distance image, and the motion vector at the reference viewpoint. However, since the motion vector at the reference viewpoint is two-dimensional and the distance image is not perfect three-dimensional, the prediction vector is not always correct. In terms of coding efficiency, the use of motion vectors representing actual motion does not always achieve the highest coding efficiency.

  Therefore, in the present invention, as described in [1-11], when encoding the parallax difference image, the motion vector, the distance image, and the positional relationship between the camera used for encoding the reference viewpoint image One of the motion vectors estimated based on the reference image or the motion vector estimated from its own reference image is selected and encoded.

  According to this processing function, by selecting the motion vector obtained by closing each viewpoint and the motion vector obtained by parallax compensation, which has the highest coding efficiency, a high code that allows a motion vector error obtained by parallax compensation is allowed. It is possible to achieve efficiency. That is, it is possible to increase the coding efficiency of the parallax difference image by using the correlation of the motion components included in the moving image at each viewpoint, which comes from shooting the same subject.

  (L) Meaning of the processing function described in [1-12]

  In the processing function described in [1-11], the motion vector is estimated based on the motion vector used when the reference viewpoint image is encoded, the distance image, and the positional relationship between the cameras. In the processing function described in -12], in estimating the motion vector, the motion vector is estimated using the distance image obtained by decoding the encoded data of the distance image in accordance with the processing on the decoding side. To process.

  According to this processing function, since the motion vector is estimated by the same method as that on the decoding side, it is possible to estimate the motion vector by removing the influence of the coding distortion included in the distance image. It becomes possible to prevent accumulation.

  According to the present invention, it is possible to improve the encoding efficiency of a multi-view video by using the parallax direction prediction and the temporal direction prediction at the same time. By using the image and the distance image at the reference viewpoint, it is possible to realize a function of extracting an image of an arbitrary camera required by the receiver while suppressing transmission of unnecessary camera information as much as possible.

  Then, since encoding is performed using estimated distance information of the subject from a certain viewpoint, geometric information of the subject is provided, and the amount of processing for estimating the geometric information necessary to generate the free viewpoint moving image Can be reduced, and the number of videos required to generate a free viewpoint moving image with the required quality can be reduced.

It is a figure which shows an example of a camera structure. 1 is an embodiment of a video encoding device according to the present invention. It is a figure which shows the detail of a structure of a camera information initial setting part. It is a figure which shows the detail of a structure of a reference | standard viewpoint moving image processing part. It is a figure which shows the detail of a structure of a distance image process part. It is a figure which shows the detail of a structure of a non-reference | standard viewpoint moving image process part. It is a processing flow which the video coding apparatus of this invention performs. It is a processing flow which the video coding apparatus of this invention performs. It is a processing flow which the video coding apparatus of this invention performs. It is a processing flow which the video coding apparatus of this invention performs. It is a processing flow which the video coding apparatus of this invention performs. It is explanatory drawing of the projection process of a reference | standard viewpoint image. It is a figure which shows the kind of data which the video coding apparatus of this invention encodes, and the order of the encoding. It is an example of one Embodiment of the video decoding apparatus of this invention. It is a figure which shows the detail of a structure of a non-reference | standard viewpoint moving image decoding part. It is a processing flow which the video decoding apparatus of this invention performs. It is a processing flow which the video decoding apparatus of this invention performs. It is a processing flow which the video decoding apparatus of this invention performs. It is explanatory drawing of a prior art.

  Hereinafter, the present invention will be described in detail according to embodiments.

  Here, in the exemplary embodiment described below, it is assumed that a multi-view video captured by three cameras is encoded.

  FIG. 1 shows a conceptual diagram of a camera configuration used in this embodiment. The square figure in the figure represents the frame of each camera.

  In the case of this camera configuration, in the present invention, first, the positional relationship between the cameras C1, C2, and C3 is obtained, and any one of the cameras is determined as a reference viewpoint. Next, a distance image from the reference viewpoint (an image indicating the distance between the subject and the background) is generated. For frames at the same time, encoding is performed in the order of the image of the camera set as the reference viewpoint, the distance image, and the image of the camera other than the reference viewpoint. Encoding is performed with reference to the image, and the images of the cameras other than the reference viewpoint are encoded with reference to the image of the camera at the reference viewpoint and the distance image.

  In order to simplify the description, encoding is performed in the order described in the frame diagram in FIG.

  2 to 6 show an embodiment of the video encoding apparatus 1 according to the present invention.

  As shown in FIG. 2, the video encoding device 1 of the present invention includes an image information input unit 11 that inputs the frames of the cameras C1, C2, and C3 in the order shown in FIG. The image memory 12 to be accumulated, the camera positional relationship (camera installation position and camera orientation), a camera information initial setting unit 13 for selecting a camera as a reference viewpoint, and a reference for encoding an image of the reference viewpoint The viewpoint moving image processing unit 14, the distance image processing unit 15 that generates and encodes a distance image from the reference viewpoint based on the frames of all the cameras at the same time, and the positional relationship between the distance image, the reference viewpoint image, and the camera And a non-reference viewpoint moving image processing unit 16 that estimates a frame of a camera other than the reference viewpoint and encodes a difference from the input image of the camera.

  FIG. 3 is a diagram showing details of the configuration of the camera information initial setting unit 13.

  As shown in this figure, the camera information initial setting unit 13 includes a camera positional relationship estimation unit 130 that estimates the positional relationship of the cameras, a reference viewpoint setting unit 131 that selects a reference viewpoint from the input camera group, A camera positional relationship encoding unit 132 that encodes positional relationship information.

  Here, the camera positional relationship information estimated by the camera positional relationship estimation unit 130 is notified to the distance image processing unit 15 and the non-reference viewpoint moving image processing unit 16.

  FIG. 4 is a diagram showing details of the configuration of the reference viewpoint moving image processing unit 14.

  As shown in this figure, the reference viewpoint moving image processing unit 14 encodes the reference viewpoint moving image encoding unit 140 that encodes the reference viewpoint image as a normal two-dimensional moving image, and the encoded reference viewpoint moving image. A reference viewpoint moving image decoding unit 141 for decoding.

  Here, regarding the encoding target information such as the block division type and the motion vector generated when the reference viewpoint moving image encoding unit 140 encodes the image of the reference viewpoint, the distance image processing unit 15 and the non-reference viewpoint moving image The processing unit 16 is notified. The reference viewpoint moving image decoded by the reference viewpoint moving image decoding unit 141 is notified to the non-reference viewpoint moving image processing unit 16.

  FIG. 5 is a diagram showing details of the configuration of the distance image processing unit 15.

  As shown in this figure, the distance image processing unit 15 estimates and generates a distance image from the reference viewpoint based on the multi-viewpoint moving images and the positional relationship between the cameras that captured them, A distance video encoding unit 151 that encodes the distance image as a normal two-dimensional video, and a distance video decoding unit 152 that decodes the encoded distance video.

  Here, the distance moving image decoded by the distance moving image decoding unit 152 is notified to the non-reference viewpoint moving image processing unit 16.

  FIG. 6 is a diagram showing details of the configuration of the non-reference viewpoint moving image processing unit 16.

  As shown in this figure, the non-reference viewpoint moving image processing unit 16 is a once-encoded and decoded distance moving image (hereinafter referred to as a decoded distance moving image) at the same time and a once-encoded and decoded reference viewpoint. Based on a moving image (hereinafter referred to as a decoding reference viewpoint moving image), a parallax compensation image generation unit 160 that generates an image (hereinafter referred to as a parallax compensation image) at the viewpoint of the camera, and a parallax compensation image are input. A parallax difference image generation unit 161 that generates a difference image (hereinafter referred to as a parallax difference image) with an image at the same viewpoint and the same time, and a parallax that encodes the generated parallax difference image as a normal two-dimensional moving image A differential video encoding unit 162; a parallax differential video decoding unit 163 that decodes the encoded parallax differential video; a reference viewpoint video, a distance video, and a parallax differential video that are encoded and decoded; And a non-reference viewpoint image memory 164 for storing the image of the non-reference viewpoint is al restored.

  7 to 11 show a processing flow executed by the video encoding apparatus 1 of the present invention configured as described above.

  Next, processing executed by the video encoding device 1 of the present invention configured as described above will be described in detail according to these processing flows.

  In the video encoding apparatus 1 of the present invention, as shown in the processing flow of FIG. 7 showing the outline of the entire encoding process, the frames of the cameras C1, C2, and C3 are sequentially displayed in the order shown in FIG. 11 [A1]. The input images are stored in the image memory 12 until images from all cameras at the same time are input [A2, A3]. That is, in the case of the present embodiment, the image memory 12 has a memory capacity for three frames.

  Next, if no encoding process has been performed so far, the camera information initial setting unit 13 checks the positional relationship of each camera, and after one reference viewpoint is selected from all cameras. The input frame is encoded [A4, A5, A6]. On the other hand, if even one frame has been encoded so far, the input frame is encoded using the already obtained camera positional relationship and reference viewpoint [A4, A6].

  FIG. 8 shows a processing flow of processing performed by the camera information initial setting unit 13.

  As shown in this processing flow, in the camera information initial setting unit 13, first, the camera positional relationship estimation unit 130 estimates the relative positional relationship of each camera from the frames of the three cameras at the same given time [ B1].

  As this estimation method, any camera parameter estimation method represented by those described in the following references can be used.

References: Oliver Faugeras, Three-Dimension Computer Vision-MIT Press; BCTC /UFF-006.37 F259 1993-ISBN: 0-262-06158-9.
Since the information on the estimated positional relationship of the camera is necessary accurately on the decoding side, the camera positional relationship encoding unit 132 performs lossless encoding [B2].

  Then, the reference viewpoint setting unit 131 calculates a space that can be photographed by the cameras from the positional relationship of the cameras, and performs a process of setting the camera having the largest space overlapping with other cameras as the reference viewpoint [B3].

In other words, it performs processing for the C _b which satisfy the following expression with reference view camera. Here, S _j (k) indicates an area that can be captured by the camera k in the image of the camera j.

  FIG. 9 shows a processing flow of the input frame encoding process.

  As shown in this processing flow, in the distance image generation unit 150 in the distance image processing unit 15, all camera frames stored in the image memory 12 and the camera positional relationship obtained by the camera information initial setting unit 13. Then, a distance image between the subject and the background from the reference viewpoint is generated using the reference viewpoint information [C1].

  At this time, the reference viewpoint image is divided into certain blocks, and the distance image is generated by estimating the distance for each block. The distance in a certain block of the reference viewpoint image is estimated by the following method.

  That is, when the distance between the pixels included in the block and the surrounding M pixels is assumed to be d, the corresponding pixel in the non-reference viewpoint image is obtained from the positional relationship of the camera, and the following evaluation function is obtained. Then, the estimation is performed with d giving the minimum evaluation value as the distance of the block.

Here, in this equation, B represents a set of pixels in the reference viewpoint image included in the block whose distance is to be obtained and M pixels around it, and when the distance of a certain pixel bεB is d In the non-reference viewpoint image, a pixel corresponding to the pixel is represented by r (b, d), and a pixel value of the pixel _b is represented by _Ib .

Further, when the minimum evaluation value E _min is obtained at the distance D _cur (a value determined by d), the above evaluation function is used by using the distance D _pre of the same block in the past at the previous time. If the difference between the two evaluation values does not exceed the threshold value D _th as shown in the following equation using the obtained evaluation value E _pre , the distance in the past is used as the distance of the block.

Here, this threshold value D _th is prepared for determining whether to use the distance estimated in the above [Equation 2] as it is or to change it to one that does not change with time. It is prepared to realize the use of the one with the better coding efficiency. By providing this threshold function, the distance is estimated in consideration of the correlation in the time direction in the parallax difference image, so it is possible to prevent the time direction correlation from being lost in the parallax difference image. Thus, more efficient encoding can be performed in encoding the parallax difference image.

  In this way, in [C1] of the processing flow of FIG. 9, the distance image is generated by obtaining the distance corresponding to all the pixels of the reference viewpoint.

  Next, the reference viewpoint moving image encoding unit 140 in the reference viewpoint moving image processing unit 14 encodes the reference viewpoint image as a normal two-dimensional moving image (reference viewpoint moving image) [C2]. Then, using the motion vector (motion vector obtained between frames of the reference viewpoint image) used for encoding the reference viewpoint moving image, the distance moving image encoding unit 151 in the distance image processing unit 15 The generated distance image is encoded as a two-dimensional moving image (distance moving image) [C3].

  Here, since the viewpoint positions in the reference viewpoint moving image and the distance moving image are the same, common motion vectors, encoding block sizes, and the like can be used. Specifically, whether the block size and motion vector information used when encoding the reference viewpoint video are used as they are when encoding the distance video, or whether a new block size and motion vector are set again Select selectively.

In this case, a certain amount of constant λ is used to set and code a new block size and motion vector, and a code amount R _new necessary for encoding and a residual amount D _new when using this information are used. The cost COST _new obtained by the following equation is calculated from the above, and the code amount R _old necessary to indicate that the block size and the motion vector used for encoding the reference viewpoint moving image are diverted. Then, the cost COST _old obtained by the same expression is calculated from the residual amount D _old when the diverted information is used, and the one having the smaller cost is selected.

  In addition, as a method of using the motion vector used for encoding the reference viewpoint moving image when encoding the distance moving image, a method of weighting the prediction value or encoding the reference viewpoint moving image. There is also a method of using a residual vector for the motion vector at the time.

Specifically, the method of weighting the predicted value is that when the motion vector used in the encoding of the reference viewpoint moving image is (V _x , V _y ), the distance moving image is encoded. As a motion compensation option, the value calculated by the following equation is used by adding only information of a certain constant Weight.

  Here, estimatedValue (i, j) represents the predicted value of position (i, j), and previousValue (i, j) represents the value of position (i, j) in the reference frame.

Similarly, the method of using a residual vector refers to using the one calculated by the following equation by adding only information of a certain constant vector (MV _x , MV _y ).

  The reference viewpoint moving image encoded in this way is decoded by the reference viewpoint moving image decoding unit 141 in the reference viewpoint moving image processing unit 14, and the distance moving image thus encoded is encoded. The distance moving image decoding unit 152 in the distance image processing unit 15 decodes them [C4, C5].

  The decoded reference viewpoint moving image and distance moving image are sent to the non-reference viewpoint moving image processing unit 16.

  First, the parallax compensation image generation unit 160 in the non-reference viewpoint moving image processing unit 16 obtains a parallax compensation image estimated from the decoded reference viewpoint moving image and the distance moving image for a camera other than the reference viewpoint. Generate [C6]. Note that this parallax compensation image generation processing will be described later in the processing flow of FIG.

  Subsequently, the parallax difference image generation unit 161 in the non-reference viewpoint moving image processing unit 16 generates a parallax difference image by taking a difference between the generated parallax compensation image and the input frame [C7]. The parallax difference image is encoded as a normal two-dimensional moving image by the parallax difference moving image encoding unit 162 in the non-reference viewpoint moving image processing unit 16 [C8].

  Subsequently, the encoded parallax difference moving image is decoded by the parallax difference moving image decoding unit 163 in the non-reference viewpoint moving image processing unit 16 and stored at the same time stored in the non-reference viewpoint image memory 164. A decoded non-reference viewpoint moving image is generated by adding the parallax compensation image, and stored again in the non-reference viewpoint image memory 164 for use in encoding the next frame [C9].

  Next, a parallax compensation image generation process executed by the parallax compensation image generation unit 160 in the non-reference viewpoint moving image processing unit 16 will be described according to the processing flow of FIG.

  In order to generate the parallax compensation image, the parallax compensation image generation unit 160 projects the values of all the pixels of the reference viewpoint image onto the target camera image from the distance image and the positional relationship of the camera [D1].

This projection is executed by the equation shown in FIG. Here, H _Cn is a transformation matrix from points on the image taken at the reference viewpoint to points on the image taken by the camera C _n , (i, j) is the reference viewpoint coordinates, and d _ij is The distance image value corresponding to the coordinates, (I, J) is the position on the image taken by the camera C _n corresponding to the position (i, j) on the reference viewpoint image, and f _Cn is the camera The focal length of C _n , and A, X, and Y are arbitrary real numbers that hold the equation. At this time, H _Cn can be made from the distance image and the positional relationship of the camera.

  Since the projection performed in [D1] generally does not correspond one-to-one, values are not always assigned to the pixels of the entire frame.

For example, a camera that can use a general pinhole model with a focal length f at all viewpoints is used, and the camera of the reference viewpoint and the camera C _n are directed in the same direction, and the camera position is Δx in the horizontal direction. In the case where the distance is only a distance, H _Cn takes the form shown in FIG.

In other words, two pixels on the reference viewpoint (i1, j1), (i2 , j2) is 12 if it meets the formula (c), the one of the pixels since the will correspond to the same pixel in the camera C _n , H _Cn is not a one-to-one projection. Here, d _i1j1 is the value of the distance image corresponding to the pixel (i1, j1), and d _i2j2 is the value of the distance image corresponding to the pixel (i2, j2).

  Since it does not correspond one-on-one in this way, the following processing is performed on all pixels that do not have a value assigned by projection [D2].

  First, spatial prediction is performed [D3]. Spatial prediction is a method for obtaining the value of its own pixel from the value of an adjacent pixel to which a value has already been assigned. Next, temporal prediction is performed [D4]. Temporal prediction is performed on the basis of motion vectors obtained by referring to past frames stored in the non-reference viewpoint image memory 164 in the non-reference viewpoint moving image processing unit 16 in peripheral blocks to which values have already been assigned. The motion vector is estimated, and the corresponding pixel is found from the past frame using the motion vector and complemented.

  Then, the difference value with the encoding target frame is actually compared, and prediction that enables efficient encoding is performed from the relationship between the amount of difference value and the amount of code necessary to indicate the type of prediction [D5 ]. The prediction mode used at this time is sent to the next step together with the parallax compensation image as encoding target information.

  Thus, when the parallax compensation image is generated by the parallax compensation image generation unit 160 in the non-reference viewpoint moving image processing unit 16 [C6], as described in the processing flow of FIG. The parallax difference image is generated [C7], the parallax difference image is encoded as a two-dimensional moving image [C8], and is further decoded and stored in the non-reference viewpoint image memory 164. The decoded non-reference viewpoint moving image is generated by adding the parallax compensation images at the same time and stored again in the non-reference viewpoint image memory 164 for use in encoding the next frame [C9].

  In the encoding performed in [C8], motion vector information is shared if possible using the encoding target information used when encoding the distance moving image and the reference viewpoint moving image.

  FIG. 11 shows a processing flow relating to the use of this motion vector.

  As shown in this processing flow, first, the block (I, J) of the reference viewpoint moving image corresponding to the encoded block (i, j) of the parallax difference moving image to be encoded is determined from the positional relationship of the camera and the distance image. ) Is obtained [E1].

Next, a two-dimensional vector (V _I , V _J ) used when coding the block (I, J) of the reference viewpoint moving image is extracted. This vector is two-dimensional. If the start point is called a reference block and the end point is called an encoding target block, the estimated distance D _now of the block corresponding to the encoding target block in the distance image at the same time is assumed to be the end point depth. By assuming the estimated distance D _pre of the block corresponding to the reference block in the distance image at the same time as the reference frame as the depth of the starting point, the three-dimensional motion vector (V _I , V _J , D _now −D _pre ) is obtained. Define [E2].

  Subsequently, using the positional relationship of the camera, the three-dimensional motion vector is converted into a two-dimensional vector in the camera plane of the non-reference viewpoint to be encoded [E3].

  Necessary for expressing the amount of distortion necessary for expressing the amount of distortion (prediction error amount) when using this vector and the amount of distortion when using the motion vector required for encoding other ordinary two-dimensional moving images. A vector with good coding efficiency is adopted [E4].

  At this time, when a motion vector obtained in encoding of a normal two-dimensional moving image is used, the motion vector is encoded, and the two-dimensional vector obtained by conversion is used as a motion vector (estimated motion vector). If so, information representing that is encoded.

  In this way, the video encoding device 1 of the present invention encodes images taken by the cameras C1, C2, and C3.

  FIG. 13 illustrates the types of data encoded by the video encoding device 1 of the present invention and the encoding order when the camera configuration shown in FIG. 1 is used.

  As shown in this figure, the video encoding apparatus 1 according to the present invention firstly has camera information about cameras C1, C2, and C3 (information about which camera is used as a reference viewpoint and information about the positional relationship of the cameras). Etc.), and then, for time T1, reference viewpoint image / distance image / parallax difference image for cameras other than the reference viewpoint are encoded, and then reference viewpoint image / distance for time T2. The parallax difference image for the camera other than the image / reference viewpoint is encoded, and then encoded at a time T3 in such a manner that the image of the reference viewpoint / distance image / the parallax difference image for the camera other than the reference viewpoint is encoded. It performs.

  Next, the video decoding apparatus of the present invention for decoding the encoded data generated in this way will be described.

  14 and 15 show an embodiment of the video decoding device 2 of the present invention.

  As shown in FIG. 14, the video decoding device 2 of the present invention includes a camera information decoding unit 21 that decodes camera information, a camera information memory 22 that stores the decoded camera information, and a camera selected as a reference viewpoint. A reference viewpoint moving picture decoding unit 23 for decoding video, an encoding target information memory 24 for storing encoding target information such as a block division type and a motion vector generated when decoding the reference viewpoint moving picture, and a distance video A distance moving image decoding unit 25 that decodes an image, a parallax difference moving image decoding unit 26 that decodes a parallax difference moving image, a non-reference viewpoint moving image decoding unit 27 that decodes video of a camera other than the reference viewpoint, and And an image output unit 28 for outputting the image.

  FIG. 15 is a diagram illustrating details of the configuration of the non-reference viewpoint moving image decoding unit 27.

  As shown in this figure, the non-standard viewpoint moving image decoding unit 27 includes a distance image memory 270 that stores a distance image at the current time and a distance image at the same time as the image used for the reference image, and parallax compensation. A parallax compensation image generating unit 271 that generates an image, a parallax compensation image complementing unit 272 that supplements pixel values from the time direction and the spatial direction with respect to the parallax compensation image generated by the parallax compensation image generation unit 271, A non-reference viewpoint moving image generation unit 273 that generates a final non-reference viewpoint moving image from the decoded parallax difference moving image, and stores the generated non-reference viewpoint image for use in decoding of subsequent frames. A non-reference viewpoint image memory 274.

  FIGS. 16 to 18 show processing flows executed by the video decoding apparatus 2 of the present invention configured as described above.

  Next, processing executed by the video decoding apparatus 2 of the present invention configured as described above will be described in detail according to these processing flows.

  Here, in the present embodiment, it is assumed that camera information is input first, and then encoded data is input in order of the reference viewpoint moving image, the distance moving image, and the parallax difference moving image for each time. .

  From now on, in the video decoding apparatus 2 of the present invention, as shown in the processing flow of FIG. 16, first, camera information is input, and thereafter, in order of the reference viewpoint moving image, the distance moving image, and the parallax difference moving image for each time. Encoded data is input, and the encoded data is input [F1].

  If the input encoded data is camera information, the camera information decoding unit 21 decodes it and stores it in the camera information memory 22 [F2, F3].

  On the other hand, if the input encoded data is the reference viewpoint moving image, the reference viewpoint moving image decoding unit 23 decodes the input encoded data, and the encoding target such as the motion vector and the encoded block division type obtained at the time of decoding is decoded. Information is stored in the encoding target information memory 24, and the decoded reference viewpoint image is sent to the non-reference viewpoint moving image decoding unit 27 and the image output unit 28 [F4, F5].

  On the other hand, if the input encoded data is a distance moving image, the distance moving image decoding unit 25 decodes the encoded object information using the encoding target information stored in the encoding target information memory 24, and generates a disparity difference. It is sent to the moving picture decoding unit 26 and the non-reference viewpoint moving picture decoding unit 27 [F6, F7].

  On the other hand, if the input encoded data is a parallax differential video, the parallax differential video decoding unit 26 decodes the encoded target information stored in the encoding target information memory 24 and the distance video decoding unit 25. The obtained distance image and the camera positional relationship stored in the camera information memory 22 are decoded and sent to the non-reference viewpoint moving image decoding unit 27 [F8].

  At this time, the parallax difference moving image is decoded while performing motion compensation from the reference frame using the motion vector, similarly to the decoding of the normal moving image. The motion vector used here is included in the encoded data of the parallax difference video, or the encoding target information stored in the encoding target information memory 24 and the distance video decoded by the distance video decoding unit 25. This is estimated from the image and the camera positional relationship stored in the camera information memory 22.

  Then, the non-reference viewpoint moving image decoding unit 27 stores the reference viewpoint moving image decoded by the reference viewpoint moving image decoding unit 23, the distance moving image decoded by the distance moving image decoding unit 25, and the camera information memory 22. The non-reference viewpoint moving image is decoded using the existing camera positional relationship and sent to the image output unit 28 [F9].

  The image output unit 28 receives the reference viewpoint moving image sent from the reference viewpoint moving image decoding unit 23 and the non-reference viewpoint moving image sent from the non-reference viewpoint moving image decoding unit 27, and finally sends them. The decoded image thus received is output [F10].

  Next, the estimated motion vector derivation process executed by the parallax difference video decoding unit 26 will be described according to the processing flow of FIG.

  In the estimation motion vector derivation process executed by the disparity difference moving image decoding unit 26, first, the block (X, Y) of the reference viewpoint moving image corresponding to the block (x, y) of the disparity difference moving image to be decoded is determined. It calculates | requires using a camera positional relationship and the decoded distance image of the time [G1].

Next, the motion vector (U _X, U _Y ) used when encoding the block (X, Y) of the reference viewpoint moving image stored as the encoding target information is extracted [G2].

Subsequently, this vector is "block (X, Y) is a block _{(X-U X, Y-} U Y) in the reference frame of video in which what has been moved" With that indicates that the decoding The position value D _n corresponding to the block (X, Y) of the reference viewpoint image in the distance image at the same time as the target frame is assumed to be the depth at the end point of the vector, and the reference viewpoint image in the distance image at the same time as the reference frame By assuming the position value D _p corresponding to the block (X-U _X, Y-U _Y ) as the depth at the starting point of the vector, the three-dimensional motion vector (UX _, U _Y, D _n -D _p ) Is defined [G3].

  Finally, using the positional relationship of the camera, this three-dimensional motion vector is converted into a two-dimensional vector in the camera plane of the non-reference viewpoint corresponding to the parallax difference moving image to be decoded to obtain an estimated motion vector [ G4].

  Next, the non-reference viewpoint video decoding process executed by the non-reference viewpoint video decoding unit 27 will be described according to the processing flow of FIG.

  In the decoding process of the non-reference viewpoint moving image executed by the non-reference viewpoint moving image decoding unit 27, first, the decoded distance moving image, the decoded reference viewpoint moving image, and the camera positional relationship are input to the parallax compensated image generating unit 271, and the parallax The compensation image generation unit 271 generates a parallax compensation image [H1].

  Next, in the parallax compensation image, a pixel whose value has not been assigned in the previous processing is subjected to spatial complementation or temporal complementation in the parallax compensation image complementing unit 272 [H2, H3, H4, H5]. . Which complement is performed is embedded in the encoded data of the parallax difference moving image.

  Here, spatial complementation is a complementation process that predicts a pixel from the values of surrounding pixels. In addition, temporal complementation predicts a motion vector of a surrounding pixel from an image stored in the non-reference viewpoint image memory 274, predicts a motion vector of the pixel from a motion vector of the surrounding pixel, and moves the motion. This is complementation in which the value of the corresponding pixel in the image stored in the non-reference viewpoint image memory 274 is represented by the vector. This complementing process is repeated until there are no more assigned pixels [H2].

  Then, the non-reference viewpoint moving image generation unit 273 generates a non-reference viewpoint moving image by adding the generated parallax compensation image and the parallax difference moving image decoded by the parallax difference moving image decoding unit 26 [H6 ]. Since the generated non-reference viewpoint image is used in the subsequent temporal interpolation processing, the non-reference viewpoint moving image generation unit 273 stores it in the non-reference viewpoint image memory 274 [H7].

  Thus, the video decoding device 2 of the present invention decodes the images captured by the cameras C1, C2, and C3 by decoding the encoded data generated by the video encoding device 1 of the present invention. .

  In the embodiment described above, the positional relationship of the camera is assumed not to change. However, if there is a change in the positional relationship, the positional relationship of the camera can be recalculated and encoded and transmitted each time. . Further, the positional relationship information of the camera can be updated in units of GOP while allowing a certain amount of deviation.

  In the embodiment described above, the positional relationship of the camera and the reference viewpoint are obtained from the camera frame. However, such information can also be given from the outside. In that case, it is possible to omit the above-described estimation of the positional relationship of the cameras and the selection of the reference viewpoint.

  Although not described in the embodiment described above, any technique can be used as a technique for obtaining the positional relationship of the cameras without affecting both the encoding process and the decoding process.

  In addition, there are some processes for encoding / decoding two-dimensional moving images in the embodiment, but any method can be used as long as the method is a moving image encoding / decoding method that performs motion prediction / compensation. The method can also be applied. As an encoding technique using motion prediction / compensation, H.I. There are many techniques such as H.264 and MPEG-4.

DESCRIPTION OF SYMBOLS 1 Video coding apparatus 2 Video decoding apparatus 11 Image information input part 12 Image memory 13 Camera information initial setting part 14 Reference viewpoint moving image processing part 15 Distance image processing part 16 Non-reference viewpoint moving image processing part 21 Camera information decoding part 22 Camera Information memory 23 Reference viewpoint video decoding unit 24 Encoding target information memory 25 Distance video decoding unit 26 Parallax difference video decoding unit 27 Non-reference viewpoint video decoding unit 28 Image output unit

Claims (16)

A video encoding method for encoding images captured by a plurality of cameras that capture a subject,
Encoding a reference viewpoint image captured by a camera serving as a reference viewpoint;
Generating a distance image indicating an estimated distance from the camera that captured the reference viewpoint image to the subject;
Encoding the distance image;
Pixels for which a parallax compensation image at a viewpoint other than the reference viewpoint is estimated based on the reference viewpoint image, the distance image, and a camera positional relationship that defines the installation position and orientation of the camera, and a pixel value cannot be estimated at this time For estimating the pixel motion value of the pixel based on the estimated motion information and the pixel value of the encoded image, estimating the pixel motion information from the motion information of the surrounding pixels,
Calculating a parallax difference image indicating a difference between the estimated parallax compensation image and an encoding target image captured by a camera associated with the estimation target viewpoint;
Encoding the parallax difference image.
A characteristic video encoding method. The video encoding method according to claim 1,
In the step of estimating the parallax compensation image, a reference viewpoint image obtained by decoding the encoded data of the reference viewpoint image and a distance image obtained by decoding the encoded data of the distance image are used. Estimating a parallax compensation image at a viewpoint other than the reference viewpoint,
A characteristic video encoding method. The video encoding method according to claim 1,
Acquiring the positional relationship of the camera according to information from the outside, or setting the positional relationship of the camera by estimating the positional relationship of the camera based on images of all cameras;
Encoding information on the positional relationship of the set camera,
A characteristic video encoding method. The video encoding method according to claim 3, wherein
In the step of estimating the parallax compensation image, a reference viewpoint image obtained by decoding encoded data of the reference viewpoint image, a distance image obtained by decoding encoded data of the distance image, and the camera Estimating a parallax compensation image at a viewpoint other than the reference viewpoint based on the positional relation of the camera obtained by decoding the encoded data of the positional relation information,
A characteristic video encoding method. The video encoding method according to any one of claims 1 to 4,
Having a step of setting a camera that is shooting a space most overlapping with a space shot by another camera as a reference viewpoint camera,
A characteristic video encoding method. The video encoding method according to any one of claims 1 to 5,
In the step of generating the distance image, the distance image is generated by dividing the image into blocks and estimating the distance for each block.
A characteristic video encoding method. The video encoding method according to any one of claims 1 to 6,
In the step of generating the distance image, when the distance image is generated according to a prescribed algorithm, the difference value between the evaluation value of the distance image generated at the current time and the evaluation value of the distance image generated at the previous time Is determined by comparing the magnitude of the difference value with a predetermined threshold, and when the difference value is determined to be large, it is determined that the distance image generated at the current time is used as it is. When determining that the difference value is small, generating a distance image by determining to change to the distance image generated at the previous time,
A characteristic video encoding method. The video encoding method according to any one of claims 1 to 7,
In the step of encoding the distance image, encoding the distance image using a motion vector used in encoding the reference viewpoint image,
A characteristic video encoding method. The video encoding method according to any one of claims 1 to 8,
In the step of encoding the parallax difference image, the motion vector used in encoding the reference viewpoint image, the motion vector estimated based on the positional relationship between the distance image and the camera, or its own reference Selecting and encoding one of the motion vectors estimated from the image with the best coding efficiency,
A characteristic video encoding method. The video encoding method according to claim 9, wherein
In the step of encoding the parallax difference image, when estimating a motion vector, using a distance image obtained by decoding encoded data of the distance image,
A characteristic video encoding method. A video decoding method for decoding encoded data of images taken by a plurality of cameras that photograph a subject,
Decoding encoded data for a reference viewpoint image captured by a camera serving as a reference viewpoint;
Decoding encoded data for a distance image indicating an estimated distance from the camera that captured the reference viewpoint image to the subject;
Decoding encoded data for a parallax difference image indicating a difference from an encoding target image captured by a camera associated with a viewpoint other than the reference viewpoint;
Based on the decoded reference viewpoint image, the decoded distance image, and the positional relationship of the camera that defines the installation position and orientation of the camera, a parallax compensation image at a viewpoint other than the reference viewpoint is estimated, and at this time, the pixel value For a pixel that cannot be estimated, the motion information of the pixel is estimated from the motion information of surrounding pixels, and the pixel value of the pixel is estimated based on the estimated motion information and the pixel value of the decoded image Steps,
Reconstructing an encoding target image captured by a camera associated with a viewpoint other than the reference viewpoint based on the estimated parallax compensation image and the decoded parallax difference image.
A video decoding method. The video decoding method according to claim 11,
Having the step of decoding the encoded data about the positional relationship information of the camera,
A video decoding method.   A video encoding program for causing a computer to execute the video encoding method according to any one of claims 1 to 10.   11. A computer-readable recording medium on which a video encoding program for causing a computer to execute the video encoding method according to claim 1 is recorded.   A video decoding program for causing a computer to execute the video decoding method according to claim 11.   A computer-readable recording medium on which a video decoding program for causing a computer to execute the video decoding method according to claim 11 or 12 is recorded.

Images