JP2006086707A - Image processing method and image processing apparatus, program and program recording medium, and data structure and data recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To compress at a high compression rate moving picture data of a high frame rate and to uncompress the compressed moving picture data. <P>SOLUTION: Each of a transmitter and a receiver detects a positional relation with a high correlation between a past reference image V<SB>P</SB>of a frame just preceding to a target block and a future reference image V<SB>T</SB>of a frame just after the target block, as to the target block, and obtains an estimate value of the target block on the basis of image data in a region R<SB>8201</SB>of the past reference image and image data in a region R<SB>8202</SB>of the future reference image within the positional relation. Then the transmitter compresses the target block by using the estimate value. On the other hand, the receiver uncompresses the target block by using the estimate value. The present invention can be applied to the case where moving pictures are compressed and uncompressed. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and an image processing method, a program and a program recording medium, and a data structure and a data recording medium. In particular, for example, high frame rate moving picture data can be highly compressed and restored. The present invention relates to an image processing apparatus, an image processing method, a program, a program recording medium, a data structure, and a data recording medium.

  Conventionally, moving image (moving image) data is recorded (recorded) and reproduced at a frame rate of about 30 to 60 frames per second (30 to 60 fps (Frame per second)). However, at such a frame rate, a moving subject (moving subject) is perceived as a blurred image in human vision, and thus it cannot be said that good image quality is obtained for humans.

  By the way, it is known that when moving image data is recorded and reproduced at a frame rate of about 240 fps, image quality good for human beings can be obtained.

  Video data with a frame rate of 240 fps can be displayed on a display device that supports a high frame rate of 240 fps, and further, for example, by thinning out frames and reducing the frame rate, a low frame rate such as 30 fps or 60 fps It can also be displayed on the display device.

  However, when the frames of moving image data having a frame rate of 240 fps are simply thinned out, the smoothness of the motion is lost in the image.

  Therefore, Patent Document 1 proposes a method in which an average value of a plurality of frames of high frame rate moving image data is used as low frame rate moving image data, instead of simply thinning out high frame rate moving image data. . Further, Patent Document 1 proposes a method of hierarchically encoding high frame rate moving image data into a plurality of layers corresponding to a plurality of frame rates.

Here, for example, when two-layer encoding of a 120 fps layer and a 60 fps layer is performed, for example, data D ₁ of a certain frame and data D of the next frame of 120 fps moving image data are used. average of _{2 (D 1 + D 2)} / 2 is set to the hierarchy of the moving image data of 60 fps, of one of the data D ₁ and D _2, and the example, only the data D ₁ is 120 fps hierarchy Video data. In this case, the data D ₂ is the moving image data of 60fps hierarchy _{(D 1 + D 2) /} 2, can be obtained from the moving picture data D ₁ Metropolitan of 120fps hierarchy.

  As described above, high-frame-rate video data such as 240 fps can provide good image quality, but 240 fps video data is simpler than conventional 30-fps or 60-fps video data. The amount of data is 8 times or 4 times larger. This data amount does not change even when the above-described hierarchical encoding is performed. An enormous amount of moving image data increases the data processing load of the moving image data, and is not preferable for transmission or recording of moving image data.

JP 2004-088244 A.

  The present invention has been made in view of such a situation. For example, high-frame-rate moving image data can be highly compressed and restored.

  The first image processing apparatus according to the present invention converts the first moving image data from the first moving image data to the second moving image data having a frame rate lower than the frame rate of the first moving image data. Separating means for separating the remaining third moving image data excluding data, detecting means for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data, and a positional relationship detected by the detecting means Using the estimation means for obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data, and using the estimated value of the block, Compression means for compressing moving image data in units of blocks.

  According to the first image processing method of the present invention, the first moving image data is divided into second moving image data having a frame rate lower than the frame rate of the first moving image data, and from the first moving image data to the second moving image. A separation step for separating the remaining third moving image data excluding data, a detection step for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data, and a positional relationship detected in the detection step. Using an estimation step for obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data, and an estimated value of the block, A compression step of compressing the moving image data in units of blocks.

  The first program of the present invention converts the first moving image data, the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the second moving image data from the first moving image data. A separation step that separates into the remaining third moving image data, a detection step that detects a positional relationship having a high correlation in a plurality of frames of the second moving image data, and a positional relationship that is detected in the detection step; An estimation step for obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data, and the third moving image data using the estimated value of the block And a compression step of compressing each block in units of blocks.

  The program recorded on the first program recording medium of the present invention includes the first moving image data, the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. A separation step for separating the data from the remaining third moving image data excluding the second moving image data, a detecting step for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data, and a detecting step An estimation step for obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from the plurality of frames of image data of the second moving image data in the detected positional relationship, and an estimated value of the block And a compression step of compressing the third moving image data in units of blocks.

  In the first data structure of the present invention, the first moving image data is divided into second moving image data having a frame rate lower than the frame rate of the first moving image data, and from the first moving image data to the second moving image data. Are separated from the remaining third moving image data, and a positional relationship having a high correlation is detected in a plurality of frames of the second moving image data, and a plurality of frames of the second moving image data in a positional relationship having a high correlation are detected. A block estimated value obtained by dividing the frame of the third moving image data into blocks from the image data, and compressed data obtained by compressing the block using the block estimated value, and the second moving image Data.

  The data recorded on the first data recording medium of the present invention includes the first moving image data, the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. The data is separated from the data to the remaining third moving image data excluding the second moving image data, a positional relationship having a high correlation is detected in a plurality of frames of the second moving image data, and the positional relationship having a high correlation is detected. Compression obtained by calculating an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data, and compressing the block using the estimated value of the block Data and second moving image data.

  The second image processing apparatus according to the present invention converts the first moving image data from the first moving image data to the second moving image data having a frame rate lower than the frame rate of the first moving image data. The second moving image data is separated into the remaining third moving image data excluding the data, the positional relationship having a high correlation is detected in the plurality of frames of the second moving image data, and the plurality of second moving image data having the high positional relationship is detected. An estimated value of a block obtained by dividing the frame of the third moving image data from the image data of the frame is obtained, and the compressed data obtained by compressing the block using the estimated value of the block is converted into the second A first restoration unit that restores the third video data using the video data; and a second restoration unit that combines the second and third video data to restore the first video data, First restoration The stage includes a detecting unit that detects a positional relationship having a high correlation in a plurality of frames of the second moving image data, and a plurality of frames of image data of the second moving image data that are in the positional relationship detected by the detecting unit. A block restoring unit that restores the third moving image data in units of blocks using an estimation unit that obtains an estimated value of a block obtained by dividing the frame of the three moving image data into blocks, a block estimated value, and compressed data; It is characterized by having.

  In the second image processing method of the present invention, the first moving image data is divided into second moving image data having a frame rate lower than the frame rate of the first moving image data, and second moving image from the first moving image data. The second moving image data is separated into the remaining third moving image data excluding the data, the positional relationship having a high correlation is detected in the plurality of frames of the second moving image data, and the plurality of second moving image data having the high positional relationship is detected. An estimated value of a block obtained by dividing the frame of the third moving image data from the image data of the frame is obtained, and the compressed data obtained by compressing the block using the estimated value of the block is converted into the second A first restoration step for restoring the third movie data using the movie data; a second restoration step for synthesizing the second and third movie data to restore the first movie data; First The restoration step includes a detection step for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data, and a plurality of frames of image data of the second moving image data in the positional relationship detected in the detection step. A block for restoring the third moving image data in units of blocks using an estimation step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data into blocks, the estimated value of the block, and the compressed data And a restoration step.

  The second program of the present invention converts the first moving image data, the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the second moving image data from the first moving image data. It is separated into the remaining third moving image data, and a positional relationship having a high correlation is detected in a plurality of frames of the second moving image data, and a plurality of frames of the second moving image data having a high correlation are detected. An estimated value of a block obtained by dividing the frame of the third moving image data from the image data is obtained, and the compressed data obtained by compressing the block using the estimated value of the block is converted into the second moving image data. Using a first restoration step for restoring to the third moving image data, and a second restoration step for combining the second and third moving image data to restore the first moving image data. The restoration step includes a detection step for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data, and a plurality of frames of image data of the second moving image data in the positional relationship detected in the detection step. A block restoration that restores the third moving image data in units of blocks using an estimation step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data into blocks, the estimated value of the block, and the compressed data. And a step.

  The program recorded on the second program recording medium of the present invention includes the first moving image data, the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. The data is separated from the data to the remaining third moving image data excluding the second moving image data, a positional relationship having a high correlation is detected in a plurality of frames of the second moving image data, and the positional relationship having a high correlation is detected. Compression obtained by calculating an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data, and compressing the block using the estimated value of the block A first restoration step for restoring the data to the third movie data using the second movie data; and the second and third movie data are synthesized to restore the first movie data. A second restoration step, wherein the first restoration step is in a position relationship detected in the detection step and a position relationship detected in the detection step in a plurality of frames of the second moving image data. Using an estimation step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data, the estimated value of the block, and the compressed data; And a block restoration step for restoring the three moving image data in units of blocks.

  The third image processing apparatus of the present invention converts the first moving image data from the first moving image data to the second moving image data having a frame rate lower than the frame rate of the first moving image data. Separation means for separating the remaining third moving image data excluding data, and a block obtained by dividing a frame of the third moving image data into blocks, there is a correlation with the block in a plurality of frames of the second moving image data. A block is estimated from image data of a plurality of frames of the second moving image data in which the detection means for detecting one motion vector representing a high positional relationship and the positional relationship between the block and the block are obtained from the single motion vector. It is characterized by comprising estimation means for obtaining a value and compression means for compressing the third moving image data in units of blocks using the estimated value of the block.

  According to the third image processing method of the present invention, the first moving image data is divided into second moving image data having a frame rate lower than the frame rate of the first moving image data, and from the first moving image data to the second moving image. The separation step for separating the third moving image data excluding the data and the block obtained by dividing the frame of the third moving image data into blocks are correlated with the blocks in the plurality of frames of the second moving image data. A detection step for detecting one motion vector representing a high positional relationship and a block estimation from image data of a plurality of frames of the second moving image data in which the positional relationship between the block and the block is a positional relationship obtained from the single motion vector A prediction step for obtaining a value, and a compression step for compressing the third moving image data in units of blocks using the estimated value of the block. To.

  The third program of the present invention converts the first moving image data, the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the second moving image data from the first moving image data. The separation step for separating the remaining third moving image data from the removed block and the block obtained by dividing the frame of the third moving image data into blocks are positions having a high correlation with the blocks in the plurality of frames of the second moving image data. From the detection step of detecting one motion vector representing the relationship and the positional relationship between the block and the block, the estimated value of the block is obtained from the image data of a plurality of frames of the second moving image data. And a compression step of compressing the third moving image data in units of blocks using the estimated value of the block. To.

  The program recorded on the third program recording medium of the present invention includes the first moving image data, the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. A plurality of frames of the second moving image data with respect to a block obtained by dividing the third moving image data into blocks, and a separation step for separating the data into the remaining third moving image data excluding the second moving image data from the data; A plurality of frames of the second moving image data in which the detection step of detecting one motion vector representing a positional relationship having a high correlation with the block and the positional relationship with the block obtained from the single motion vector The third video data is compressed in units of blocks using an estimation step for obtaining an estimated value of the block from the data and the estimated value of the block. Characterized in that it comprises a compression step that.

  According to the second data structure of the present invention, the first moving image data is divided into second moving image data having a frame rate lower than the frame rate of the first moving image data, and the first moving image data to the second moving image data. The block obtained by dividing the third moving image data frame into the remaining third moving image data excluding the block has a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data. One motion vector to be expressed is detected, and an estimated value of the block is obtained from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is obtained from the one motion vector, and the block is estimated The compressed data obtained by compressing the block using the value and the second moving image data are included, and the compressed data includes one motion vector. And it features.

  The data recorded on the second data recording medium of the present invention includes the first moving image data, the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. The blocks obtained by separating the third moving image data from the data into the remaining third moving image data and dividing the third moving image data into blocks are divided into blocks in a plurality of frames of the second moving image data. One motion vector representing a positional relationship having a high correlation is detected, and a block is estimated from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship obtained from one motion vector A compressed value obtained by compressing the block using the estimated value of the block and the second moving image data, and the compressed data is 1 Characterized in that it comprises a motion vector.

  The fourth image processing apparatus of the present invention converts the first moving image data from the first moving image data to the second moving image data having a frame rate lower than the frame rate of the first moving image data. Positional relationship in which the blocks obtained by dividing the third moving image data frame into blocks that are separated from the remaining third moving image data excluding the data have a high correlation with the blocks in the plurality of frames of the second moving image data Is detected, and an estimated value of the block is obtained from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship obtained from the single motion vector. 1st decompression | restoration means which decompress | restores the compressed data obtained by compressing a block using an estimated value to 3rd moving image data using 2nd moving image data , And a second restoration unit that synthesizes the second and third moving image data and restores the first moving image data. The first restoration unit uses the second moving image for one block from one motion vector. Detection means for obtaining a positional relationship having a high correlation with a block in a plurality of frames of data, and image data of a plurality of frames of second moving image data in which the positional relationship with the block is a positional relationship obtained from one motion vector In addition, the present invention is characterized by comprising: an estimation means for obtaining an estimated value of a block; and a block restoration means for restoring a block using the estimated value of the block and the compressed data.

  According to the fourth image processing method of the present invention, the first moving image data is divided into second moving image data having a frame rate lower than the frame rate of the first moving image data, and from the first moving image data to the second moving image. Positional relationship in which the blocks obtained by dividing the third moving image data frame into blocks that are separated from the remaining third moving image data excluding the data have a high correlation with the blocks in the plurality of frames of the second moving image data Is detected, and an estimated value of the block is obtained from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship obtained from the single motion vector. A first restoration step for restoring the compressed data obtained by compressing the block using the estimated value to the third moving image data using the second moving image data. And a second restoration step of synthesizing the second and third moving image data and restoring the first moving image data, wherein the first restoration step is performed on the second block for the block from one motion vector. A detection step for obtaining a positional relationship having a high correlation with a block in a plurality of frames of the moving image data, and an image of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship obtained from one motion vector It includes an estimation step for obtaining an estimated value of a block from data, and a block restoration step for restoring the block using the estimated value of the block and the compressed data.

  According to a fourth program of the present invention, the first moving image data is divided into second moving image data having a frame rate lower than the frame rate of the first moving image data, and second moving image data from the first moving image data. The block obtained by separating the remaining third moving image data from the third moving image data and dividing the frame of the third moving image data into blocks represents a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data. One motion vector is detected, and an estimated value of the block is obtained from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is obtained from the one motion vector, and the estimated value of the block The first restoration step of restoring the compressed data obtained by compressing the block to the third moving image data using the second moving image data. And a second restoration step of synthesizing the second and third moving image data and restoring the first moving image data, wherein the first restoration step uses a second motion vector for a block from one motion vector. A detection step for obtaining a positional relationship having a high correlation with a block in a plurality of frames of moving image data, and a plurality of frames of image data of the second moving image data in which the positional relationship with the block is a positional relationship obtained from one motion vector And a block restoring step for restoring the block using the estimated value of the block and the compressed data.

  The program recorded on the fourth program recording medium of the present invention includes the first moving image data, the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. The blocks obtained by separating the third moving image data from the data into the remaining third moving image data and dividing the third moving image data into blocks are divided into blocks in a plurality of frames of the second moving image data. One motion vector representing a positional relationship having a high correlation is detected, and a block is estimated from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship obtained from one motion vector A compressed value obtained by compressing the block using the estimated value of the block and the third moving image data using the second moving image data. A first restoration step for restoring data, and a second restoration step for synthesizing the second and third moving image data to restore the first moving image data. A detection step for obtaining a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data from the vector, and a positional relationship with the block determined by a single motion vector; And a block restoring step for restoring a block using the estimated value of the block and the compressed data from the image data of a plurality of frames of the moving image data.

  In the first image processing apparatus, the first image processing method, the first program, and the program recorded in the first program recording medium of the present invention, the first moving image data includes the first moving image data. The second moving image data is separated into second moving image data having a frame rate lower than the frame rate of the data and remaining third moving image data obtained by removing the second moving image data from the first moving image data. A positional relationship having a high correlation is detected in the frame. Further, an estimated value of a block obtained by dividing the frame of the third moving image data into blocks is obtained from the image data of the second moving image data in the detected positional relationship, and the block is estimated. Using the value, the third moving image data is compressed in units of blocks.

  The data recorded on the first data structure and the first data recording medium of the present invention includes the first moving image data and the second moving image having a frame rate lower than the frame rate of the first moving image data. Data and the remaining third moving image data obtained by removing the second moving image data from the first moving image data, and detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data. The estimated value of the block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in a high positional relationship is obtained, and the block is calculated using the estimated value of the block. Compressed data obtained by compression and second moving image data are included.

  In the second image processing apparatus, the second image processing method, the second program, and the program recorded in the second program recording medium of the present invention, the third moving image data is restored, and The second moving image data and the third moving image data are combined to restore the first moving image data. In this case, for the restoration of the third moving image data, a positional relationship having a high correlation is detected in a plurality of frames of the second moving image data, and the image data of the plurality of frames of the second moving image data in the positional relationship is detected. Thus, an estimated value of a block obtained by dividing the frame of the third moving image data into blocks is obtained. Then, the third moving image data is restored in block units using the estimated value of the block and the compressed data.

  In the third image processing apparatus, the third image processing method, the third program, and the program recorded on the third program recording medium of the present invention, the first moving image data includes the first moving image data. The second moving image data having a frame rate lower than the frame rate of the data and the remaining third moving image data obtained by removing the second moving image data from the first moving image data are separated into frames of the third moving image data. As for a block obtained by dividing the block, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected. Then, an estimated value of the block is obtained from the image data of a plurality of frames of the second moving image data, the positional relationship with the block being obtained from one motion vector, and using the estimated value of the block, The third moving image data is compressed in units of blocks.

  In the data recorded on the second data structure and the second data recording medium of the present invention, the first moving image data is the second moving image having a frame rate lower than the frame rate of the first moving image data. For the block obtained by separating the data and the remaining third moving image data obtained by removing the second moving image data from the first moving image data and dividing the frame of the third moving image data into blocks, the second moving image One motion vector representing a positional relationship having a high correlation with a block in a plurality of frames of data is detected, and the positional relationship with the block is a positional relationship obtained from one motion vector. From the image data, an estimated value of the block is obtained, and the compressed data obtained by compressing the block using the estimated value of the block, the second moving image data, It is included.

  In the fourth image processing apparatus, the fourth image processing method, the fourth program, and the program recorded on the fourth program recording medium of the present invention, the third moving image data is restored, The second moving image data and the third moving image data are combined to restore the first moving image data. In this case, for the restoration of the third moving image data, a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is obtained from one motion vector, and the positional relationship with the block is determined. An estimated value of the block is obtained from the image data of a plurality of frames of the second moving image data having the positional relationship obtained from one motion vector. Then, the block is restored using the estimated value of the block and the compressed data.

  According to the present invention, high frame rate moving image data can be highly compressed. Further, according to the present invention, it is possible to restore such highly compressed moving image data.

  Embodiments of the present invention will be described below. Correspondences between constituent elements described in the claims and specific examples in the embodiments of the present invention are exemplified as follows. This description is to confirm that specific examples supporting the invention described in the claims are described in the embodiments of the invention. Therefore, even though there are specific examples that are described in the embodiment of the invention but are not described here as corresponding to the configuration requirements, the specific examples are not included in the configuration. It does not mean that it does not correspond to a requirement. On the contrary, even if a specific example is described here as corresponding to a configuration requirement, this means that the specific example does not correspond to a configuration requirement other than the configuration requirement. not.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, in the future, a divisional application will be made. Nor does it deny the existence of an invention added by amendment.

The image processing apparatus according to claim 1,
In an image processing apparatus (for example, the transmission apparatus 1 in FIG. 79) that processes moving image data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. Separating means (for example, a separation circuit 213 in FIG. 79) that separates the remaining third moving image data excluding the second moving image data (for example, 240-60 fps moving image data);
Detecting means for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data (for example, the maximum correlation position detecting unit 254 in FIG. 94)
Estimating means for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected by the detecting means. (For example, the average value calculation unit 255 in FIG. 94)
Compression means (for example, a subtracting unit 256 in FIG. 94) for compressing the third moving image data in units of blocks using the estimated value of the block.

An image processing apparatus according to claim 3 is provided.
Other detection means (for example, the maximum correlation position detection unit 274 in FIG. 96) for detecting one motion vector representing a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data for the block. ,
Other estimation means for determining another estimated value of the block from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship determined from the one motion vector (for example, The average value calculator 275) of FIG.
Other compression means (for example, the subtraction unit 276 in FIG. 96) that compresses the third moving image data in units of blocks using another estimated value of the block;
Selecting either one of the compressed data obtained by compressing the third moving image data by the compressing unit or the compressed data obtained by compressing the third moving image data by the other compressing unit; And selection means (for example, a selection circuit 240 in FIG. 81) that adds and outputs identification information for identifying the compressed data.

An image processing apparatus according to claim 7 is provided.
In the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction, the region extends in the principal component direction that is the direction of the principal component of the first moving image data, and the frequency axis in the time direction Filter means for filtering the first moving image data with a region having a specific width in the direction as a pass band (for example, a band limiting filter unit 212 in FIG. 79),
The separation unit separates the first moving image data after filtering by the filter unit into the second and third moving image data.

The image processing method according to claim 8 comprises:
In an image processing method for processing video data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. A separation step (for example, step S282 in FIG. 108) that separates the remaining third moving image data excluding the second moving image data (for example, 240-60 fps moving image data);
A detection step (for example, step S212 in FIG. 95) for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. (For example, step S213 in FIG. 95),
A compression step (for example, step S214 in FIG. 95) of compressing the third moving image data in units of blocks using the estimated value of the block.

The image processing method according to claim 9 comprises:
With respect to the block, another detection step (for example, step S222 in FIG. 97) for detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data;
Another estimation step for obtaining another estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector (for example, Step S223) of FIG. 97;
Another compression step (for example, step S224 in FIG. 97) for compressing the third moving image data in units of blocks using another estimated value of the block;
One of the compressed data obtained by compressing the third moving image data by the compression step or the compressed data obtained by compressing the third moving image data by the other compression step is selected. And a selection step (for example, steps S292 and S293 in FIG. 109) for adding and outputting identification information for identifying the compressed data.

The program recorded in the program recording medium according to claim 10 and the program recording medium according to claim 12,
In a program that causes a computer to process video data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. A separation step (for example, step S282 in FIG. 108) that separates the remaining third moving image data excluding the second moving image data (for example, 240-60 fps moving image data);
A detection step (for example, step S212 in FIG. 95) for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. (For example, step S213 in FIG. 95),
A compression step (for example, step S214 in FIG. 95) of compressing the third moving image data in units of blocks using the estimated value of the block.

The program according to claim 11 and the program recorded on the program recording medium according to claim 13 are:
With respect to the block, another detection step (for example, step S222 in FIG. 97) for detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data;
Another estimation step for obtaining another estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector (for example, Step S223) of FIG. 97;
Another compression step (for example, step S224 in FIG. 97) for compressing the third moving image data in units of blocks using another estimated value of the block;
One of the compressed data obtained by compressing the third moving image data by the compression step or the compressed data obtained by compressing the third moving image data by the other compression step is selected. And a selection step (for example, steps S292 and S293 in FIG. 109) for adding and outputting identification information for identifying the compressed data.

An image processing apparatus according to claim 18 is provided.
In an image processing device (for example, the receiving device 2 in FIG. 110) that processes moving image data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. Separated into the remaining third video data (for example, 240-60fps video data) excluding the second video data,
Detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data having a high correlation with the correlation,
First restoration means for restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving image data using the second moving image data (for example, FIG. 110 difference information restoration unit 364),
A second restoring means (for example, the synthesizing unit 365 in FIG. 110) that synthesizes the second and third moving image data and restores the first moving image data;
The first restoration means includes
Detecting means for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data (for example, the maximum correlation position detecting unit 377 in FIG. 113)
Estimating means for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected by the detecting means. (For example, the estimation unit 378 in FIG. 113);
Block reconstruction means (for example, an adding unit 379 in FIG. 113) that restores the third moving image data in units of blocks using the estimated value of the block and the compressed data.

The image processing method according to claim 21,
In an image processing method for processing video data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. Separated into the remaining third video data (for example, 240-60fps video data) excluding the second video data,
Detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data having a high correlation with the correlation,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving picture data using the second moving picture data (for example, FIG. 111 step S302),
A second restoration step (for example, step S303 in FIG. 111) of combining the second and third moving image data and restoring the first moving image data,
The first restoration step includes
A detection step (for example, step S351 in FIG. 116 or step S361 in FIG. 117) for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. (For example, step S352 in FIG. 116 and S362 in FIG. 117)
A block restoration step (for example, step S353 in FIG. 116 or step S363 in FIG. 117) for restoring the third moving image data in units of blocks using the estimated value of the block and the compressed data. It is characterized by.

The program according to claim 23 and the program recorded on the program recording medium according to claim 25 are:
In a program that causes a computer to process video data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. Separated into the remaining third video data (for example, 240-60fps video data) excluding the second video data,
Detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data having a high correlation with the correlation,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving picture data using the second moving picture data (for example, FIG. 111 step S302),
A second restoration step (for example, step S303 in FIG. 111) of combining the second and third moving image data and restoring the first moving image data,
The first restoration step includes
A detection step (for example, step S351 in FIG. 116 or step S361 in FIG. 117) for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. (For example, step S352 in FIG. 116 and S362 in FIG. 117)
A block restoration step (for example, step S353 in FIG. 116 or step S363 in FIG. 117) for restoring the third moving image data in units of blocks using the estimated value of the block and the compressed data. It is characterized by.

The image processing apparatus according to claim 27,
In an image processing apparatus (for example, the transmission apparatus 1 in FIG. 79) that processes moving image data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. Separating means (for example, a separation circuit 213 in FIG. 79) that separates the remaining third moving image data excluding the second moving image data (for example, 240-60 fps moving image data);
For a block obtained by dividing the third moving image data frame into blocks, a detecting means for detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data (for example, 96, the maximum correlation position detector 274) of FIG.
Estimating means for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship obtained from the one motion vector (for example, the average of FIG. 96 Value calculation unit 275),
Compression means (for example, a subtracting unit 276 in FIG. 96) that compresses the third moving image data in units of blocks using the estimated value of the block.

An image processing apparatus according to claim 30 is provided.
In the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction, the region extends in the principal component direction that is the direction of the principal component of the first moving image data, and the frequency axis in the time direction Filter means for filtering the first moving image data with a region having a specific width in the direction as a pass band (for example, a band limiting filter unit 212 in FIG. 79),
The separation unit separates the first moving image data after filtering by the filter unit into the second and third moving image data.

The image processing method according to claim 31,
In an image processing method for processing video data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. A separation step (for example, step S282 in FIG. 108) that separates the remaining third moving image data excluding the second moving image data (for example, 240-60 fps moving image data);
A detection step (for example, detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for a block obtained by dividing the frame of the third moving image data into blocks (for example, , Step S222 in FIG. 97,
An estimation step for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship obtained from the one motion vector (for example, the step of FIG. 97 S223),
And a compression step (for example, step S224 in FIG. 97) of compressing the third moving image data in units of blocks using the estimated value of the block.

The program according to claim 32 and the program recorded in the program recording medium according to claim 33 are:
In a program that causes a computer to process video data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. A separation step (for example, step S282 in FIG. 108) that separates the remaining third moving image data excluding the second moving image data (for example, 240-60 fps moving image data);
A detection step (for example, detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for a block obtained by dividing the frame of the third moving image data into blocks (for example, , Step S222 in FIG. 97,
An estimation step for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship obtained from the one motion vector (for example, the step of FIG. 97 S223),
And a compression step (for example, step S224 in FIG. 97) of compressing the third moving image data in units of blocks using the estimated value of the block.

The image processing apparatus according to claim 36,
In an image processing device (for example, the receiving device 2 in FIG. 110) that processes moving image data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. Separated into the remaining third video data (for example, 240-60fps video data) excluding the second video data,
For a block obtained by dividing the frame of the third moving image data into blocks, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
First restoration means for restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving image data using the second moving image data (for example, FIG. 110 difference information restoration unit 364),
A second restoring means (for example, the synthesizing unit 365 in FIG. 110) that synthesizes the second and third moving image data and restores the first moving image data;
The first restoration means includes
From the one motion vector, detection means (for example, a maximum correlation position detection unit 377 in FIG. 113) for obtaining a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data for the block;
Estimating means for obtaining an estimated value of the block from a plurality of frames of image data of the second moving image data in which the positional relationship with the block is a positional relationship obtained from the one motion vector (for example, FIG. 113) An estimation unit 378),
Block reconstruction means (for example, an adding unit 379 in FIG. 113) that restores the block using the estimated value of the block and the compressed data is provided.

The image processing method according to claim 37,
In an image processing method for processing video data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. Separated into the remaining third video data (for example, 240-60fps video data) excluding the second video data,
For a block obtained by dividing the frame of the third moving image data into blocks, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving picture data using the second moving picture data (for example, FIG. 111 step S302),
A second restoration step (for example, step S303 in FIG. 111) of combining the second and third moving image data and restoring the first moving image data,
The first restoration step includes
A detection step (for example, step S361 in FIG. 117) for obtaining a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data from the one motion vector;
An estimation step for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector (for example, FIG. 117). Step S362)
A block restoration step (for example, step S363 in FIG. 117) for restoring the block by using the estimated value of the block and the compressed data is included.

The program according to claim 38 and the program recorded in the program recording medium according to claim 39 are:
In a program that causes a computer to process video data,
First moving image data (for example, 240 fps moving image data) is obtained from second moving image data (for example, 60 fps moving image data) having a frame rate lower than the frame rate of the first moving image data, and the first moving image data. Separated into the remaining third video data (for example, 240-60fps video data) excluding the second video data,
For a block obtained by dividing the frame of the third moving image data into blocks, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving picture data using the second moving picture data (for example, FIG. 111 step S302),
A second restoration step (for example, step S303 in FIG. 111) of combining the second and third moving image data and restoring the first moving image data,
The first restoration step includes
A detection step (for example, step S361 in FIG. 117) for obtaining a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data from the one motion vector;
An estimation step for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector (for example, FIG. 117). Step S362)
A block restoration step (for example, step S363 in FIG. 117) for restoring the block by using the estimated value of the block and the compressed data is included.

  Next, before describing the embodiment of the present invention, a theoretical matter will be described for moving images.

  FIG. 1 shows moving image data in a three-dimensional space in which a time direction (t) is added to a two-dimensional plane (x, y).

In FIG. 1, an image P ₁₀₁ is an image at a certain time (frame), an image P ₁₀₂ is an image at the next time, and an image P ₁₀₃ is an image at the next time. The image P ₁₀₄ is an image at the next time. Although illustration is omitted, there are also images at a time before the image P ₁₀₁ and a time after the image P ₁₀₄ .

In FIG. 1, the images P _{101 to} P ₁₀₄ show an object (subject) that moves in the y direction over time.

  When a movie such as the above (moving image data) is viewed by a human, when the frame rate of the moving image is higher than a certain frame rate, the difference between two adjacent frames (images) of the moving image can be recognized. Absent. As a result of actually conducting a visual experiment on the subject, it has been found that the frame rate at which a human cannot recognize the difference between two adjacent frames (images) is about 240 fps or more.

Here, if the time (frame period) between two adjacent frames is expressed as t ₀ , the frame rate can be expressed as 1 / t ₀ .

In the following description, the frame period t ₀ is about 1/240 seconds. However, the frame period t ₀ may not be about 1/240 seconds, for example, about 1/120 seconds. good. However, when the frame period t ₀ is longer than the frame period (about 1/240 seconds) in which a human cannot recognize the difference between two adjacent frames (images thereof), for example, 1/120 second. The image quality of the moving image displayed on the display device 3 (FIG. 24) described later is somewhat perceived.

Next, r ₀ represents a human's recognition ability in the spatial direction, that is, a limit distance at which two adjacent points can be recognized (perceived) as two points instead of one point.

  In this case, the range of moving images that can be recognized (perceived) by human vision can be expressed as shown in FIG.

  That is, FIG. 2 shows a frequency domain defined by the frequency axis T in the time t direction and the frequency axes X and Y in the spatial directions x and y.

In order to avoid complication of the figure, in FIG. 2, the frequency axis X in the spatial direction x and the frequency axis Y in the spatial direction y are represented by one axis. That is, in FIG. 2, the frequency axes X and Y are taken from left to right. In FIG. 2, the frequency axis T is taken from the top to the bottom. Further, in FIG. 2, the frequency axis T (vertical direction) is divided in units of 2π / (4t ₀ ), and the frequency axes X and Y directions (lateral direction) are divided in units of 2π / r ₀ .

  Here, other diagrams of the frequency domain, which will be described later, are the same as those in FIG.

Humans, shown in FIG. 2, in the horizontal is _{2 × 2π / (2r 0)} , the vertical of _{2 × 2π / (2r 0)} , recognizes the high frequency components are outside the range of the region R ₂₀₁ centered at the origin Can not do it. Therefore, the following description will be made assuming that all moving image data exists in the region _R201 .

In FIG. 2, the region R ₂₀₁ is illustrated as a rectangular region, but the region R ₂₀₁ is actually a rectangular parallelepiped region. That is, the region R ₂₀₁ has an X direction of − (π / r ₀ ) to + (π / r ₀ ), a Y direction of − (π / r ₀ ) to + (π / r ₀ ), and T The direction is a region in the range of − (π / t ₀ ) to + (π / t ₀ ).

  Next, the position (distribution) on the frequency domain of the data of each part of the moving image will be described.

  In a moving image, a subject projected on a certain portion is stationary, and a subject projected on the moving portion is moving in another portion.

  First, the stationary moving image portion (the moving image portion on which the stationary subject is projected) does not change with respect to the time direction.

FIG. 3 shows a region R ₃₀₁ on the frequency domain in which data of such a still moving image portion is distributed.

The region R ₃₀₁ is a region in which the X and Y directions are −2π / (2r ₀ ) to + 2π / (2r ₀ ) and the T direction is −2π / (4t ₀ ) to + 2π / (4t ₀ ). .

Note that if the data is a completely stationary part of the video, T = 0, that is, the region R ₃₀₁ has a width in the T direction of 0, but here the subject is moving slightly. In consideration of the above, the region R ₃₀₁ has a width in the T direction that is not 0 but 2π / (2t ₀ ). The X and Y directions are limited to the range of −2π / (2r ₀ ) to + 2π / (2r ₀ ) as described with reference to FIG. This is because it does not exist in a range exceeding the range of −2π / (2r ₀ ) to + 2π / (2r ₀ ) (outside the range of the region R _{201 in} FIG. 2).

  Here, in the frequency domain, the data of the stationary part is distributed in the direction of the straight line (plane) of T = 0, so this direction is the main component (for example, the first data of the stationary part). Main component) direction.

Next, FIG. 4 shows a region R401 on the frequency domain in which data of a portion where a subject projected on a moving image moves at a speed (r ₀ / t ₀ ) / 2 is distributed.

  Here, in order to simplify the explanation as appropriate, the points on the frequency domain are expressed by using only two coordinates A and B in the X and Y directions, and (A, B). It expresses.

  The region R401 extends from the origin (0,0) to the point (π / r0, −π / (2t0)) and has a width of 2π / (2t0) in the T direction. .

It should be noted that if the subject does not deform and moves accurately at a constant linear velocity at a speed (r ₀ / t ₀ ) / 2, the region R401 has a width in the T direction of 0. The width in the T direction is not 0 but 2π / (2t0) in consideration of some deformation and the movement speed slightly deviating from (r ₀ / t ₀ ) / 2.

In the region R401, the X and Y directions are limited to the range of −2π / (2r ₀ ) to + 2π / (2r ₀ ), but this is limited to the range of the region R _{301 in} the X and Y directions. For the same reason that has been done.

Here, the data of the portion where the subject projected on the moving image moves at a speed (r ₀ / t ₀ ) / 2 is obtained from the origin (0,0) to the point (π / r0, −π / (2t0) ) Is distributed for the following reason.

That is, the waveform R _{501 in} FIG. 5 shows the distribution of the subject in the spatial direction x at a certain time t ₁ .

For example, when the subject is moving at a speed (r ₀ / t ₀ ) / 2, for example, in the spatial direction x, the moving subjects are waveform R ₅₀₁ , R ₅₀₂ , R ₅₀₃ , R ₅₀₄ , R ₅₀₅ , R ₅₀₆ ,...

Here, the waveform R ₅₀₁ represents the subject at a certain time (frame) t ₁ . Waveform R ₅₀₂ represents a subject at the next time (frame) t ₁ + t ₀ of time t ₁ of the waveform R _501, from the position of the waveform R _501, only r _0/2, moves in the x-direction Yes. Hereinafter, similarly, waveforms R ₅₀₃ , R ₅₀₄ , R ₅₀₅ , and R ₅₀₆ represent subjects at times t ₁ + 2t ₀ , t ₁ + 3t ₀ , t ₁ + 4t ₀ , and t ₁ + 5t _{0, respectively} . Yes.

The waveform in FIG. 5 is considered as a function of the spatial position x and time t, and this waveform is defined as Func (x, t). As is clear from the waveforms R ₅₀₁ , R ₅₀₂ , R ₅₀₃ , R ₅₀₄ , R ₅₀₅ , R ₅₀₆ ,..., Func (x, t) = Func (x − ((r ₀ / t ₀ ) / 2) × t, 0).

Assuming that the data on the frequency domain obtained by two-dimensional Fourier transform of the function Func with two variables x and t is (X, T), the component of (X, T) is apparent from the definition of Fourier transform. In addition, a component in which the period in the spatial direction x is X and the period in the time direction t is T is represented.

From the relationship Func (x, t) = Func (x − ((r ₀ / t ₀ ) / 2) × t, 0), (X, T) is a two-dimensional vector (r _0/2 , t ₀ ) Can be said to be orthogonal.

  Although described with reference to FIG. 5, the spatial direction is actually two-dimensional, and when the time axis is added, it becomes three-dimensional, and a three-dimensional Fourier transform must be considered. Then, it demonstrates again using FIG. FIG. 6 is a diagram for explaining three-dimensionally as in FIG. 1.

A waveform R ₆₀₁ on the two-dimensional space (x, y) in FIG. 6 shows the distribution in the spatial direction (x, y) of the subject at a certain time t ₁ .

For example, when the subject is moving at a speed (r ₀ / t ₀ ) / 2, the moving subject is represented by waveforms R ₆₀₁ , R ₆₀₂ , R ₆₀₃ , R _604,. Can do. Note that r ₀ is a two-dimensional vector on a two-dimensional space (x, y) unlike the description of FIG.

Here, the waveform R ₆₀₁ represents the subject at a certain time (frame) t ₁ . Waveform R ₆₀₂ represents a subject at the next time (frame) t ₁ + t ₀ of time t ₁ of the waveform R _601, from the position of the waveform R _601, is moved by r _0/2. Hereinafter, similarly, waveforms R ₆₀₃ and R ₆₀₄ represent subjects at times t ₁ + 2t ₀ and t ₁ + 3t _0, respectively.

The waveform in FIG. 6 is considered as a function of the spatial position (x, y) and time t, and this waveform is defined as Func ((x, y), t). As apparent from the waveforms R ₆₀₁ , R ₆₀₂ , R ₆₀₃ , R ₆₀₄ ,... In FIG. 6, Func ((x, y), t) = Func ((x, y) − ((r ₀ / t ₀ ) / 2) × t, 0).

  Now, if the data on the frequency domain obtained by three-dimensional Fourier transform of the function Func with three variables x, y and t is (X, Y, T), the component of (X, Y, T) is Fourier transform. As is clear from the definition of FIG. 4, a component in which the period in the spatial direction x is X, the period in the spatial direction y is Y, and the period in the time direction t is T is represented.

From the relationship Func ((x, y), t) = Func ((x, y)-((r ₀ / t ₀ ) / 2) × t, 0), (X, Y, T) is three-dimensional It can be said that it is orthogonal to the vector (r _0/2 , t ₀ ).

As shown in FIGS. 5 and 6, since (X, T) or (X, Y, T) is orthogonal to the vector (r _0/2 , t ₀ ), for example, as shown in FIG. There is data at a point (2π / (2r ₀ ), −2π / (4t ₀ )) (position indicated by a circle in FIG. 7) on the frequency domain, or a point (2π / (4r ₀ ), −2π / There is data on (8t ₀ )) (the position indicated by Δ in FIG. 7).

That is, each waveform moving at a speed (r ₀ / t ₀ ) / 2 is a straight line (plane) R passing through the origin (0,0) and the point (π / r0, -π / (2t0)). Distributed on ₇₀₁ .

Similarly, the data (waveform) of the other frequency component X of the portion moving at the speed (r ₀ / t ₀ ) / 2 is also distributed on the straight line R ₇₀₁ .

The moving parts at a rate _{(r 0 / t 0) /} 2 , when the time advances by t _0, since r _0/2 only spatial directions position shifts, as described above, in a frequency domain, the origin ( 0,0) and the point (π / r0, -π / ( 2t0) distributed on a straight line R ₇₀₁ through a).

In the frequency domain, the data of the portion moving at the speed (r ₀ / t ₀ ) / 2 is distributed in the direction of the straight line (plane) R ₇₀₁ (π / r0, −π / (2t0)). This direction (π / r0, −π / (2t0)) is the direction of the principal component of the data of the portion moving at the speed (r ₀ / t ₀ ) / 2.

Next, FIG. 8 shows a region R ₈ 01 on the frequency domain in which data of a portion in which a subject projected on a moving image moves at a speed r ₀ / t ₀ is distributed.

The region R ₈ 01 is a region extending in the direction from the origin (0,0) to the point (π / r0, −π / t0) and having a width of 2π / (2t0) in the T direction. .

That is, the portion moving at the speed r ₀ / t ₀ is shifted in position in the spatial direction by r ₀ when the time advances by t _0, so that the origin (0,0) and the point (π / r0, -π / t0).

Note that if the subject does not deform and moves accurately linearly at a constant velocity r ₀ / t ₀ , the region R ₈ 01 has a width of 0 in the T direction. In consideration of deformation and the movement speed slightly deviating from r ₀ / t ₀ , the width in the T direction is not 0 but 2π / (2t 0).

In the region R ₈ 01, the X and Y directions are limited to the range of −2π / (2r ₀ ) to + 2π / (2r ₀ ), which is the same as the region R _{301 in} FIG. For the same reason that the range of directions is limited.

Further, in the region R ₈ 01, the T direction is limited to a range of −2π / (2t ₀ ) to + 2π / (2t ₀ ), which is the frequency of the moving image data as described with reference to FIG. This is because the component does not exist in the T direction beyond the range of −2π / (2t ₀ ) to + 2π / (2t ₀ ) (outside the range of the region R _{201 in} FIG. 2).

Here, in the frequency domain, the data of the portion moving at the velocity r ₀ / t ₀ is the direction of the straight line (π / t) passing through the origin (0,0) and the point (π / r0, −π / t0). r0, -π / t0), this direction (π / r0, -π / t0) is the direction of the main component of the data of the portion moving at the speed r ₀ / t ₀ .

Next, FIG. 9 shows a region R ₉ 01 on the frequency domain data of the part object is projected onto a video is moving at about the speed 2r ₀ / t ₀ is distributed.

Region R ₉ 01 is the point from the origin (0,0) (π / r0, -2π / t0) an area extending in the direction of, and is a region having a width of 2π / (2t0) in the T direction .

That is, the portion moving at the speed 2r ₀ / t ₀ is shifted in the spatial direction by 2r ₀ when the time advances by t _0, so that the origin (0,0) and the point (π / r0, -2π / t0).

Incidentally, without the subject is deformed, if exactly uniform linear motion at a speed 2r ₀ / t _0, the region R ₉ 01, although the width of the T direction is 0, here, some of the subject In consideration of deformation and the movement speed slightly deviating from r ₀ / t ₀ , the width in the T direction is not 0 but 2π / (2t 0).

In the region R ₉ 01, the T direction is limited to the range of −2π / (2t ₀ ) to + 2π / (2t ₀ ). This is because the frequency of the moving image data is as described with reference to FIG. This is because the component does not exist in the T direction beyond the range of −2π / (2t ₀ ) to + 2π / (2t ₀ ) (outside the range of the region R _{201 in} FIG. 2).

Here, in the frequency domain, the data of the portion moving at the speed 2r ₀ / t ₀ is the direction of the straight line (π / 0) passing through the origin (0,0) and the point (π / r0, -2π / t0). r0, -2π / t0), this direction (π / r0, -2π / t0) is the direction of the main component of the data of the portion moving at the speed 2r ₀ / t ₀ .

As described above, when the subject is stationary, and when the subject is moving at a speed (r ₀ / t ₀ ) / 2, r ₀ / t ₀ , 2r ₀ / t ₀ , such a subject is The distribution on the frequency domain of the data of the portion of the moving image being projected has been described, but the data of the portion moving at other speeds is similarly distributed in the frequency domain.

  Next, human visual effects will be described.

  The human eye recognizes the light emitted from the object by integrating it in the time direction. This integration function (visual integration function) in the human eye is called the Block's Law. The integration time of light in the human eye is approximately 1/60 seconds, although it varies depending on the environment.

Here, if t ₀ is about 1/240 seconds as described above, the integration time of light in the human eye can be expressed as 4 × t ₀ seconds.

An image in which a stationary subject is projected in a moving image is recognized by a human gaze without moving the line of sight. This is equivalent to an image obtained by filtering with a low-pass filter of about “4 × t ₀ ” seconds.

The pass band of this low-pass filter can be represented as a region R ₁₀₀₁ shown in FIG. 10 on the frequency domain.

In the region R _{1001 in} FIG. 10, the range in the X and Y directions is −2π / (2r ₀ ) to + 2π / (2r ₀ ), which is the same as the region R _{201 in} FIG. 2, and the range in the T direction is − ( The region is 2π / (4t ₀ )) / 2 to (2π / (4t ₀ )) / 2. The width in the T direction of this region R ₁₀₀₁ is 2π / (4t ₀ ), which represents the integration of the time of 4t ₀ .

As shown in FIG. 3, the data of the still part of the moving image exists in the region R ₃₀₁ , but the human recognizes only the information (data) in the region R ₁₀₀₁ in the region R _301. Can not do it. Therefore, information outside the region R ₁₀₀₁ in the region R ₃₀₁ is useless information for humans.

Next, while moving the line of sight in the direction in which the subject moves in the part of the moving image where the subject being projected is moving at a speed of “(r ₀ / t ₀ ) / 2” An image that is recognized by gazing, that is, an image that a human recognizes by following vision, uses data of a portion moving at a speed of “(r ₀ / t ₀ ) / 2” in the time direction by a visual integration function. This is equivalent to an image obtained by filtering with a low-pass filter of about “4 × t ₀ ” seconds.

However, in the following tracking in this case, the gazing point moves at a speed of about “(r ₀ / t ₀ ) / 2” with the passage of time. Therefore, the pass band of the low-pass filter is as shown in FIG. Can be represented as a region R ₁₁₀₁ shown in FIG.

A region R _{1101 in} FIG. 11 is a region extending in the direction from the origin (0,0) to the point (π / r0, −π / (2t0)), similarly to the region R401 in FIG. Further, a region R ₁₁₀₁ is a region having a width of 2π / (4t0) in the T direction, and this represents an integration of time of 4t ₀ . Note that the area R ₁₁₀₁ is limited to the area R ₂₀₁ where the moving image data exists, as shown in FIG.

Speed _{"(r 0 / t 0) /} 2 " data of the portion moving at a degree, as shown in FIG. 4, are present in the region R _401, human, of its region R ₄₀₁ Only the information in the region R ₁₁₀₁ can be recognized. Therefore, information outside the region R ₁₁₀₁ in the region R ₄₀₁ is useless information for humans.

Next, the image in which the subject projected on the moving image moves at a speed of about “r ₀ / t ₀ ” by human follow-up is still the visual integration. This function is equivalent to an image obtained by filtering data of a portion moving at a speed of about “r ₀ / t ₀ ” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

However, in the following tracking in this case, the gazing point moves at a speed of about “r ₀ / t ₀ ” with the passage of time, so the pass band of the low-pass filter is the region R shown in FIG. 12 on the frequency domain. Can be represented as ₁₂₀₁ .

Region R _{1201 in} FIG. 12 is a region extending in the direction from the origin (0, 0) to the point (π / r 0, −π / t 0), similarly to the region R ₈ 01 in FIG. Further, the region R ₁₂₀₁ is a region having a width of 2π / (4t0) in the T direction, and this represents the integration of time of 4t ₀ . Note that the region R ₁₂₀₁ is limited to the region R ₂₀₁ where the moving image data exists as shown in FIG.

As shown in FIG. 8, the data of the portion moving at the speed “r ₀ / t ₀ ” is present in the region R ₈₀₁ , but the human is the region R _{1201 in the} region R _801. Only the information inside can be recognized. Therefore, information outside the region R ₁₂₀₁ in the region R ₈₀₁ is useless information for humans.

Next, an image that is recognized by humans following the moving part of the moving subject at a speed of about “2r ₀ / t ₀ ” in the moving image is still a visual integral. This function is equivalent to an image obtained by filtering data of a portion moving at a speed of about “2r ₀ / t ₀ ” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

However, in the follow-up view in this case, since the gazing point moves at a speed of about “2r ₀ / t ₀ ” with time, the pass band of the low-pass filter is the region R shown in FIG. 13 on the frequency domain. Can be represented as ₁₃₀₁ .

A region R _{1301 in} FIG. 13 is a region extending in the direction from the origin (0,0) to the point (π / r0, −2π / t0), similarly to the region R ₉₀₁ in FIG. Further, the region R ₁₃₀₁ is a region having a width of 2π / (4t0) in the T direction, and this represents the integration of time of 4t ₀ . Note that the region R ₁₃₀₁ is limited to the region R ₂₀₁ where the moving image data exists as shown in FIG.

As shown in FIG. 9, the data of the portion moving at the speed “2r ₀ / t ₀ ” exists in the region R ₉₀₁ , but the human is the region R _{1301 in the} region R _901. Only the information inside can be recognized. Therefore, information outside the region R ₁₃₀₁ in the region R ₉₀₁ is useless information for humans.

  Next, a case where a human performs fixed vision will be described. Here, fixed vision is different from follow-up vision in the case of moving the line of sight different from the movement of the subject. For example, humans do not move the line of sight (keep the viewing direction constant) Applicable when watching a video.

An image that is recognized by a human fixed view of a moving part of a moving image at a speed of about “(r ₀ / t ₀ ) / 2” in a moving image is obtained by a visual integration function. This is equivalent to an image obtained by filtering data in a portion moving at a speed of “(r ₀ / t ₀ ) / 2” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

The pass band of the low-pass filter can be represented by the same region on the frequency domain as that when a stationary portion is watched, that is, a region R ₁₀₀₁ in FIG.

Here, FIG. 14 shows a region R ₄₀₁ (FIG. 4) in which data of a portion moving at a speed of “(r ₀ / t ₀ ) / 2” in the frequency domain is distributed, and a case where a human performs fixed vision. A region R ₁₀₀₁ (FIG. 10) as a pass band of the low-pass filter is shown.

Humans, of area R _401, information in the region R _1001, that is, can not be a region R ₄₀₁ and R ₁₀₀₁ are only recognized information in the region R ₁₄₀₁ of the portions overlapping. Therefore, information outside the region R ₁₀₀₁ in the region R ₄₀₁ is useless information for humans.

In FIG. 14, a region R ₁₀₀₁ as a pass band of the low-pass filter when a human performs fixed vision is a region R ₁₄₀₂ and R ₁₄₀₃ (parts R ₁₄₀₂ and R _{1403 in the} figure) that do not overlap with the region R ₄₀₁ where the moving image data exists. In this case, there is no moving image data in the regions R ₁₄₀₂ and R ₁₄₀₃ .

Further, as described above, when a human is fixedly viewing a portion that is moving at a speed of about “(r ₀ / t ₀ ) / 2”, the human gaze direction is the moving subject (moving subject). Therefore, even a slightly blurred image is recognized as a good moving image.

Therefore, the data in the region R ₁₄₀₁ in FIG. 14 has a low-pass filter when the gazing point moves at a speed of “(r ₀ / t ₀ ) / 2”, that is, the region R _{1101 in} FIG. Even if it is filtered with a low-pass filter, it is recognized as a moving image with good image quality.

From the above, as shown in FIG. 15, if there is data in a region R ₁₅₀₁ of the region R ₁₄₀₁ that overlaps the region R ₁₁₀₁ (shown in black in the drawing), the speed “(r ₀ / A portion moving at about t ₀ ) / 2 ”is recognized as a moving image with good image quality.

Therefore, in the case where a portion of a moving image moving at a speed of “(r ₀ / t ₀ ) / 2” is fixedly viewed by a human, the data of that portion, that is, the data in the region R ₄₀₁ is fixed. Region R ₁₀₀₁ as the pass band of the low-pass filter when viewing is performed, and region R as the pass band of the low-pass filter when the gazing point moves at a speed of “(r ₀ / t ₀ ) / 2” The data in the area excluding the area R ₁₅₀₁ that overlaps the area ₁₁₀₁ is useless (even if it does not, it hardly affects the image quality recognized by humans).

Next, an image that is recognized by a human fixed view of a moving part of a moving image at a speed of about “r ₀ / t ₀ ” in a moving image is obtained by a visual integration function, This is equivalent to an image obtained by filtering data in a portion moving at a speed of about “r ₀ / t ₀ ” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

The pass band of the low-pass filter can be represented by the same region on the frequency domain as that when a stationary portion is watched, that is, a region R ₁₀₀₁ in FIG.

Here, FIG. 16 shows a region R ₈₀₁ (FIG. 8) in which data of a portion moving at a speed of “r ₀ / t ₀ ” in the frequency domain is distributed, and a low-pass filter when a human performs fixed vision. A region R ₁₀₀₁ (FIG. 10) as a pass band is shown.

Humans, of area R _801, information in the region R _1001, that is, can not be a region R ₈₀₁ and R ₁₀₀₁ are only recognized information in the region R ₁₆₀₁ of the portions overlapping. Therefore, information outside the region R ₁₀₀₁ in the region R ₈₀₁ is useless information for humans.

In FIG. 16, a region R ₁₀₀₁ as a pass band of the low-pass filter when a human performs fixed vision is a portion of regions R ₁₆₀₂ and R ₁₆₀₃ that do not overlap with the region R ₈₀₁ where moving image data exists (in the drawing) In this case, there is no moving image data in these areas R ₁₆₀₂ and R ₁₆₀₃ .

In addition, as described above, when a human is fixedly viewing a portion that is moving at a speed of about “r ₀ / t ₀ ”, the direction of the human gaze is the movement of the moving subject (moving subject). Since it moves differently, even a slightly blurred image is recognized as a good moving image.

Therefore, the data in the region R ₁₆₀₁ in FIG. 16 is filtered by a low-pass filter when the gazing point moves at a speed of “r ₀ / t ₀ ”, that is, a low-pass filter having the region R _{1201 in} FIG. Even so, it is recognized as a moving image with good image quality.

From the above, as shown in FIG. 17, if there is data in a region R ₁₇₀₁ (shown in black in the drawing) that overlaps the region R ₁₂₀₁ in the region R ₁₆₀₁ , the speed “r ₀ / t A portion moving at about “ ₀ ” is recognized as a moving image with good image quality.

Therefore, when a human views a moving part of a moving image at a speed of about “r ₀ / t ₀ ”, the fixed view is performed from the data of that part, that is, the data in the region R ₈₀₁ . Region R ₁₀₀₁ as the pass band of the low-pass filter in the case where the gazing point moves and the region R ₁₂₀₁ as the pass band of the low-pass filter when the gazing point moves at a speed of about “r ₀ / t ₀ ” The data in the area excluding ₁₇₀₁ is useless.

Next, an image that is recognized by a human fixed view of a moving part of a moving image at a speed of about “2r ₀ / t ₀ ” in a moving image is obtained by a visual integration function, This is equivalent to an image obtained by filtering data in a portion moving at a speed of about “r ₀ / t ₀ ” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

The pass band of the low-pass filter can be represented by the same region on the frequency domain as that when a stationary portion is watched, that is, a region R ₁₀₀₁ in FIG.

Here, FIG. 18 shows a region R ₉₀₁ (FIG. 9) in which data of a portion moving at a speed of about “2r ₀ / t ₀ ” is distributed in the frequency domain, and a low-pass filter when a human performs fixed vision. A region R ₁₀₀₁ (FIG. 10) as a pass band is shown.

Humans, of area R _901, information in the region R _1001, that is, can not be a region R ₉₀₁ and R ₁₀₀₁ are only recognized information in the region R ₁₈₀₁ of the portions overlapping. Therefore, information outside the region R ₁₀₀₁ in the region R ₉₀₁ is useless information for humans.

In FIG. 18, a region R ₁₀₀₁ as a pass band of the low-pass filter when a human performs fixed vision is a region R ₁₈₀₂ and R _{1803 of} portions that do not overlap with the region R ₉₀₁ where moving image data exists (in the drawing) In this case, there is no moving image data in the regions R ₁₈₀₂ and R ₁₈₀₃ .

In addition, as described above, when a human is fixedly viewing a portion that is moving at a speed of about “2r ₀ / t ₀ ”, the direction of the human line of sight is the movement of the moving subject (moving subject). Since it moves differently, even a slightly blurred image is recognized as a good moving image.

Accordingly, the data in the region R ₁₈₀₁ in FIG. 18 is filtered by a low-pass filter when the point of interest moves at a speed of about “2r ₀ / t ₀ ”, that is, a low-pass filter having the region R _{1301 in} FIG. Even so, it is recognized as a moving image with good image quality.

From the above, as shown in FIG. 19, if there is data in a region R ₁₉₀₁ (shown in black in the drawing) that overlaps the region R ₁₃₀₁ in the region R ₁₈₀₁ , the speed “2r ₀ / t A portion moving at about “ ₀ ” is recognized as a moving image with good image quality.

Therefore, when a human views a moving part of a moving image at a speed of about “2r ₀ / t ₀ ”, the fixed view is performed from the data of that part, that is, the data in the region R ₉₀₁ . Region R ₁₀₀₁ as the pass band of the low-pass filter when the gazing point moves and the region R ₁₃₀₁ as the pass band of the low-pass filter when the gazing point moves at a speed of about “2r ₀ / t ₀ ” The data in the area excluding ₁₉₀₁ is useless.

  The ranges (regions) on the frequency domain that can be recognized by humans in the case of fixed vision and follow-up vision are summarized as shown in FIGS.

  That is, FIG. 20 shows an area where there is data that can be recognized by humans in a portion where a stationary object (subject) is projected in a moving image.

As shown in FIG. 20, the region on the frequency domain of the data that can be recognized by humans for a portion where a stationary object is projected in the moving image is a region R ₁₀₀₁ shown in FIG. The data outside the area R ₁₀₀₁ is useless.

Next, FIG. 21 shows data in which a person can recognize a portion of a moving image where a subject projected on the moving image moves at a speed of “(r ₀ / t ₀ ) / 2”. The area to be shown is shown.

A region on the frequency domain of data that can be recognized by humans by following vision for a portion moving at a speed of “(r ₀ / t ₀ ) / 2” in the moving image is a region R shown in FIG. ₁₁₀₁ . Furthermore, the region on the frequency domain of data that can be recognized by fixed vision is a region R ₁₅₀₁ shown in FIG. 15, and this region R ₁₅₀₁ is included in the region R ₁₁₀₁ shown in FIG.

Therefore, the region on the frequency domain of the data that can be recognized by humans for the portion moving at the speed “(r ₀ / t ₀ ) / 2” is shown in FIG. 11 as shown in FIG. a region R _1101, data in the region other than the region R ₁₁₀₁ is waste.

Next, FIG. 22 shows an area where data that can be recognized by humans exists in a portion of the moving image where the subject projected on the moving image moves at a speed of about “r ₀ / t ₀ ”. ing.

A region on the frequency domain of data that can be recognized by a human by tracking vision for a portion moving at a speed of “r ₀ / t ₀ ” in the moving image is a region R ₁₂₀₁ shown in FIG. Furthermore, the region on the frequency domain of data that can be recognized by fixed vision is a region R ₁₇₀₁ shown in FIG. 17, and this region R ₁₇₀₁ is included in the region R ₁₂₀₁ shown in FIG.

Therefore, as shown in FIG. 22, the region on the frequency domain of the data that can be recognized by humans for the portion moving at the speed “r ₀ / t ₀ ” is a region R ₁₂₀₁ shown in FIG. _Yes , the data outside the area R ₁₂₀₁ is useless.

Next, FIG. 23 shows an area where data that can be recognized by humans exists in a portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”. ing.

A region on the frequency domain of the data that can be recognized by humans by following vision for a portion moving at a speed of “2r ₀ / t ₀ ” in the moving image is a region R ₁₃₀₁ shown in FIG. Furthermore, the region on the frequency domain of data that can be recognized with fixed vision is a region R ₁₉₀₁ shown in FIG. 19, and this region R ₁₉₀₁ is included in the region R ₁₃₀₁ shown in FIG.

Therefore, as shown in FIG. 23, the region on the frequency domain of the data that can be recognized by humans for the portion moving at the speed “2r ₀ / t ₀ ” is a region R ₁₃₀₁ shown in FIG. _Yes , data outside the area R ₁₃₀₁ is useless.

In the above case, the width in the T direction of the region R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. 23 is 2π / (4t ₀ ). However, this width may be a width corresponding to the integration time by the visual integration function, and is not limited to 2π / (4t ₀ ).

  The same can be said for the data of the moving part of the video at other speeds.

  As described above, by utilizing the human visual effect (visual characteristics), that is, the visual integration function, on the frequency domain, moving image data can be recognized as data in a region that can be recognized by humans and other waste. Data.

  Then, if only data in a region that can be recognized by humans is extracted from video data on the frequency domain, that is, if unnecessary data other than that is deleted, image quality is not degraded ( The image quality recognized when viewing the video), and the video data can be compressed.

  FIG. 24 shows an example of the configuration of an image processing system that compresses moving image data using a visual integration function and decodes (decompresses) the compressed moving image data.

In FIG. 24, the transmission apparatus 1 is supplied with moving image data having a high frame rate such as 1 / t ₀ = 240 fps. The transmission apparatus 1 filters the moving image data supplied to the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R _{1301 in} FIG. Band-pass filter or band-limiting filter), and outputs the data in the frequency domain that can be recognized by humans. Data output from the transmission device 1 is recorded on a recording medium 11 such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magneto Optical) disk, DVD (Digital Versatile Disc), magnetic disk, or semiconductor memory. Alternatively, the data is transmitted via a transmission medium 12 such as a telephone line, a terrestrial wave, a satellite line, the Internet, a wired line, or a wireless LAN (Local Area Network).

  The receiver 2 is supplied with data reproduced from the recording medium 11 or data transmitted via the transmission medium 12. The receiving device 2 receives the data supplied thereto, performs predetermined processing, and the moving image data obtained as a result is composed of, for example, a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), or the like. It is supplied to the display device 3 and displayed.

  In FIG. 24, the transmission device 1, the reception device 2, and the display device 3 can be configured as physically independent devices. Further, the transmitter 1 and the receiver 2 can be physically configured as one device as a whole as shown by being surrounded by a dotted line in FIG. Furthermore, the whole of the transmission device 1, the reception device 2, and the display device 3, or the reception device 2 and the display device 3 can be physically configured as one device.

  Next, FIG. 25 illustrates a first configuration example of the transmission device 1 of FIG.

  The buffer unit 21 is supplied with the high frame rate moving image data supplied to the transmission apparatus 1. The buffer unit 21 sequentially stores the moving image data supplied thereto.

  The filter unit 22 appropriately reads out the moving image data stored in the buffer unit 21, filters the moving image data according to the filter information supplied from the filter generation unit 23, which will be described later, and obtains the moving image data obtained by the filtering, This is supplied to the encoding unit 24.

  The filter generation unit 23 appropriately reads the moving image data stored in the buffer unit 21, generates filter information that is filter information for filtering each part of the moving image data, and supplies the filter information to the filter unit 22.

  That is, the filter generation unit 23 includes a principal component direction acquisition unit 31 and a filter information supply unit 32.

  The principal component direction acquisition unit 31 acquires the principal component direction which is the direction of the principal component on the frequency domain for each part of the moving image data read from the buffer unit 21, and filters the information on the principal component direction. The information is supplied to the information supply unit 32.

The filter information supply unit 32 is a region extending in the principal component direction in the frequency domain according to the information on the principal component direction from the principal component direction acquisition unit 31, and has a specific width in the direction of the frequency axis T in the time direction. For example, a region having 2π / (4t ₀ ), that is, a region R _{1001 in} FIG. 20, a region R ₁₁₀₁ in FIG. 21, a region R ₁₂₀₁ in FIG. 22, and a region R _{1301 in} FIG. The pass band is determined, and the pass band is output to the filter unit 22 as filter information.

  The encoding unit 24 encodes the moving image data supplied from the filter unit 22 by a predetermined encoding (encoding) method such as MPEG (Moving Picture Experts Group) 1 or 2, and outputs the encoded data obtained as a result. To do. This encoded data is recorded on the recording medium 11 of FIG. 24 or transmitted via the transmission medium 12.

  Next, processing of the transmission apparatus 1 in FIG. 25 will be described with reference to the flowchart in FIG.

The buffer unit 21 is supplied with moving image data having a high frame rate of 240 fps, for example, at a frame rate of 1 / t ₀ and sequentially stored.

  In step S1, the filter generation unit 23 reads the moving image data stored in the buffer unit 21, inputs the moving image data to itself (filter generation unit 23), and proceeds to step S2.

In step S2, a region in which the filter generation unit 23 (its principal component direction acquisition unit 31 and filter information supply unit 32) can recognize each part of the moving image data input in step S1 on the frequency domain. That is, for example, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. 23 are obtained as filter passbands, and the passbands are used as filter information. The “processing for obtaining a pass band of necessary information” described later with reference to FIG.

  In step S3, the filter unit 22 reads data used for filtering by the filter of the filter information supplied from the filter generation unit 23 in step S2 from the buffer unit 21, and the filter generation unit 23 in step S2 for the data. Downsampling is performed by thinning out the number of samples in the time direction to 1/4 while applying the filter of the passband represented by the filter information supplied from.

In other words, the filter unit 22 filters the data read from the buffer unit 21 with a filter in the passband represented by the filter information supplied from the filter generation unit 23 in step S2, while sampling in the time direction, that is, the frame period. However, moving image data having a low frame rate of 4t ₀ , which is four times the original moving image data, is output to the encoding unit 24, and the process proceeds to step S4.

  In step S4, the encoding unit 24 encodes the moving image data from the filter unit 22, and outputs the resulting encoding.

  Note that the processing in steps S1 to S4 is performed for all the data of each part of the moving image stored in the buffer unit 21.

  Further, the downsampling in step S3 is a process of outputting a filtering result of one frame for every four frames of moving image data stored in the buffer unit 21. Accordingly, for every four frames of the moving image data stored in the buffer unit 21, one frame out of the four frames may be filtered by a passband filter represented by the filter information supplied from the filter generation unit 23. .

As described above, regions that can be recognized by humans on the frequency domain, that is, for example, region R _{1001 in} FIG. 20, region R ₁₁₀₁ in FIG. 21, region R ₁₂₀₁ in FIG. 22, region R _{1301 in} FIG. Since the moving image data is filtered by the filter having the pass band and further down-sampling is performed, the moving image data can be reduced in consideration of the visual effect of the human eye.

That is, as shown in FIG. 2, the regions on the frequency domain that humans can visually recognize are 2 × 2π / (2r ₀ ) in the X and Y directions and 2 × 2π / (2t in the T direction. ₀ ) of the region R ₂₀₁ centered on the origin, the highest quality video that can be recognized by humans is the video with a frame rate of 1 / t ₀ and a pixel pitch r ₀ in the spatial direction. It is.

If the visual effects described with reference to FIGS. 10 to 23 are not considered, for example, if the frame rate is reduced below 1 / t ₀ by down-sampling the moving image in the time direction, the frame rate is reduced. When a reduced moving image is displayed, a person who views the moving image recognizes the deterioration of the image quality.

On the other hand, in the transmission apparatus 1, the processing in consideration of the visual effect described with reference to FIGS. 10 to 23, that is, an area that can be recognized by a human on the frequency domain, for example, an area R _{1001 in} FIG. In addition, since the moving image data is filtered by a filter having the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22 and the region R _{1301 in} FIG. 23 as a pass band, the frame rate of the moving image is reduced to 1 / (4t ₀ ). Even if it is reduced, that is, even if the data amount of the moving image data is reduced to ¼, it is possible to display a moving image in which humans do not feel deterioration in image quality.

That is, FIG. 27 to FIG. 30 show a high frame rate by a filter in which the region R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. videos while filtering data (frame rate video 1 / t _0), low frame rate video obtained by performing 1/4 down-sampling in the time direction (frame rate 1 / (4t ₀₎ Video) distribution on the frequency domain.

First, the data of the stationary part in the moving image is filtered by the filter unit 22 with a filter having a region R _{1001 in} FIG. 20 as a pass band. Therefore, the moving image data after filtering exists only in the region R ₁₀₀₁ .

When the moving image data in the region R ₁₀₀₁ is downsampled to 1/4 in the time direction, the moving image data is sampled at intervals of r _{0 in} the spatial direction x, y and 4 × in the time direction t. t Data sampled at ₀ intervals.

For this reason, as shown in FIG. 27, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / (4t ₀ ) in the T direction. Occurs.

However, since the region R ₁₀₀₁ is a region in which the X and Y directions are 2π / r ₀ and the T direction is 2π / (4t ₀ ) as described with reference to FIG. 10, the aliasing components do not overlap.

  In FIG. 27, the shaded part is the part where the down-sampled video data exists.

Next, the data of the portion of the moving image where the subject projected on the moving image moves at a speed of about “(r ₀ / t ₀ ) / 2” is stored in the region R _{1101 of} FIG. Filtered by a filter as a pass band. Therefore, the moving image data after filtering exists only in the region R ₁₁₀₁ .

When the moving image data in the region R ₁₁₀₁ is downsampled to 1/4 in the time direction, the moving image data is sampled at intervals of r _{0 in} the spatial direction x, y and 4 × in the time direction t. t Data sampled at ₀ intervals.

Therefore, as shown in FIG. 28, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / (4t ₀ ) in the T direction. Occurs.

However, the region R ₁₁₀₁ has a width of 2π / (4t0) in the T direction from the straight line connecting the origin (0,0) and the point (π / r0, −π / (2t0)), as described in FIG. And the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ) and the T direction is-(π / t ₀ ) to + (π / t ₀ ). Since it is a range area, the aliasing components do not overlap.

  In FIG. 28, the shaded part is the part where the down-sampled moving image data exists.

Next, the data of the portion of the moving image where the subject projected on the moving image is moving at a speed of about “r ₀ / t ₀ ” has a region R _{1201 in} FIG. Filtered by the filter. Therefore, the moving image data after filtering exists only in the region R ₁₂₀₁ .

When the moving image data in the region R ₁₂₀₁ is downsampled to 1/4 in the time direction, the moving image data is sampled at intervals of r _{0 with} respect to the spatial directions x and y, and 4 × with respect to the time direction t. t Data sampled at ₀ intervals.

Therefore, as shown in FIG. 29, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / (4t ₀ ) in the T direction. Occurs.

However, the region R ₁₂₀₁ has a width of 2π / (4t0) in the T direction from the straight line connecting the origin (0,0) and the point (π / r0, -2π / (2t0)), as described in FIG. And the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ) and the T direction is-(π / t ₀ ) to + (π / t ₀ ). Since it is a range area, the aliasing components do not overlap.

  In FIG. 29, the shaded portion is the portion where the down-sampled moving image data exists.

Next, the data of the portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ” has a region R _{1301 in} FIG. Filtered by the filter. Therefore, the moving image data after filtering exists only in the region R ₁₃₀₁ .

When the moving image data in the region R ₁₃₀₁ is downsampled to 1/4 in the time direction, the moving image data is sampled at intervals of r _{0 in} the spatial direction x, y and 4 × in the time direction t. t Data sampled at ₀ intervals.

Therefore, as shown in FIG. 30, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / (4t ₀ ) in the T direction. Occurs.

However, the region R ₁₃₀₁ has a width of 2π / (4t0) in the T direction from the straight line connecting the origin (0,0) and the point (π / r0, −2π / t0), as described in FIG. area, and, X, Y direction, - at (π / r ₀₎ to _{+ (π / r 0),} T direction, - the range of (π / t ₀₎ to + (π / t ₀₎ Since it is a region, the aliasing components do not overlap.

  In FIG. 30, the shaded part is the part where the down-sampled moving image data exists.

As shown in FIGS. 27 to 30 above, the fact that the aliasing components of the downsampled video data do not overlap in the frequency domain means that the original data, that is, 20 area R ₁₀₀₁ , area R ₁₁₀₁ in FIG. 21, area R ₁₂₀₁ in FIG. 22, and area R ₁₃₀₁ in FIG. 23 can be extracted. This means that necessary information (information that can be recognized by humans) taking into consideration human visual characteristics is accurately held.

  Note that in order to decode (restore) the moving image data after downsampling into moving image data before downsampling, information on what kind of filter is applied to the data of each part of the moving image is required.

Therefore, in the transmission device 1 in FIG. 25, the filter information supply unit 32 transmits to the encoding unit 24 the passband as the filter information of the filter applied to the data of each part of the moving image, for example, the region R in FIG. ₁₀₀₁ , information on the area R ₁₁₀₁ in FIG. 21, area R ₁₂₀₁ in FIG. 22, area R _{1301 in} FIG. 23, and information on the principal component directions of the areas R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , and R ₁₃₀₁ , In the encoding unit 24, the information can be multiplexed with the encoded data and output.

  Next, with reference to the flowchart of FIG. 31, the “process for obtaining the passband of necessary information” performed by the filter generation unit 23 in step S2 of FIG. 26 will be described.

  First, in step S11, the principal component direction acquisition unit 31 of the filter generation unit 23 (FIG. 25) performs the frequency axis X in the spatial directions x and y for each part of the moving image data input in step S1 of FIG. , Y, and the principal component direction in the frequency domain defined by the frequency axis T in the time direction t are acquired, supplied to the filter information supply unit 32, and the process proceeds to step S12. The principal component direction can be obtained, for example, by performing Fourier transform (three-dimensional Fourier transform) on the three-dimensional direction of the spatial direction x, y and the time direction t.

In step S12, the filter information supply unit 32 is a region extending in the principal component direction from the principal component direction acquisition unit 31 from the origin (0,0) in the frequency domain, and 2π / (4 × t in the T direction. ₀ ), the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ), and the T direction is-(π / t ₀ ) to + (π / t ₀ ). a range of area, i.e., and a region R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R ₁₃₀₁ in FIG. 23, determined as the pass band of the filter, the flow proceeds to step S13.

  In step S13, the filter information supply unit 32 supplies the filter pass band (information representing) obtained in step S12 to the filter unit 22 as filter information, and performs a “process for obtaining the pass band of necessary information”. finish.

According to the “process for obtaining the pass band of necessary information” as described above, for example, the region R _{1001 in} FIG. 20 is used as filter information for a portion where a stationary subject is projected in a moving image. Can be sought. Further, for a portion where the subject projected in the moving image moves at a speed of “(r ₀ / t ₀ ) / 2”, the region R _{1101 in} FIG. 21 can be obtained as filter information. Furthermore, the region R _{1201 in} FIG. 22 can be obtained as filter information for a portion where the subject projected in the moving image moves at a speed of about “r ₀ / t ₀ ”. Further, for a portion where the subject projected in the moving image moves at a speed of about “2r ₀ / t ₀ ”, the region R _{1301 in} FIG. 23 can be obtained as filter information. Even in a portion of the moving image where the subject is moving at other speeds, a region in the frequency domain that can be recognized by humans can be obtained as filter information.

  In the present embodiment, in the filter information supply unit 32 of the filter generation unit 23 (FIG. 25), the region on the frequency domain extending in the main component direction supplied from the main component direction acquisition unit 31 is defined as the passband of the filter. Although the pass band is supplied to the filter unit 22 as filter information, the filter information supply unit 32 obtains filter information as the pass band as described above for each of a plurality of principal component directions in advance. It may be stored and the filter information corresponding to the principal component direction supplied from the principal component direction acquisition unit 31 may be selected from the stored filter information and supplied to the filter unit 22.

  Further, the filter generation unit 23 can be configured as an independent device that outputs filter information different from the transmission device 1.

  By the way, in step S11 of FIG. 31, the principal component direction of each part of the moving image data is obtained by the three-dimensional Fourier transform. However, since the calculation amount of the three-dimensional Fourier transform is enormous, the principal component in step S11 The amount of calculation required for the process of acquiring the direction is enormous.

  A method for acquiring the principal component direction with a smaller amount of calculation than in the case of performing the three-dimensional Fourier transform in step S11 of FIG. 31 will be described with reference to the flowchart of FIG.

  First, in step S21, the principal component direction acquisition unit 31 of the filter generation unit 23 (FIG. 25) stores the data of each frame of the moving image data stored in the buffer unit 21, for example, horizontal × vertical is 16 × 16. Dividing into blocks such as pixels, the process proceeds to step S22. Here, the image data of each block is the data of each part of the above-described moving image data.

  Here, the block does not need to be composed of a plurality of pixels, and may be one pixel.

  In step S22, the principal component direction acquisition unit 31 sequentially sets each block obtained in step S21 as a target block, and for the target block, a frame subsequent to the frame of the target block (hereinafter, referred to as a target frame as appropriate). Correlation information representing the correlation with is obtained. Further, in step S22, the principal component direction acquisition unit 31 obtains the position in the spatial direction on the frame next to the frame of interest where the correlation represented by the correlation information of the block of interest is the highest. That is, in step S22, the principal component direction acquisition unit 31 detects a two-dimensional motion vector in the spatial direction x, y of the block of interest by so-called block matching or the like.

  Here, as the correlation information, a so-called correlation coefficient can be adopted. Here, considering the calculation cost, for example, the target block is placed in the spatial direction x, y within the search range with the motion vector. , The square error of the pixel value of each pixel of the target block at the position shifted by u, v and the pixel of the next frame of the target frame at the same position as the pixel, the sum of absolute differences, etc. adopt. In this case, the position u, v in the spatial direction where the “value” of the correlation information is the minimum is the position in the spatial direction on the next frame of the frame of interest where the correlation represented by the correlation information is the highest.

Note that the spatial direction positions u and v at which the correlation represented by the correlation information is maximum (correlation information value is minimum) are represented by u ₀ and v ₀ , respectively, in the spatial directions x and y detected in step S22. The motion vector is represented as (u ₀ , v ₀ ).
The

Thereafter, the process proceeds from step S22 to S23, and the principal component direction acquisition unit 31 adds the frame period t ₀ of the original moving image data to the motion vector (u ₀ , v ₀ ) detected in step S22 and the component in the time direction t. The direction orthogonal to the direction of the three-dimensional motion vector (u ₀ , v ₀ , t ₀ ) added as is detected as the principal component direction, supplied to the filter information supply unit 32, and the process is terminated.

According to the above processing, the principal component direction of the block is detected as a plane (expansion direction) perpendicular to the motion vector (u ₀ , v ₀ , t ₀ ).

That is, when a motion vector (u ₀ , v ₀ , t ₀ ) is detected for the block of interest, ideally, in a three-dimensional space defined by the spatial direction x, y and the temporal direction t, the motion vector ( u ₀ , v ₀ , t ₀ ) in the direction of the pixel value of the target block, that is, the value of the video data at the point (x, y, t) where the target block is This is the same as the value of the moving image data at the point (x + mu ₀ , y + mv ₀ , t + mt ₀ ) on the three-dimensional space represented by x, y, t.

Therefore, the frequency component of the data of the spatio-temporal block of interest, that is, the data obtained by converting the spatio-temporal block of interest data into the data on the frequency domain is the motion vector (u ₀ , v ₀ , t ₀ ). It exists only on a vertical plane.

In the method according to the flowchart of FIG. 32, this is used to detect a plane perpendicular to the motion vector (u ₀ , v ₀ , t ₀ ) as the principal component direction. Therefore, in this case, in order to acquire the principal component direction, it is only necessary to detect a motion vector, and it is not necessary to perform a three-dimensional Fourier transform. Therefore, the principal component direction can be detected with a small amount of calculation.

  As the correlation information, for example, a correlation coefficient can be adopted. In this case, the correlation information value (correlation coefficient) is the maximum when the correlation represented by the correlation information is the maximum.

  Further, the encoding unit 24 of the transmission device 1 (FIG. 25) can output the moving image data from the filter unit 22 as it is without encoding.

  Next, FIG. 33 illustrates a configuration example of the receiving device 2 of FIG.

The encoded data output from the transmission apparatus 1 in FIG. 25 is encoded low-frame-rate video data with a frame rate of 1 / (4t ₀ ). When the image is supplied and displayed on a display device having the same frame rate as the rate, in addition to the frequency component that should be displayed, a part of the aliasing component shown in FIGS. 28 to 30 is also displayed. As a result, the image quality of the moving image displayed on the display device deteriorates.

Therefore, the receiving apparatus 2 upsamples the low frame rate moving image data with the frame rate of 1 / (4t ₀ ) to the original frame rate 1 / t ₀ , and the resulting frame rate is 1 / t _0. The high-frame-rate moving image data is supplied to the display device 3 (FIG. 24), so that the display device 3 displays the high-quality moving image data (the deterioration of the image quality is not recognized). Here, it is assumed that the display device 3 is configured to display a moving image at a frame rate of 1 / t ₀ .

In the receiving device 2, as described above, in order to display a high-quality moving image on the display device 3, low-frame-rate moving image data with a frame rate of 1 / (4t ₀ ) supplied from the transmitting device 1 is used. Thus, upsampling in the time direction is performed while applying a filter that allows a frequency component in a predetermined passband to pass.

  That is, the decoding unit 50 is supplied with encoded data from the transmission device 1 (FIG. 24) via the recording medium 11 or the transmission medium 12. The decoding unit 50 decodes the encoded data, and supplies the low frame rate moving image data obtained as a result to the buffer unit 51.

  The buffer unit 51 sequentially stores the low frame rate moving image data supplied from the decoding unit 50.

  The filter unit 52 appropriately reads out the moving image data stored in the buffer unit 51, up-samples the moving image data while filtering according to the filter information supplied from the filter generation unit 53 described later, and performs the filtering and up-sampling. The resulting high frame rate video data is output. The high frame rate moving image data output from the filter unit 52 is supplied to the display device 3 (FIG. 24) and displayed.

  The filter generation unit 53 appropriately reads out the moving image data stored in the buffer unit 51, generates filter information that is filter information to be filtered using each portion of the moving image data, and supplies the filter information to the filter unit 52.

  That is, the filter generation unit 53 includes a main component direction acquisition unit 61 and a filter information supply unit 62.

  The principal component direction acquisition unit 61 acquires, for each part of the moving image data read from the buffer unit 51, a principal component direction that is the direction of the principal component on the frequency domain, and filters the information on the principal component direction. The information is supplied to the information supply unit 62.

The filter information supply unit 62 is an area extending in the principal component direction in the frequency domain according to the information of the principal component direction from the principal component direction acquisition unit 61, and has a specific width in the direction of the frequency axis T in the time direction. For example, a region having 2π / (4t ₀ ), that is, a region R _{1001 in} FIG. 20, a region R ₁₁₀₁ in FIG. 21, a region R ₁₂₀₁ in FIG. 22, and a region R _{1301 in} FIG. The pass band is determined, and the pass band is output to the filter unit 52 as filter information.

  Next, processing of the receiving device 2 in FIG. 33 will be described with reference to the flowchart in FIG.

The decoding unit 50 is supplied with encoded data. The decoding unit 50 decodes the encoded data into moving image data with a frame rate of 1 / (4t ₀ ) and a low frame rate, and supplies it to the buffer unit 51. The buffer unit 51 sequentially stores moving image data having a low frame rate of 1 / (4t ₀ ) supplied from the decoding unit 50.

  In step S31, the filter generation unit 53 generates a certain part of the moving image data stored in the buffer unit 51, that is, for example, a block of 16 × 16 pixels as described in step S21 of FIG. Data is read out and input to itself (filter generation unit 53), and the process proceeds to steps S32 to S34 in sequence.

  In steps S32 to S34, the filter generation unit 53 (the principal component direction acquisition unit 61 and the filter information supply unit 62) can recognize a region that can be recognized by humans on the frequency domain for the data input in step S31. A process for obtaining the pass band of the filter and supplying the pass band as filter information to the filter unit 52, that is, a process similar to the above-described "process for obtaining the pass band of necessary information" in FIG. 31 is performed.

  That is, in step S32, the principal component direction acquisition unit 61 of the filter generation unit 53 uses the frequency axes X and Y in the spatial directions x and y and the frequency axis T in the time direction t for the moving image data input in step S31. The principal component direction in the defined frequency domain is acquired and supplied to the filter information supply unit 62, and the process proceeds to step S33.

  In step S32, the principal component direction can be obtained by performing a three-dimensional Fourier transform as described in FIG. 31, or can be obtained by detecting a motion vector as described in FIG. it can. Further, as described above, when the principal component direction (information thereof) is multiplexed in the encoded data from the transmission device 1, the decoding unit 50 separates the principal component direction from the encoded data, In the direction acquisition unit 61, the principal component direction can be acquired by receiving the supply of the principal component direction (information thereof) from the decoding unit 50.

Furthermore, encoded data in the spatial direction x, in the case that contains the y motion vector (u _0, v ₀₎ is the motion vector (u _0, v _0), the frame synchronization 4t of the moving image data included in the encoded data ₀ is added as a component in the time direction t to obtain a three-dimensional motion vector (u ₀ , v ₀ , 4t ₀ ) in the spatial direction x, y and the time direction t, and as shown in FIG. The plane orthogonal to the motion vector (u ₀ , v ₀ , 4t ₀ ) can be the principal component direction.

  That is, in the encoding unit 24 of the transmission apparatus 1 of FIG. 25, when moving image data is encoded by an encoding method such as MPEG that uses at least motion compensation, the encoded data is used for motion compensation. Contains motion vectors. Therefore, the decoding unit 50 extracts the motion vector (information thereof) from the encoded data, and supplies the motion vector included in the encoded data to the principal component direction acquisition unit 61 via the buffer unit 51, thereby the principal component direction. The acquisition unit 61 can obtain (detect) a plane orthogonal to the motion vector as the principal component direction.

  In MPEG, a motion vector does not exist in an I picture block (macroblock), but an I picture motion vector may be estimated from a motion vector of a P or B picture block, for example.

In step S33, the filter information supply unit 62 is a region extending in the principal component direction from the principal component direction acquisition unit 61 from the origin (0, 0) in the frequency domain, and is 2π / (4 × t in the T direction. ₀ ), the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ), and the T direction is-(π / t ₀ ) to + (π / t ₀ ). a range of area, i.e., and a region R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R ₁₃₀₁ in FIG. 23, determined as the pass band of the filter, the flow proceeds to step S34.

  In step S34, the filter information supply unit 62 supplies the filter passband (information representing) obtained in step S33 to the filter unit 52 as filter information, and the process proceeds to step S35.

  In step S35, the filter unit 52 reads the moving image data from the buffer unit 51, and applies the filter of the passband represented by the filter information supplied from the filter generation unit 53 in step S34 to the moving image data. Upsampling is performed to quadruple the number of samples in the direction.

  That is, for example, assuming that the moving image data input in step S31 is a block (data) of a certain frame of moving image data at a low frame rate, that block is the attention block, and the frame of the attention block is noted. Frame.

The filter unit 52 converts the moving image data read from the buffer unit 51 into a passband filter represented by the filter information supplied from the filter generation unit 53, for example, the region R _{1001 in} FIG. 20 or the region R _{1101 in} FIG. The data (pixel value) of each pixel of the block of interest is obtained by filtering with a filter that uses the region R ₁₂₀₁ in FIG. 22 and the region R _{1301 in} FIG. 23 as a pass band.

Further, the filter unit 52 filters the moving image data read from the buffer unit 51 in the passband represented by the filter information supplied from the filter generation unit 53, that is, for example, the region _{R1001 in} FIG. 20 or the region in FIG. By filtering with R ₁₁₀₁ , a region R ₁₂₀₁ in FIG. 22 and a filter having a region R _{1301 in} FIG. 23 as a pass band, the time 4 t ₀ from the frame of interest to the next frame is divided into three equal to the time t ₀ . Data of pixels on the straight line extending in the direction of the three-dimensional motion vector of the block of interest through each pixel of the block of interest is obtained at each time. Note that the three-dimensional motion vector of the block of interest can be obtained by, for example, the principal component direction acquisition unit 61 and supplied to the filter unit 52 together with the filter information via the filter information supply unit 62.

The filter unit 52 performs the above filtering and upsampling to obtain high frame rate moving image data with a frame rate of 1 / t ₀ , and proceeds from step S35 to step S36.

  In step S36, the filter unit 52 outputs the high frame rate moving image data obtained in step S35 to the display device 3 (FIG. 24), and ends the process.

  Note that the processing in steps S31 to S34 is performed for all portions of the moving image data stored in the buffer unit 51.

As described above, regions that can be recognized by humans on the frequency domain, that is, for example, region R _{1001 in} FIG. 20, region R ₁₁₀₁ in FIG. 21, region R ₁₂₀₁ in FIG. 22, region R _{1301 in} FIG. Since the video data is filtered by a filter that uses a passband, and further upsampling is performed, the amount of data is reduced in consideration of the visual effects of the human eye. , Obtain high-quality video data, that is, video data with the same image quality as the original high-frame-rate video data (restore video data that can recognize the same image quality as the original high-frame-rate video data Can be displayed).

That is, FIGS. 35 to 38, respectively, the regions R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R ₁₃₀₁ in FIG. 23, the filter having a pass band to 27 A high frame rate video (frame rate is obtained by performing upsampling four times in the time direction while filtering the data of the low frame rate video (video with a frame rate of 1 / (4t ₀ )) in FIG. 1 / t ₀ (moving data) on the frequency domain.

First, the data of the still portion in the moving image is up-sampled in the filter unit 52 while performing filtering by a filter having the region R _{1001 in} FIG. 20 as a pass band. Accordingly, the moving image data obtained by the filtering and upsampling has frequency components in the region R ₁₀₀₁ in FIG. 20, and is further sampled at intervals of r _{0 with} respect to the spatial directions x and y and in the time direction t. On the other hand, the data is sampled at t ₀ intervals.

Therefore, as shown in FIG. 35, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / t _{0 in the} T direction. . That is, the period in the T direction in which the aliasing component occurs is four times that shown in FIG.

  In FIG. 35, the shaded portion is a portion where moving image data exists.

Next, with respect to data of a portion of the moving image where the subject projected on the moving image moves at a speed of about “(r ₀ / t ₀ ) / 2”, the filter unit 52 uses the area shown in FIG. Upsampling is performed while filtering with a filter having R ₁₁₀₁ as a pass band. Accordingly, the moving image data obtained by the filtering and upsampling has frequency components in the region R ₁₁₀₁ in FIG. 21, and is sampled at intervals of r _{0 with} respect to the spatial direction x, y, and in the time direction t. On the other hand, the data is sampled at t ₀ intervals.

Therefore, as shown in FIG. 36, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / t _{0 in the} T direction. . That is, the period in the T direction in which the aliasing component occurs is four times that shown in FIG.

  In FIG. 36, the shaded portion is a portion where moving image data exists.

Next, for the data of the portion of the moving image where the subject projected on the moving image moves at a speed of about “r ₀ / t ₀ ”, the filter unit 52 passes the region R _{1201 in} FIG. Upsampling is performed while filtering with a band filter. Therefore, the moving image data obtained by the filtering and up-sampling has frequency components in the region R ₁₂₀₁ in FIG. 22, and is further sampled at intervals of r _{0 with} respect to the spatial directions x and y, and in the time direction t. On the other hand, the data is sampled at t ₀ intervals.

Therefore, as shown in FIG. 37, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / t _{0 in the} T direction. . That is, the period in the T direction in which the aliasing component occurs is four times that shown in FIG.

  In FIG. 37, the shaded portion is a portion where moving image data exists.

Next, for the data of the portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”, the filter unit 52 passes the region R _{1301 in} FIG. Upsampling is performed while filtering with a band filter. Accordingly, the moving image data obtained by the filtering and upsampling has frequency components in the region R ₁₃₀₁ in FIG. 23, and is further sampled at intervals of r _{0 with} respect to the spatial directions x and y, and in the time direction t. On the other hand, the data is sampled at t ₀ intervals.

Therefore, as shown in FIG. 38, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / t _{0 in the} T direction. . That is, the period in the T direction in which the aliasing component occurs is four times that shown in FIG.

  In FIG. 38, the shaded portion is a portion where moving image data exists.

As described in FIG. 2, humans can recognize that the X and Y directions are − (π / r ₀ ) to + (π / r ₀ ) and the T direction is − (π / t ₀ ) to + (π / t ₀ ) range R ₂₀₁ , and in FIG. 35 to FIG. 38, there is no aliasing component in this region R ₂₀₁ .

Therefore, when moving image data with a high frame rate of 1 / t ₀ obtained by filtering and upsampling in the filter unit 52 is displayed on the display device 3, The same image quality as that of the moving image (the moving image supplied to the transmission device 1) can be recognized.

  As described above, the encoding unit 24 of the transmission device 1 in FIG. 25 can multiplex the filter information of the filter used in the filter unit 22 into the encoded data, but the filter information is multiplexed into the encoded data. If it is configured, the receiving device 2 (FIG. 33) can be configured without the filter generation unit 53. In this case, the receiving device 2 may be configured such that the decoding unit 50 separates the filter information from the encoded data and supplies it to the filter unit 52.

In the above case, 1 / (4t ₀ ) low frame rate moving image data obtained by processing the high frame rate moving image data of 1 / t ₀ in the transmitting device 1 of FIG. And the receiving device 2 converts the low-frame-rate moving image data into high-frame-rate moving image data of 1 / t ₀ , so that the display device 3 has high image quality (input to the transmitting device 1). Although a moving image having the same image quality as the moving image is displayed, even if moving image data with a low frame rate of 1 / (4t ₀ ) is originally input to the receiving device 2, the display device 3 has a certain amount of high quality. It is possible to display a moving image of image quality.

That is, in the moving image, in a portion where a (substantially) stationary subject is projected, there is almost no frequency component in the time direction, and thus the region obtained by removing the region R ₁₀₀₁ from the region R ₃₀₁ shown in FIG. There are almost no frequency components in.

In the moving image, in the part where the subject projected there is moving at a speed of “(r ₀ / t ₀ ) / 2”, the direction of the movement (spatial direction x, y and time direction t three-dimensional Since there is almost no frequency component other than the frequency component in the direction perpendicular to the direction of the movement of (1), there is almost no frequency component in the region excluding the region R ₁₁₀₁ from the region R ₄₀₁ shown in FIG.

Further, in the moving image, there is almost no frequency component other than the frequency component in the direction perpendicular to the direction of the movement in the portion where the object projected there is moving at a speed of about “r ₀ / t ₀ ”. There are almost no frequency components in the region excluding the region R ₁₂₀₁ from the region R ₈₀₁ shown in FIG.

Further, in the moving image, in the portion where the subject projected there is moving at a speed of about “2r ₀ / t ₀ ”, there is almost no frequency component in the direction perpendicular to the direction of the movement. There are almost no frequency components in the region excluding the region R ₁₃₀₁ from the region R ₉₀₁ shown in FIG.

However, in actuality, in the moving image, the projected image of the subject changes in a complicated manner with time, so even a portion that is stationary in the moving image is a region from the region R ₃₀₁ shown in FIG. There is some data (frequency components) in the area excluding _R1001 .

Similarly, in the moving part of the moving image at a speed of “(r ₀ / t ₀ ) / 2”, there is some data in the area excluding the area R ₁₁₀₁ from the area R ₄₀₁ shown in FIG. Further, in the moving part of the moving image at the speed “r ₀ / t ₀ ”, there is some data in the area excluding the area R ₁₂₀₁ from the area R ₈₀₁ shown in FIG. Furthermore, in the moving part of the moving image at a speed of about “2r ₀ / t ₀ ”, there is some data in the area excluding the area R ₁₃₀₁ from the area R ₉₀₁ shown in FIG.

In order to delete data existing outside the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , and R ₁₃₀₁ , the filter unit 22 of the transmission device 1 (FIG. 25) uses the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , R Filtering is performed by a filter having a pass band of ₁₃₀₁ .

Therefore, when the low frame rate moving image data input to the receiving device 2 is not obtained as a result of the processing of the transmitting device 1, some data outside the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , R ₁₃₀₁ There can be data. Then, when data exists outside the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , and R ₁₃₀₁ , moving image data obtained by filtering and upsampling in the filter unit 52 of the receiving device 2 (FIG. 33) is shown in FIG. The aliasing component is mixed in the region R ₂₀₁ that can be recognized by the human being, and the image quality of the moving image deteriorates due to the aliasing component.

However, the data existing outside the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , and R ₁₃₀₁ is very small, and image quality degradation caused by the aliasing component due to such a small amount of data is also small. Therefore, originally when moving image data with a low frame rate of 1 / (4t ₀ ) is originally input to the receiving device 2, it is compared with when inputting moving image data with a low frame rate obtained by the transmitting device 1. Although there is some degradation in image quality, it is still possible to display movies with improved image quality compared to the case of displaying 1 / (4t ₀ ) low frame rate movie data as it is. it can.

Note that when the 1 / (4t ₀ ) low-frame-rate moving image data input to the receiving device 2 is captured by an imaging device that employs a so-called electronic shutter, the receiving device 2 (FIG. 33). In the filter information supply unit 62, a region in which the width in the T direction is limited corresponding to the exposure time at the time of capturing the moving image data at the low frame rate can be determined as the pass band of the filter.

That is, 1 / the (4t ₀₎ exposure time for the imaging of each frame of the moving image data of a low frame rate, with greater than t _0, when it is t ₀ 'is smaller than 4t ₀ is on each frame The projected image of the subject is equal to a value (light reception amount) obtained by integrating light from the subject for a time t ₀ ′. Therefore, when the subject is moving, blur (motion blur) corresponding to time t ₀ ′ is generated in the image of each frame. When t ₀ is about 1/240 seconds, for example, t ₀ ′ is about 1/120 seconds, for example.

The integration of only time t ₀ ′ is equivalent to performing filtering by a low-pass filter having a region R ₃₉₀₁ having a width of 2π / t ₀ ′ in the frequency domain T direction as shown in FIG. Therefore, it can be said that the data of each frame of the moving image captured with the exposure time t ₀ ′ by the electronic shutter is the data that has passed through the low-pass filter having the region R ₃₉₀₁ as the pass band.

Note that the region R ₃₉₀₁ has a width of 2π / t ₀ ′ in the range of −π / t ₀ ′ to π / t ₀ ′ in the T direction, and a human recognizes the X and Y directions. This is a region having a width of 2π / r _{0 in} the range of −π / r _{0 to} π / r ₀ . As described above, for example, when t ₀ is 1/240 seconds and t ₀ ′ is 1/120 seconds, and therefore 2t ₀ = t ₀ ′, the region R ₃₉₀₁ has the T direction It has a width of π / t _{0 in} the range of −π / (2t ₀ ) to π / (2t ₀ ).

As described above, the data of each frame of the imaged moving image exposure time as t ₀ ', since the data that has passed through the low-pass filter having a pass band region R _3901, in the frequency domain, in addition to regions R ₃₉₀₁ There is no data.

Therefore, for example, even the data in the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R ₁₃₀₁ in FIG. Since the data outside the area R ₃₉₀₁ is not the original data of the moving image such as noise, it is desirable to delete it as unnecessary data.

Therefore, in the filter information supply unit 62 of the receiving device 2 (FIG. 33), the T direction of the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R ₁₃₀₁ in FIG. A region limited by 39 regions R ₃₉₀₁ can be determined as the passband of the filter.

For example, as described above, assuming that 2t ₀ = t ₀ ′, the region R _{1001 in} FIG. 20 is included in the region R ₃₉₀₁ in FIG. For the stationary part, the region R _{1001 in} FIG. 20 can be determined as it is as the pass band of the filter.

In addition, as shown in FIG. 40, the region R _{1101 in} FIG. 21 has X, Y in the T direction in the region R ₁₁₀₁ in the range of −π / (2t ₀ ) to π / (2t ₀ ). direction, in the region R ₄₀₀₁ in the range of - [pi] / r ₀ to [pi / r _0, since overlapping the region R ₃₉₀₁ in FIG. 39, the filter information supply section 62, the subject being projected video speed "( For a portion moving at about “r ₀ / t ₀ ) / 2”, the region R _{4001 in} FIG. 40 can be determined as the passband of the filter.

Furthermore, as shown in FIG. 41, the region R _{1201 in} FIG. 22 has X, Y in the T direction in the region R ₁₂₀₁ in the range of −π / (2t ₀ ) to π / (2t ₀ ). Since the direction R overlaps with the region R _{3901 in} FIG. 39 in the region R ₄₁₀₁ in the range of −π / r _{0 to} π / r ₀ , the filter information supply unit 62 uses the velocity “r For a portion moving at about “ ₀ / t ₀ ”, the region R _{4101 in} FIG. 41 can be determined as the passband of the filter.

In addition, as shown in FIG. 42, the region R _{1301 in} FIG. 23 has X, Y in the T direction in the region R ₁₃₀₁ in the range of −π / (2t ₀ ) to π / (2t ₀ ). direction, in the region R ₄₂₀₁ in the range of - [pi] / r ₀ to [pi / r _0, since overlapping the region R ₃₉₀₁ in FIG. 39, the filter information supply unit 62, the object being projected video speed "2r For the portion moving at about “ ₀ / t ₀ ”, the region R _{4201 in} FIG. 42 can be determined as the passband of the filter.

In the filter information supply unit 62 of the receiving apparatus 2 (FIG. 33), the pass band of the filter is determined as described above, and filtering is performed by the filter unit 52, so that the original moving image such as noise outside the region R ₃₉₀₁ is obtained. Since unnecessary data other than the above data is removed, moving image data with improved S / N (Signal to Noise ratio) and the like can be obtained.

Here, in the receiving apparatus 2, when the pass band of the filter is limited by the region R ₃₉₀₁ in FIG. 39, the exposure time t ₀ ′ at the time of capturing the low frame rate moving image data input to the receiving apparatus 2 is Although necessary, the exposure time t ₀ ′ can be input to the receiving device 2 by multiplexing the moving image data input to the receiving device 2, for example.

  When the data input to the receiving device 2 is not encoded, the decoding unit 50 supplies the data to the buffer unit 51 without decoding the data.

By the way, for example, the principal component direction acquisition unit 31 of the transmission device 1 (FIG. 25) detects the motion vector (u ₀ , v ₀ ) of the block of interest to the next frame of the frame of interest, as described above. From the motion vector (u ₀ , v ₀ ), the principal component direction in the frequency domain of the block of interest can be obtained. That is, in the principal component direction acquisition unit 31, a three-dimensional motion vector (u ₀ , v ₀ , t ₀ ) obtained by adding the component t ₀ in the time direction t to the motion vector (u ₀ , v ₀ ) of the block of interest An orthogonal plane can be obtained as the principal component direction in the frequency domain of the block of interest.

In this case, for example, a vector obtained by multiplying the motion vector (u ₀ , v ₀ ) of the block of interest to the next frame of the frame of interest by, for example, -2, -1, 2 or 3 times, If it matches the motion vector to the previous frame, previous frame, next next frame, next next next frame of the frame of interest, it can be obtained from the motion vector (u ₀ , v ₀ ) The principal component direction of the block of interest is accurate.

However, a vector obtained by multiplying the motion vector (u ₀ , v ₀ ) by, for example, −2 times, −1 times, 2 times, or 3 times is the frame before the attention frame or the previous frame of the attention block. If the motion vector is shifted from the motion vector to the next next frame, the next next frame, the principal component direction of the target block obtained from the motion vector (u ₀ , v ₀ ) is It contains errors and is inaccurate.

  That is, FIG. 43 shows high frame rate moving image data supplied to the transmission apparatus 1.

  In FIG. 43, the moving image data is shown with the horizontal axis as the spatial direction x. In addition, a moving subject is projected on the moving image, and the moving subject moves only in the spatial direction x (does not move in the spatial direction y). In FIG. 43, the moving subject moves from left to right as time passes.

In FIG. 43, in order from the top, time (frames) t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t + 3t ₀ , t + Image data in each of 4t ₀ is shown.

If the image data at time t is used as a reference, the image data at time t-4t ₀ is only K ₄ , the image data at time t-3t ₀ is only K ₃ , and the time t Image data at -2t ₀ is K ₂ only, image data at time tt ₀ is K ₁ only, image data at time t + t ₀ is H ₁ only, image data at time t + 2t ₀ is H ₂ only, time t The image data at + 3t ₀ is shifted by H _{3 and} the image data at time t + 4t ₀ is shifted by H ₄ in the spatial direction x.

Therefore, when a certain block R ₄₃₀₀ in the image data at time t and the subject block, the same image data and the block of interest R ₄₃₀₀ (image data corresponding to the target block R _4300), the image data at time t-4t ₀ is , from the target block R ₄₃₀₀ present in a region R ₄₃₁₄ of the position shifted by K ₄ in the spatial direction x, the image data at time t-3t _0, from the target block R ₄₃₀₀ of position shifted in the spatial direction x by K ₃ Exists in region R ₄₃₁₃ . Similarly, the same image data as the _target block R ₄₃₀₀ is the image data at time tt _{0 in} the region R ₄₃₁₂ at a position shifted by K _{2 in the} spatial direction x from the _target block R _{4300 in} the image data at time t−2t _0. In the data, the region R ₄₃₁₁ at a position shifted by K _{1 in the} spatial direction x from the _target block R ₄₃₀₀ , and at the time t + t ₀ image data at the position shifted by H _{1 in the} spatial direction x from the _target block R ₄₃₀₀ . In the region R ₄₃₀₁ , in the image data at time t + 2t ₀ , the image data at time t + 3t ₀ is shifted from the _target block R ₄₃₀₀ to R ₄₃₀₂ at a position shifted by H _{2 in the} spatial direction x from the _target block R _4300. In the region R ₄₃₀₃ at a position shifted by H _{3 in the} spatial direction x, the image data at time t + 4t ₀ exists in the region R ₄₃₀₄ at a position shifted by H _{4 in the} spatial direction x from the target block R ₄₃₀₀ , respectively. .

In FIG. 43, the magnitude of the movement of the moving subject is not constant. Therefore, the rate of increase from K ₄ to K ₃ , the rate of increase from K ₃ to K ₂ , the rate of increase from K ₂ to K ₁ , The rate of increase from K ₁ to 0, the rate of increase from 0 to H ₁ , the rate of increase from H ₁ to H ₂ , the rate of increase from H ₂ to H ₃ and the rate of increase from H ₃ to H ₄ are also constant. It is not.

As shown in FIG. 43, when the subject is moving, the motion vector (u ₀ , v ₀ ) of the block of interest R ₄₃₀₀ is obtained as shown in FIG. 44 as a vector (H ₁ , 0). Therefore, the principal component direction acquisition unit 31 of the transmission device 1 (FIG. 25) obtains the principal component direction of the _target block R _{4300 from} the motion vector (H ₁ , 0) of the target block R ₄₃₀₀ .

Then, in the filter information supply unit 32 of the transmission apparatus 1, for example, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R _{1301 in} FIG. Is determined as the pass band of the filter that performs filtering in the filter unit 22.

As the pass band of the filter, for example, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R _{1301 in} FIG. / (4t ₀ ), therefore, filtering by such a passband filter, image data of about 4 frames before and after the frame of the block of interest R ₄₃₀₀ (frame of interest), that is, for example, time t-4t ₀ , T-3t ₀ , t-2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t + 3t ₀ , t + 4t ₀ range of image data or time t-3t ₀ , T−2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t + 3t _{0 and} other operations using image data in a range of 7 frames (performed by the filter unit 22 of the transmission apparatus 1) By calculation), the image data of the _target block R ₄₃₀₀ as the filtering result is obtained.

That is, the block of interest position with R ₄₃₀₀ of pixels (x, y), the pixel value as a filtering result, for example, the position in the image data at time _{t + t 0 (x, y} ) a motion vector (H ₁ from Pixel value near the position shifted by 0), position shifted by the vector (2H ₁ , 0) twice the motion vector (H ₁ , 0) from the position (x, y) in the image data at time t + 2t ₀ Near pixel value, pixel value near the position shifted by the vector (3H ₁ , 0) three times the motion vector (H ₁ , 0) from the position (x, y) in the image data at time t + 3t ₀ A pixel value in the vicinity of a position shifted by a vector (4H ₁ , 0) four times the motion vector (H ₁ , 0) from the position (x, y) in the image data at t + 4t _{0, in} the image data at time tt ₀ position (x, y) -1 times the vector (-H _1, 0) shifted by the pixel value of the vicinity of the position of the motion vector from the (H _1, 0), the position in the image data at time t-2t ₀ (x, -2 times the motion vector (H _1, 0) from y) Vector (-2H _1, 0) only the pixel values of the shifted around position, the position in the image data at time _{t-3t 0 (x, y} ) a motion vector (H _1, 0) from -3 fold vector (-3H _1, 0) -4 times the vector of pixel values around a position shifted by a position in the image data at time _{t-4t 0 (x, y} ) from the motion vector (H _1, 0) (-4H _1, 0) It is obtained by using pixel values in the vicinity of the position shifted by a certain amount.

Here, the motion vector (H _1, 0), as shown in FIG. 44, from the target block R _4300, the image data at time t + t _0, corresponding to the target block R ₄₃₀₀ and (attention block R ₄₃₀₀ Since this is a vector to the position of the region R ₄₃₀₁ (where the same image data exists), the position on the image data at time t + t ₀ shifted by the motion vector (H ₁ , 0) from the block of interest R ₄₃₀₀ It matches the position of the region R ₄₃₀₁ corresponding to the block R ₄₃₀₀ .

However, since the vector (2H ₁ , 0) is a vector obtained by doubling the motion vector (H ₁ , 0) as shown in FIG. 45, the vector (2H ₁ , 0) is shifted from the _target block R ₄₃₀₀ by the vector (2H ₁ , 0). The position on the image data at time t + 2t ₀ does not necessarily match the position of the region R ₄₃₀₂ corresponding to the block of interest R ₄₃₀₀ .

Further, as shown in FIG. 46, the vector (3H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by three, so that it is shifted from the _target block R ₄₃₀₀ by the vector (3H ₁ , 0). The position on the image data at time t + 3t ₀ does not always match the position of the region R ₄₃₀₃ corresponding to the block of interest R ₄₃₀₀ .

Furthermore, since the vector (4H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by 4 as shown in FIG. 47, the vector (4H ₁ , 0) is shifted from the _target block R ₄₃₀₀ by the vector (4H ₁ , 0). The position on the image data at time t + 4t ₀ does not necessarily match the position of the region R ₄₃₀₄ corresponding to the target block R ₄₃₀₀ .

Further, as shown in FIG. 48, the vector (−H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by −1. Therefore, the vector (−H ₁ , 0) from the _target block R ₄₃₀₀ The position on the image data at time tt _{0 that} is shifted by a certain amount does not necessarily match the position of the region R ₄₃₁₁ corresponding to the block of interest R ₄₃₀₀ .

Furthermore, since the vector (-2H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by -2 as shown in FIG. 49, the vector (-2H ₁ , 0) from the target block R ₄₃₀₀ The position on the image data at time t−2t _{0 that} is shifted by a certain amount does not always match the position of the region R ₄₃₁₂ corresponding to the block of interest R ₄₃₀₀ .

Further, since the vector (−3H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by −3 as shown in FIG. 50, the vector (−3H ₁ , 0) from the target block R ₄₃₀₀ The position on the image data at time t-3t _{0 that} is shifted by a certain amount does not necessarily match the position of the region R ₄₃₁₃ corresponding to the block of interest R ₄₃₀₀ .

Furthermore, since the vector (−4H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by −4 as shown in FIG. 51, the vector (−4H ₁ , 0) from the target block R ₄₃₀₀ The position on the image data at time t-4t _{0 that} is shifted by a certain amount does not necessarily match the position of the region R ₄₃₁₄ corresponding to the block of interest R ₄₃₀₀ .

For this reason, in filtering focused on the _target block R ₄₃₀₀ , the image data at times t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + 2t ₀ , t + 3t ₀ , t + 4t ₀ , The pixel values of the regions shifted from the regions R ₄₃₁₄ , R ₄₃₁₃ , R ₄₃₁₂ , R ₄₃₁₁ , R ₄₃₀₂ , R ₄₃₀₃ , R ₄₃₀₄ corresponding to the target block R ₄₃₀₀ are used, and as a result, the true main Data of only frequency components in a region on the frequency domain extending in the component direction (for example, region R _{1001 in} FIG. 20, region R ₁₁₀₁ in FIG. 21, region R ₁₂₀₁ in FIG. 22, region R _{1301 in} FIG. 23), There may be cases where it is difficult to obtain accurately.

  This is because the motion vector from the target block to the next frame can be determined by using the motion vector from the target block to the next frame. Therefore, when the principal component direction is calculated using the motion vector from the block of interest to the next frame even though the subject is not moving at a constant speed, an incorrect principal component direction is obtained. This is due to the fact that

  Therefore, in the principal component direction acquisition unit 31 of the transmission device 1 (FIG. 25), as described above, not only the correlation information indicating the correlation between the block of interest and the next frame, but also the block of interest and a plurality of frames before and after the block. By using a plurality of pieces of correlation information representing the correlation between the target block and each of the plurality of frames, a so-called average motion vector (hereinafter referred to as an average motion vector as appropriate) is obtained. The principal component direction with high accuracy can be obtained for the target block.

  FIG. 52 shows a configuration example of the principal component direction acquisition unit 31 in the case where the principal component direction acquisition unit 31 in FIG. 25 obtains the principal component direction from the average motion vector.

The buffer unit 101 is supplied with the 1 / t ₀ high frame rate moving image data read from the buffer unit 21 (FIG. 25), and the buffer unit 101 temporarily stores the moving image data.

  The block extraction unit 102 divides the moving image data stored in the buffer unit 101 into, for example, the above-described 16 × 16 pixel blocks, sequentially selects the target block, and supplies the selected block to the correlation calculation unit 103.

  The correlation calculation unit 103 reads, from the buffer unit 101, a plurality of frames (data) before and after the frame of the block of interest (the frame of interest) supplied from the block extraction unit 102. The correlation information representing the correlation with each of the frames is calculated and supplied to the scaling synthesis unit 104.

  Here, as the correlation information, for example, as in the case described in step S22 of FIG. 32, each pixel of the target block at a position where the target block is shifted by u and v in the spatial directions x and y, respectively. And the sum of absolute difference values of pixel values from pixels in other frames at the same position as the pixel. In this case, as described above, the position u, v in the spatial direction where the “value” of the correlation information is the minimum is the position in the spatial direction on the other frame where the correlation represented by the correlation information is the highest.

  Note that in the correlation calculation unit 103, the range of the frame for which the correlation information with the target block is calculated can be a range corresponding to the frame range used for filtering in the filter unit 22 (FIG. 25).

That is, as described with reference to FIG. 43, when the image data at time t is used as a reference (when time t is the frame of interest), the filtering in the filter unit 22 performs time t-4t ₀ , t-3t ₀ , t− 9 frames of image data of 2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t + 3t ₀ , t + 4t ₀ , time t-3t ₀ , t-2t ₀ , tt ₀ , t , T + t ₀ , t + 2t ₀ , t + 3t ₀ , and the like are used. In the correlation calculation unit 103, the frame for calculating the correlation information is image data in a range almost similar to the image data of 9 frames or 7 frames, that is, for example, time t-4t ₀ , t-3t ₀ , t-2t. _0, tt _0, t, can be _{t + t 0, t + 2t} 0, t + 3t 0, t + 4t 0 each image data.

Note that since the image data at time t is a frame of the block of interest (frame of interest), there is no need to calculate correlation information for that time t. That is, the correlation calculation unit 103 calculates the correlation information with the block of interest at times t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t The image data is + 3t ₀ and t + 4t ₀ respectively.

  Here, the frame for which the correlation information with the block of interest is calculated is hereinafter referred to as a correlation calculation target frame as appropriate.

  The scaling synthesis unit 104 scales the correlation information obtained from each of the plurality of correlation calculation target frames supplied from the correlation calculation unit 103 in the spatial directions x and y as described later, and further, The correlation information is synthesized to obtain synthesized correlation information. Then, the scaling combining unit 104 supplies the combined correlation information to the minimum value detecting unit 105.

The minimum value detecting unit 105 detects a maximum correlation position that is a position in the spatial direction where the correlation represented by the combined correlation information from the scaling combining unit 104 is maximum, and calculates a vector to the maximum correlation position as an average motion of the target block. Ask as a vector. Then, the minimum value detection unit 105 adds the frame period t ₀ of the moving image data read from the buffer unit 21 (FIG. 25) to the average motion vector as a component in the time direction t. A direction orthogonal to the direction is detected and output as the principal component direction.

Next, FIG. 53 shows the block of interest R ₄₃₀₀ shown in FIG. 43 and the times t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t. Correlation information with each image data (frame) of + 3t ₀ and t + 4t ₀ is shown.

Note that the correlation information shown in FIG. 53 is obtained by the correlation calculation unit 103 in FIG. However, as described above, the correlation information between the block of interest R ₄₃₀₀ and the image data at time t, which is the frame of interest, does not need to be calculated.

  In FIG. 53, as described above, the sum of the absolute differences of the pixel values is used as the correlation information. Therefore, the correlation information is a downward convex function, and the “value” of the correlation information is The position in the spatial direction that minimizes is the position in the spatial direction where the correlation represented by the correlation information is the highest. A vector from the target block to that position represents a motion vector.

As described in FIG. 43, the same image data and the block of interest R ₄₃₀₀ is at time t-4t _0, present in the block of interest R ₄₃₀₀ position of the region R ₄₃₁₄ shifted by K ₄ in the spatial direction x from the time At t−3t ₀ , it exists in a region R ₄₃₁₃ at a position shifted from the target block R ₄₃₀₀ by K _{3 in the} spatial direction x. Similarly, the same image data as the _target block R ₄₃₀₀ is _displayed at the time t-2t ₀ in the region R ₄₃₁₂ at a position shifted by K ₂ from the _target block R _{4300 in the} spatial direction x, and at the time tt ₀ , the _target block R _At time t + t ₀ at a region R ₄₃₁₁ at a position shifted by K _{1 in the} spatial direction x from ₄₃₀₀ , at time t + 2t at a region R ₄₃₀₁ at a position shifted by H _{1 in the} spatial direction x from the _target block R _{4300 At 0} , it moves to R ₄₃₀₂ at a position shifted by H _{2 in the} spatial direction x from the _target block R ₄₃₀₀ , and at time t + 3t ₀ , it _enters a region R ₄₃₀₃ at a position shifted by H ₃ from the _target block R _{4300 in the} spatial direction x. At time t + 4t ₀ , each exists in a region R ₄₃₀₄ at a position displaced from the target block R ₄₃₀₀ by H _{4 in the} spatial direction x.

Accordingly, the correlation information at time t-4t ₀ is minimum at position x = K ₄ , and the correlation information at time t-3t ₀ is minimum at position x = K ₃ . Similarly, the correlation information at time t-2t ₀ is at position x = K ₂ , the correlation information at time tt ₀ is at x = K ₁ , and the correlation information at time t + t ₀ is at position x = H ₁ . The correlation information at time t + 2t ₀ is at position x = H ₂ , the correlation information at time t + 3t ₀ is at position x = H ₃ , and the correlation information at time t + 4t ₀ is at position x = H ₄ In each, it becomes the minimum.

In the principal component direction acquisition unit 31 in FIG. 52, the motion of the _target block R ₄₃₀₀ is approximated to be constant at the time from time t-4t _{0 to} t + 4t ₀ , and the vector representing the constant motion is the average motion. Required as a vector.

Therefore, the scaling synthesis unit 104 of the principal component direction acquisition unit 31 performs the times t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t ₀ , t Correlation information at each of + 4t ₀ is scaled in the spatial directions x and y, and the correlation information after the scaling is combined to obtain combined correlation information.

That is, when it is approximated that the motion of the _target block R ₄₃₀₀ is constant at the time from time t-4t _{0 to} time t + 4t ₀ , for example, when the correlation information at time t + t ₀ is used as a reference, time t-4t The correlation information at ₀ has a difference of −4 times with respect to the position (spatial direction x, y) with respect to the correlation information at time t + t ₀ . The correlation information at time t-3t ₀ is shifted by -3 times with respect to the correlation information at time t + t ₀ . Similarly, the correlation information at time t-2t ₀ is shifted by -2 times with respect to the position, and the correlation information at time tt ₀ is shifted by -1 times with respect to the correlation information at time t + t ₀ . However, the correlation information at time t + 2t ₀ is twice as much as the position, the correlation information at time t + 3t ₀ is three times as much as the position, and the correlation information at time t + 4t ₀ is about the position. There is a 4x shift.

Therefore, the scaling synthesis unit 104 performs scaling by multiplying the scale of the position of the correlation information at time t-4t ₀ by −1/4 (reversing and reducing). Similarly, the scaling composition unit 104 performs scaling to increase the scale of the position of the correlation information at time t-3t ₀ by −1/3, and scales the position of the position of the correlation information at time t−2t ₀ by −1/2. The scaling of the correlation information position at time tt ₀ is multiplied by -1 (reversed), and the scaling of the correlation information position at time t + 2t ₀ is halved (reduced). ) Scaling is performed so that the scale of the position of the correlation information at time t + 3t ₀ is ３ times, and the scaling is performed to increase the scale of the position of the correlation information at time t + 4t ₀ to ¼. .

For the correlation information at time t + t ₀ , the scaling composition unit 104 does not perform any scaling, but it may be seen that the fact that the scaling is not performed means that the position is scaled to 1 time. it can.

Here, FIG. 54 shows time t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + 2t ₀ , t + 3t ₀ , t + based on the correlation information at time t + t ₀ . The correlation information after scaling when the correlation information at 4t ₀ is scaled is shown.

Scaling, correlation information at time t-4t ₀ is becomes minimum at the position x = -K _4/4, the correlation information at time t-3t ₀ is at the minimum position x = -K _3/3. Similarly, the correlation information at time t-2t _0, at the position x = -K _2/2, correlation information at time tt _0, at x = -K _1, the correlation information at time t + 2t _0, the position x in = H _2/2, correlation information at time t + 3t _0, at the position x = H _3/3, correlation information at time t + 4t _0, at position x = H _4/4, respectively become minimum.

Since scaling is performed with reference to correlation information at time t + t _0, the correlation information at time t + t ₀ is minimum at position x = H ₁ before and after scaling.

In the principal component direction acquisition unit 31 in FIG. 52, the scaling synthesis unit 104 performs the time t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t _0. , T + 4t ₀ , the correlation information after scaling is synthesized to obtain synthesized correlation information.

  FIG. 55 shows combined correlation information obtained by combining the correlation information of FIG.

The combined correlation information of FIG. 55 is the time t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t ₀ , t + 4t shown in FIG. It can be obtained by performing a synthesis of adding the correlation information after scaling in each of ₀ .

In the minimum value detection unit 105 of the principal component direction acquisition unit 31 in FIG. 52, the position x = L ₁ ((x, y) = (L ₁ , 0)) at which the composite correlation information in FIG. It is obtained as a vector (L ₁ , 0).

  Note that the combined correlation information can be obtained by adding the weighted addition in addition to simply adding the correlation information after scaling. In other words, the combined correlation information can be obtained by adding to the correlation information the correlation information closer to the time t of the block of interest with a greater weight. In this case, an average motion vector in which an image at a time close to the time t of the block of interest is regarded as important is obtained.

The minimum value detection unit 105 obtains the principal component direction of the block of interest from the average motion vector (L ₁ , 0) obtained as described above. Therefore, it is possible to obtain the principal component direction with relatively high accuracy for the block of interest.

That is, according to the principal component direction (that is, a plane perpendicular to (L ₁ , 0, t ₀ )) obtained from the average motion vector (L ₁ , 0) as described above, the filter unit 22 in FIG. For example, the pixel value as a filtering result of a pixel at a certain position (x, y) of the _target block R ₄₃₀₀ is a motion vector (L ₁ , 0) from the position (x, y) in the image data at time t + t ₀ The pixel value near the position shifted by a certain amount, the position near the position shifted by a vector (2L ₁ , 0) twice the motion vector (L ₁ , 0) from the position (x, y) in the image data at time t + 2t ₀ Pixel value, pixel value near time t + 3t ₀ position in image data at position (x, y), pixel value near position shifted by vector (3L ₁ , 0) three times motion vector (L ₁ , 0), time t + A pixel value in the vicinity of a position shifted by a vector (4L ₁ , 0) that is four times the motion vector (L ₁ , 0) from the position (x, y) in the image data at 4t ₀ , and a position in the image data at time tt ₀ ( x, y) Torr (L _1, 0) -1 times the vector (-L _1, 0) pixel values around a position shifted by a position in the image data at time _{t-2t 0 (x, y} ) from the motion vector (L ₁ , 0), the pixel value in the vicinity of the position shifted by -2 times the vector (−2L ₁ , 0), the position of the motion vector (L ₁ , 0) from the position (x, y) in the image data at time t-3t ₀ -3 times the pixel value in the vicinity of the position shifted by the vector (-3L ₁ , 0), from the position (x, y) in the image data at time t-4t ₀ to -4 times the motion vector (L ₁ , 0) Pixel values in the vicinity of the position shifted by the vector (−4L ₁ , 0) are respectively obtained.

Here, FIG. 56 shows the position in the image data at time t + t ₀ that is shifted from the position (x, y) of the _target block R _{4300 by} the motion vector (L ₁ , 0). Similarly, FIG. 57 to FIG. 63 show that at time t + 2t ₀ shifted from the position (x, y) of the _target block R ₄₃₀₀ by a vector (2L ₁ , 0) that is twice the motion vector (L ₁ , 0). Position in the image data at time t + 3t ₀ where the position in the image data is shifted from the position (x, y) of the _target block R ₄₃₀₀ by a vector (3L ₁ , 0) that is three times the motion vector (L ₁ , 0) Is shifted from the position (x, y) of the _target block R ₄₃₀₀ by a vector (4L ₁ , 0) four times the motion vector (L ₁ , 0) to the position in the image data at time t + 4t ₀ The position in the image data at time tt _{0 that} is shifted from the position (x, y) of R ₄₃₀₀ by a vector (−L ₁ , 0) that is −1 times the motion vector (L ₁ , 0) is the position of the target block R ₄₃₀₀ . (x, y) position in the -2 times the vector (-2L _1, 0) image data at time t-2t ₀ shifted by the motion vector (L _1, 0) from the position of the target block R ₄₃₀₀ (x , -3 times the vector of the motion vector from y) (L _1, 0) Le (-3L _1, 0) position in the image data at time t-3t ₀ shifted by -4 times the vector of position (x, y) from the motion vector of the target block _{R 4300 (L 1, 0)} ( The positions in the image data at time t-4t ₀ shifted by −4L ₁ , 0) are shown respectively.

56 to 63 and FIG. 44 to 51 described above are compared with each other, and in the case of FIGS. 56 to 63, the attention block R as a whole is more than the case of FIGS. ₄₃₀₀ using the pixel values of the position close to the corresponding region _{R 4301, R 4302, R 4303} , R 4304, R 4311, R 4312, R 4313, R 4314 , a is understood that the filtering of the block of interest R ₄₃₀₀ is performed .

  Next, processing performed by the principal component direction acquisition unit 31 in FIG. 52 will be described with reference to the flowcharts in FIGS.

  Note that the processing according to the flowcharts of FIGS. 64 to 67 corresponds to the processing of step S11 of FIG.

  52, the moving image data read from the buffer unit 21 (FIG. 25) is supplied to the buffer unit 101, and the buffer unit 101 temporarily stores the moving image data.

  64, the block extraction unit 102 divides the moving image data stored in the buffer unit 101 into blocks of 16 × 16 pixels, for example, as in step S21 of FIG. To the unit 103. In the subsequent processing, the blocks obtained by the block extraction unit 102 are sequentially performed as the target block.

Here, the position of the block of interest (for example, the position of the upper left pixel of the block of interest) is represented as (x ₀ , y ₀ ). The frame of the block of interest (frame of interest) is assumed to be a frame at time t.

After the processing in step S101, the process proceeds to step S102 in FIG. 65, in which the correlation calculation unit 103 reads out the image data at time t + t ₀ that is the image data of the frame next to the frame of interest from the buffer unit 101, and If, with the image data at time t + t _0, the function E ₁ representing the sum of the absolute differences of each other corresponding pixel (u _1, v _1), the image data at time t + t ₀ to correspond to the block of interest (X ₀ + u ₁ , y ₀ + v ₁ ) while changing the position (determined while changing (u ₁ , v ₁ )), and the function E ₁ (u ₁ , v ₁ ) is obtained at time t + t The correlation information at ₀ is supplied to the scaling synthesis unit 104, and the process proceeds to step S103.

In step S103, the correlation calculation unit 103 reads out the image data at time t + 2t ₀ that is image data of the next frame after the frame of interest from the buffer unit 101, and reads the block of interest and the image at time t + 2t ₀ . The function E ₂ (u ₂ , v ₂ ) representing the sum of absolute differences between corresponding pixels with the data is expressed as the position of the image data at the time t + 2t ₀ (x ₀ + u ₂ , y ₀ + v ₂ ) is changed and the function E ₂ (u ₂ , v ₂ ) is supplied as correlation information at time t + 2t _{0 to} the scaling synthesis unit 104, and the process proceeds to step S104.

In step S104, the correlation calculation unit 103 reads out the image data at time t + 3t ₀ that is temporally previous image data by 3 frames of the frame of interest from the buffer unit 101, and the block of interest and time t + 3t _0. The function E ₃ (u ₃ , v ₃ ) representing the sum of absolute differences between corresponding pixels with the image data of the image data position of the image data at time t + 3t ₀ (x ₀ + u ₃ , y ₀ + v ₃ ), and the function E ₃ (u ₃ , v ₃ ) is supplied as correlation information at time t + 3t _{0 to} the scaling synthesis unit 104, and the process proceeds to step S105.

In step S105, image data at time t + 4t ₀ that is temporally preceding image data by four frames of the target frame is read from the buffer unit 101, and the target block and the image data at time t + 4t ₀ are The function E ₄ (u ₄ , v ₄ ) representing the sum of absolute differences between corresponding pixels is set to the position (x ₀ + u ₄ , y ₀ + v) of the image data at time t + 4t _{0 to which} the target block is associated. ₄ ), and the function E ₄ (u ₄ , v ₄ ) is supplied to the scaling synthesis unit 104 as correlation information at time t + 4t ₀ , and the process proceeds to step S106 in FIG.

In step S106, correlation calculation section 103, the image data at time tt ₀ is image data of a previous frame of the frame of interest, from the buffer unit 101, and the block of interest, the image data at time tt _0, the corresponding The function F ₁ (r ₁ , s ₁ ) that represents the sum of absolute differences between pixels is changed by changing the position (x ₀ + r ₁ , y ₀ + s ₁ ) of the image data at time tt ₀ to which the block of interest is associated. The function F ₁ (r ₁ , s ₁ ) is obtained as correlation information at time tt ₀ and supplied to the scaling synthesizer 104, and the process proceeds to step S107.

In step S107, the correlation calculation unit 103 reads out the image data at time t-2t ₀ that is image data of the previous frame before the frame of interest from the buffer unit 101, and reads the block of interest and the image at time t-2t ₀ . The function F ₂ (r ₂ , s ₂ ) representing the sum of absolute differences between corresponding pixels with the data is expressed as the position of the image data at the time t-2t ₀ (x ₀ + r ₂ , y ₀ + s ₂ ) is changed and the function F ₂ (r ₂ , s ₂ ) is supplied as correlation information at time t−2t _{0 to} the scaling synthesis unit 104, and the process proceeds to step S108.

In step S108, the correlation calculating unit 103, the image data at time t-3t ₀ is image data before in time by three frames of the frame of interest, from the buffer unit 101, and the block of interest, the time t-3t ₀ The function F ₃ (r ₃ , s ₃ ) representing the sum of absolute differences between corresponding pixels with the image data of the image data position of the image data at time t-3t ₀ (x ₀ + r ₃ , y ₀ + s ₃ ), and the function F ₃ (r ₃ , s ₃ ) is supplied to the scaling combiner 104 as correlation information at time t-3t ₀ , and the process proceeds to step S109.

In step S109, the correlation calculation unit 103 reads out the image data at time t-4t ₀ that is temporally previous image data by four frames of the target frame from the buffer unit 101, reads the target block, and the time t-4t _0. The function F ₄ (r ₄ , s ₄ ) representing the sum of absolute differences between corresponding pixels with the image data of the image data at the time t-4t ₀ corresponding to the block of interest (x ₀ + r ₄ , y ₀ + s ₄ ), and the function F ₄ (r ₄ , s ₄ ) is supplied to the scaling synthesis unit 104 as correlation information at time t-4t ₀ , and step S110 in FIG. Proceed to

In step S <b> 110, the scaling composition unit 104 receives the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), supplied from the correlation calculation unit 103. E ₄ (u ₄ , v ₄ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), F ₄ (r ₄ , s ₄ ) Based on the scale of the position (u ₁ , v ₁ ) of the correlation information E ₁ (u ₁ , v ₁ ), the above-mentioned scaling is performed, and further, the correlation information after the scaling is synthesized, and the synthesized correlation Find information E (p, q). That is, the scaling combining unit 104, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (u 3/3, v 3/3) = (u 4 _{/ 4, v 4/4)} = (- r 1, -s 1) = (- r 2/2, -s 2/2) = (- r 3/3, -s 3/3) = (- r _4/4 , -s _4/4 ), and further scaling correlation information E ₁ (p, q), E ₂ (2p, 2q), E ₃ (3p, 3q), Add E ₄ (4p, 4q), F ₁ (-p, -q), F ₂ (-2p, -2q), F ₃ (-3p, -3q), F ₄ (-4p, -4q) Therefore, the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + E ₄ (4p, 4q) + F ₁ (- p, -q) + F ₂ (-2p, -2q) + F ₃ (-3p, -3q) + F ₄ (-4p, -4q)).

Note that the combined correlation information E (p, q) can be obtained by simply adding the correlation information after scaling, or by weighted addition as described above. That is, the combined correlation information E (p, q) is, for example, an expression 8 × E ₁ (p, q) + 4 × E _{2 in} which correlation information at a time closer to the time t of the block of interest is added with a greater weight. (2p, 2q) + 2 × E ₃ (3p, 3q) + 1 × E ₄ (4p, 4q) + 8 × F ₁ (-p, -q) + 4 × F ₂ (-2p, -2q) + It can be obtained by calculating 2 × F ₃ (−3p, −3q) + 1 × F ₄ (−4p, −4q).

  The scaling combining unit 104 obtains the combined correlation information E (p, q) in step S110, supplies the combined correlation information E (p, q) to the minimum value detecting unit 105, and proceeds to step S111. .

In step S111, the minimum value detection unit 105 has a position (relative position from the target block) in the spatial direction where the correlation represented by the combined correlation information E (p, q) from the scaling combining unit 104 is maximum (p, q) is detected as the maximum correlation position (p ₀ , q ₀ ), that is, the position (p, q) that minimizes the `` value '' of the combined correlation information E (p, q) is the maximum correlation position (p ₀ , q ₀ ), and the vector to the maximum correlation position (p ₀ , q ₀ ) is set as the average motion vector (p ₀ , q ₀ ), and the process proceeds to step S112.

In step S112, the minimum value detecting section 105, the average motion vector detected at step _{S111 (p 0, q 0)} , 3 -dimensional frame period t ₀ of the original video data was added as a component in the time direction t The direction orthogonal to the direction of the motion vector (p ₀ , q ₀ , t ₀ ) is the principal component direction, and the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) (FIG. 25) is supplied and the process is terminated.

In this case, for example, the filter information supply unit 32 directly uses the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) supplied from the minimum value detection unit 105 of the principal component direction acquisition unit 31 as filter information. It supplies to the filter part 22 (FIG. 25).

  In the filter unit 22, the process according to the flowchart shown in FIG. 68 is performed as the process of step S3 in FIG.

That is, in step S131, the filter unit 22 receives the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) supplied as filter information from the filter information supply unit 32, and proceeds to step S132. .

In step S132, the filter unit 22 calculates the pixel value C (t, x, y) of the pixel at the position (x, y) in the target block of the frame at time t, for example, the equation (1/16) × D ( t-3t ₀ , x-3p ₀ , y-3q ₀ ) + (2/16) × D (t-2t ₀ , x-2p ₀ , y-2q ₀ ) + (3/16) × D (tt ₀ , xp ₀ , yq ₀ ) + (4/16) × D (t, x, y) + (3/16) × D (t + t ₀ , x + p ₀ , y + q ₀ ) + (2 / 16) × D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) + (1/16) × D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ) Perform filtering. Note that D (t, x, y) represents a pixel value at a position (x, y) at time t of moving image data used for filtering.

Here, the equation (1/16) × D (t-3t ₀ , x-3p ₀ , y-3q ₀ ) + (2/16) × D (t-2t ₀ , x-2p ₀ , y-2q ₀ ) + (3/16) × D (tt ₀ , xp ₀ , yq ₀ ) + (4/16) × D (t, x, y) + (3/16) × D (t + t ₀ , x + p ₀ , y + q ₀ ) + (2/16) × D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) + (1/16) × D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), the so-called tap coefficient is 1/16, 2/16, 3/16, 4/16, 3/16, 2/16, 1/16, and 7 taps in the time direction It is equivalent to performing filtering by a FIR (Finite Impulse Response) filter having (filtering using image data of 7 frames). Such a tap coefficient FIR filter is a low-pass filter.

In step S132, filtering as the calculation of the pixel value C (t, x, y) is performed while down-sampling is performed to thin out the number of samples in the time direction to 1/4, that is, a ratio of one frame every four frames. Done in That is, t is a multiple of 4 × t ₀ , for example.

After the process of step S132, the process proceeds to step S133, and the filter unit 22 performs the filtering result, that is, the expression (1/16) × D (t−3t ₀ , x−3p ₀ , y−3q ₀ ) + (2 / 16) × D (t-2t ₀ , x-2p ₀ , y-2q ₀ ) + (3/16) × D (tt ₀ , xp ₀ , yq ₀ ) + (4/16) × D (t, x , y) + (3/16) × D (t + t ₀ , x + p ₀ , y + q ₀ ) + (2/16) × D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) + (1/16) × D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), the pixel value C (t, x, y) obtained by calculating , y) is output to the encoding unit 24 as a pixel value.

  In FIG. 68, the filtering is performed by the FIR filter having 7 taps in the time direction. However, for example, the filtering may be performed by the FIR filter having 9 taps in the time direction.

Also, the expression (1/16) × D (t-3t ₀ , x-3p ₀ , y-3q ₀ ) + (2/16) × D (t-2t ₀ , x-2p ₀ , y-2q ₀ ) + (3/16) × D (tt ₀ , xp ₀ , yq ₀ ) + (4/16) × D (t, x, y) + (3/16) × D (t + t ₀ , x + p ₀ , y + q ₀ ) + (2/16) × D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) + (1/16) × D (t + 3t ₀ , x + 3p ₀ , The calculation of y + 3q ₀ ) is equivalent to filtering by a low-pass filter, but it can also be regarded as weighted addition with a larger weight as it is closer to the frame of the target block.

In this way, when the pixel value C (t, x, y) as a filtering result is obtained by performing weighted addition with a greater weight as it is closer to the frame of the target block, the combined correlation information E (p, In the synthesis for obtaining q), as described above, the correlation information at the time closer to the time t of the block of interest is added with a larger weight and added to the equation 8 × E ₁ (p, q) + 4 × E ₂ ( 2p, 2q) + 2 × E ₃ (3p, 3q) + 1 × E ₄ (4p, 4q) + 8 × F ₁ (-p, -q) + 4 × F ₂ (-2p, -2q) +2 It is desirable to perform the calculation by × F ₃ (-3p, -3q) + 1 × F ₄ (-4p, -4q)

  As described above, not only the correlation information indicating the correlation between the block of interest and the next frame, but also the plurality of pieces of correlation information indicating the correlation between the block of interest and each of a plurality of frames before and after the block, Since an average motion vector representing an average motion for each of the frames is obtained, a highly accurate principal component direction can be obtained for the target block from the average motion vector.

  Further, the range of the plurality of frames used when obtaining the average motion vector is a range corresponding to the range of frames used for filtering in the filter unit 22 (FIG. 25), that is, a frame used for filtering (in FIG. 68, Since the range is almost the same as (7 frames centered on the frame of the target block) (9 frames centering on the frame of the target block in FIGS. 64 to 67), the filter unit 22 (FIG. 25) It is possible to obtain a filtering result (a filtering result of only frequency components that should be passed through).

  In FIG. 68, filtering is performed using 7 frames centered on the frame of the target block, and in FIGS. 64 to 67, the average motion vector is obtained using 9 frames centered on the frame of the target block. However, for example, filtering and calculation of the average motion vector can be performed using 7 frames or 9 frames centering on the frame of the target block. Filtering can also be performed using 9 frames centered on the frame of the target block, and the average motion vector can be calculated using 7 frames centered on the frame of the target block. That is, the frame range used for filtering and the frame range used for calculating the average motion vector may be completely coincident with each other or may be almost coincident with each other.

  As described above, by obtaining an average motion vector using a plurality of frames in a range substantially similar to the range of frames used for filtering in the filter unit 22 (FIG. 25), a plurality of average motion vectors are obtained from the average motion vector. A relatively accurate principal component direction can be obtained in the frame range.

  Next, as described above, according to the average motion vector, a relatively accurate principal component direction can be basically obtained.

  However, if the average motion vector is always obtained using all of a plurality of frames, an average motion vector having a large error may be obtained. In this case, the accuracy in the principal component direction is also deteriorated. Also, the filtering result is an inappropriate moving image (an image that cannot be restored to a moving image in which a human does not recognize deterioration in image quality in the receiving device 2).

  That is, for example, in moving image data, when a stationary object is used as a background and a moving object exists as a foreground on the front side of the background, when the block of interest includes only the background, another frame (frame of interest) In other frames, the background portion corresponding to the block of interest may be hidden behind the foreground and invisible. In this case, the value of the correlation information with the block of interest in the frame in which the background portion corresponding to the block of interest is hidden behind the foreground is not always the minimum at the position of the background portion corresponding to the block of interest. First, when the composite correlation information is calculated using such correlation information and the average motion vector is obtained, an average motion vector with a large error may be obtained.

  Specifically, it is assumed that the moving image data input to the transmission device 1 is, for example, moving image data on which a subject shown in FIG. 69 is projected.

Here, the horizontal axis in FIG. 69 represents the position x in the spatial direction. The upper side of FIG. 69 shows the waveform of the subject P _{6901 as} the background, and the lower side of FIG. 69 shows the waveform of the subject P _{6902 as} the foreground.

The subject P ₆₉₀₁ shown on the upper side of FIG. 69 is stationary, and the subject P ₆₉₀₂ shown on the lower side of FIG. 69 is moving in the spatial direction x at a speed of g / t ₀ [m / s].

In the above-described case, image data at 9 times (9 frames) from time t-4t _{0 to} t + 4t ₀ is used in calculating the average motion vector. However, hereinafter, time t-3t _{0 to} t Image data at 7 times (7 frames) of + 3t ₀ is used. Since the time from time t-3t _{0 to} t + 3t ₀ is as short as 6t ₀ , the speed of the subject P _{6902 at} that time can be regarded as being constant (approximate), and g / t ₀ described above Represents the constant speed.

70 and 71 show image data at seven times (7 frames) from time t-3t _{0 to} t + 3t ₀ on which the subjects P ₆₉₀₁ and P ₆₉₀₂ shown in FIG. 69 are projected.

In FIG. 70 and FIG. 71, image data at seven times from time t-3t _{0 to} t + 3t ₀ are illustrated in order from the top. FIG. 70 and FIG. 71 are the same figures except that the parts to which the reference numerals are attached are different.

According to the processing according to the flowcharts of FIGS. 64 to 67 described above (hereinafter, referred to as first motion detection processing as appropriate), for example, the image data at time t is divided into blocks (step S101 in FIG. 64). ). In FIGS. 70 and 71, the image data at time t is divided into 16 blocks of the block B ₁ to B _16.

For example, if the first motion detection process is performed on the assumption that the block B ₃ including only the subject P ₆₉₀₁ that is the background is the block of interest, as shown in FIG. 70, the image data at time t + t ₀ , a region corresponding to the target block B ₃ is because it is the block of interest B ₃ and the region B ₁₀₁ in the same position is determined at step S102 of FIG. 65, the correlation information at time _{t + t 0 E 1 (u} 1, v The value of ₁ ) is minimized when the position (u ₁ , v ₁ ) = (0,0).

Similarly, in the image data at times t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ , the region corresponding to the target block B ₃ is the same position as the target block B ₃ , respectively. region _{B 201, B 301, B -101} , B -201, a B _-301. Therefore, the correlation at time t + 2t ₀ Info _{E 2 (u 2, v 2} ), the correlation information E ₃ at time _{t + 3t 0 (u 3,} v 3), the correlation at time tt ₀ information F ₁ (r _1, s ₁ ), correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ , and correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ are all minimized. , (U ₂ , v ₂ ), (u ₃ , v ₃ ), (r ₁ , s ₁ ), (r ₂ , s ₂ ), (r ₃ , s ₃ ) are (0,0) It is.

Therefore, according to the first motion detection process, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u) after scaling obtained in step S111 of FIG. ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), and F ₃ (r ₃ , s ₃ ), the combined correlation information E (p, q) The minimum (p, q), that is, the average motion vector (p ₀ , q ₀ ) is (0,0). As a result, for the block of interest B ₃ , (0, 0, t ₀ ) is obtained.

In this case, in FIG. 68, for each pixel of the block of interest B _3, area _{B -301, B -201, B -101} , to _{B 3, B 101, B 201} , B 301 pixel values of respective corresponding pixels Filtering for performing weighted addition with weights 1/16, 2/16, 3/16, 4/16, 3/16, 2/16, 1/16 is performed. Region _{B -301, B -201, B -101} , B 3, B 101, B 201, B 301 are both an area corresponding to the target block B _3, therefore, the region B _-301, B _-201 , B _-101 , B ₃ , B ₁₀₁ , B ₂₀₁ , B ₃₀₁ , data of only frequency components that can be recognized by human vision, that is, (0,0, t _{0 in this case)} ) To obtain appropriate data of only the frequency component in the region R ₁₀₀₁ (FIG. 20) extending in the direction orthogonal to (the principal component direction of the data of the target block B ₃ ) and having a width in the T direction of 2π / (4t ₀ ). be able to.

Next, for example, if the first motion detection process is performed assuming that the block B ₁₀ including only the subject P ₆₉₀₂ which is the foreground is the _target block, as shown in FIG. 70, the image data at time t + t ₀ is displayed. in the region corresponding to the block of interest B ₁₀ is because it is the region B ₁₀₂ of position shifted by g in the x-direction from the target block B _10, obtained in the step S102 of FIG. 65, the correlation information at time t + t ₀ The value of E ₁ (u ₁ , v ₁ ) is minimized when the position (u ₁ , v ₁ ) = (g, 0).

Similarly, in the image data at times t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ , regions corresponding to the target block B ₁₀ are respectively in the x direction from the target block B _10. the position of the region B ₂₀₂ shifted by 2g, the block of interest B ₁₀ position of the region B ₃₀₂ shifted by 3g in the x direction from the block of interest B ₁₀ position of the region B _-102 shifted in the x direction by -g from attention area at a position shifted by -2g from the block B ₁₀ in the x direction B _-202, a region B _-302 position shifted by -3g in the x direction from the target block B _10.

Therefore, the _value of the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is minimized when the position (u ₂ , v ₂ ) is (2g, 0) and the time t + The _value of the correlation information E ₃ (u ₃ , v ₃ ) at 3t ₀ is minimized when the position (u ₃ , v ₃ ) is (3g, 0), and the correlation information F _{1 at} time tt ₀ (r _1, s ₁₎ the value of the is minimized, positions (r _1, s ₁₎ is (-g, 0) is when the correlation at time t-2t ₀ information F ₂ (r _2, The value of s ₂ ) is minimized when the position (r ₂ , s ₂ ) is (-2g, 0), and the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ The value is minimized when the position (r ₃ , s ₃ ) is ( ₋₃ g, 0).

From the above, according to the first motion detection process, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (after scaling) obtained in step S111 of FIG. u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), and the resultant correlation information E (p, q) (P, q), that is, the average motion vector (p ₀ , q ₀ ) is (g, 0). As a result, for the target block B ₁₀ , (g , 0, t ₀ ).

In this case, in FIG. 68, for each pixel of the block of interest B _10, region _{B -302, B -202, B -102} , relative to _{B 10, B 102, B 202} , B 302 pixel values of respective corresponding pixels Filtering for performing weighted addition with weights 1/16, 2/16, 3/16, 4/16, 3/16, 2/16, 1/16 is performed. Region _{B -302, B -202, B -102} , B 10, B 102, B 202, B 302 are both an area corresponding to the target block B _10, therefore, the region B _-302, B _-202 , B _-102 , B ₁₀ , B ₁₀₂ , B ₂₀₂ , B ₃₀₂ , data of only frequency components that can be recognized by human vision, that is, (g, 0, t _{0 in this case)} ) Proper data of only frequency components in a region extending in a direction orthogonal to (a principal component direction of data of the target block B ₁₀ ) and having a width in the T direction of 2π / (4t ₀ ) can be obtained.

As described above, when all of the areas corresponding to the block of interest and the block of interest in other frames include only the subject P ₆₉₀₁ that is the background or only the subject P ₆₉₀₂ that is the foreground, the block of interest By accurately obtaining the principal component direction of the data and filtering based on the principal component direction, it is possible to obtain appropriate data of only frequency components that can be recognized by human vision.

  On the other hand, if the area corresponding to the target block or the target block in another frame includes a plurality of subjects with different speeds, that is, includes a boundary portion of a plurality of subjects with different speeds, A large average motion vector may be required. In this case, from such an average motion vector, the principal component direction of the data of the block of interest cannot be accurately determined, and as a result, only frequency components that can be recognized by human vision by filtering based on the principal component direction. It may not be possible to obtain appropriate data.

That is, for example, in FIG. 71, if the first motion detection process is performed assuming that the block B ₇ including only the subject P _{6901 as the} background is the _target block, the _target block B in the image data at time t + t ₀ is performed. _Since the area corresponding to ₇ is the area B ₁₁₁ at the same position as the target block B ₇ , the _value of the correlation information E ₁ (u ₁ , v ₁ ) at time t + t ₀ obtained in step S102 of FIG. Is minimized when the position (u ₁ , v ₁ ) = (0,0).

Similarly, in the image data at time _{t + 2t 0, t + 3t} 0, tt 0, t-2t 0, a region corresponding to the block of interest B _7, which may each focus block B ₇ in the same position of the region B _211, B _311, B _-111, is a B _-211. Therefore, the correlation at time t + 2t ₀ Info _{E 2 (u 2, v 2} ), the correlation information E ₃ at time _{t + 3t 0 (u 3,} v 3), the correlation at time tt ₀ information F ₁ (r _1, s ₁ ) and the _value of the correlation information F ₂ (r ₂ , s ₂ ) at the time t-2t ₀ are minimized because the position (u ₂ , v ₂ ), (u ₃ , v ₃ ), This is when (r ₁ , s ₁ ) and (r ₂ , s ₂ ) are (0,0).

However, in the image data at time t-3t _0, a region corresponding to the block of interest B _7, which may Originally, it should be region B _-311 at the same position as the block of interest B _7, FIG. 71, moving The subject is hidden behind the subject P ₆₉₀₂ and does not exist. Therefore, the _value of the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ is minimized when the position (r ₃ , s ₃ ) is (0,0). The value of the correlation information F ₃ (r ₃ , s ₃ ) is not always the minimum when the position (r ₃ , s ₃ ) is (0,0).

Then, according to the first motion detection process, in step S111 in FIG. 67, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) (P, q), that is, an average motion vector (p ₀ , q ₀ ) is obtained, and (p ₀ , q ₀ , t ₀ ) is obtained as a three-dimensional motion vector.

The three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) is not always the minimum value when the position (r ₃ , s ₃ ) is (0,0). The correlation information F ₃ (r ₃ , s ₃ ) is obtained from the average motion vector (p ₀ , q ₀ ) that is (p, q) that minimizes the composite correlation information E (p, q) obtained using The direction orthogonal to the vector (p ₀ , q ₀ , t ₀ ) does not always accurately represent the principal component direction of the data of the block of interest B ₇ .

In such filtering based on a direction orthogonal to the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), it is difficult to obtain appropriate data of only frequency components that can be recognized by human vision. It becomes.

Next, for example, in FIG. 71, if the first motion detection process is performed assuming that the block B ₁₄ including only the subject P _{6901 as the} background is the _target block, the _target block is included in the image data at time t + t ₀ . a region corresponding to the B ₁₄ is because it is the region B ₁₁₂ of the block of interest B ₁₄ at the same position is determined at step S102 of FIG. 65, the correlation information at time t + t ₀ E ₁ of (u _1, v ₁₎ The value is minimized when the position (u ₁ , v ₁ ) = (0,0).

Similarly, in the image data at time _{tt 0, t-2t 0,} t-3t 0, a region corresponding to the block of interest B _14, respectively, the block of interest B ₁₄ at the same position in the region B _-112, B _-212, B- ₃₁₂ . Therefore, correlation information F ₁ (r ₁ , s ₁ ) at time tt _0, correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ , and correlation information F ₃ (r ₃ , r at time t-3t ₀ The value of s ₃ ) is minimized when the positions (r ₁ , s ₁ ), (r ₂ , s ₂ ), and (r ₃ , s ₃ ) are (0, 0). .

However, in the image data at time t + 2t _0, a region corresponding to the block of interest B ₁₄ is originally, but it should be region B ₂₁₂ of the block of interest B ₁₄ at the same position, moving the subject P ₆₉₀₂ are It is hidden behind and does not exist. Therefore, the _value of the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is minimized when the position (u ₂ , v ₂ ) is (0,0). The value of the correlation information E ₂ (u ₂ , v ₂ ) is not necessarily the minimum when the position (u ₂ , v ₂ ) is (0,0).

Similarly, in the image data at time t + 3t _0, a region corresponding to the block of interest B ₁₄ are originally of interest but block B ₁₄ and it should be region B ₃₁₂ in the same position, a moving subject P ₆₉₀₂ It is hidden behind and does not exist. Therefore, the _value of the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is minimized when the position (u ₃ , v ₃ ) is (0,0). The value of the correlation information E ₃ (u ₃ , v ₃ ) is not necessarily the minimum when the position (u ₃ , v ₃ ) is (0,0).

Then, according to the first motion detection process, in step S111 in FIG. 67, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) (P, q), that is, an average motion vector (p ₀ , q ₀ ) is obtained, and (p ₀ , q ₀ , t ₀ ) is obtained as a three-dimensional motion vector.

This three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) is not always the minimum when the position (u ₂ , v ₂ ) is (0,0), and the correlation information F ₂ (u ₂ , v ₂ ) and the correlation information F ₃ (u ₃ , v ₃ ), the value of which is not necessarily the minimum when the position (u ₃ , v ₃ ) is (0,0) Since it is obtained from the average motion vector (p ₀ , q ₀ ) that is (p, q) that minimizes the information E (p, q), the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) the direction perpendicular, not necessarily the main ingredient direction of the data block of interest B ₁₄ accurately represents.

In such filtering based on a direction orthogonal to the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), it is difficult to obtain appropriate data of only frequency components that can be recognized by human vision. It becomes.

Next, for example, in FIG. 71, if the block B ₉ including the subject P _{6901 that} is the background and the subject P ₆₉₀₂ that is the foreground is the _target block, and the first motion detection process is performed, the target block B ₉ is Since both the stationary subject P ₆₉₀₁ and the moving subject P ₆₉₀₂ are included, that is, the moving subject P ₆₉₀₂ is stationary at the position of the stationary subject P ₆₉₀₁ . Since P ₆₉₀₁ is hidden, the areas corresponding to such a target block B ₉ are time t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t−2t ₀ , t It does not exist in any of the image data at -3t ₀ .

For this reason, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , t at time t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) Positions (u ₁ , v ₁ ), (u ₂ ) most similar to the block of interest B ₉ in the image data at t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ , v ₂ ), (u ₃ , v ₃ ), (r ₁ , s ₁ ), (r ₂ , s ₂ ), (r ₃ , s ₃ )

Then, according to the first motion detection process, in step S111 in FIG. 67, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) (P, q), that is, an average motion vector (p ₀ , q ₀ ) is obtained, and (p ₀ , q ₀ , t ₀ ) is obtained as a three-dimensional motion vector.

This three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) has the minimum correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v) at the position most similar to the target block B _{9. 2} ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) Since it is obtained from the average motion vector (p ₀ , q ₀ ) that is (p, q) that minimizes the information E (p, q), the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) the direction perpendicular, not necessarily the main ingredient direction of the data block of interest B ₉ accurately represents.

In such filtering based on a direction orthogonal to the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), it is difficult to obtain appropriate data of only frequency components that can be recognized by human vision. It becomes.

  As described above, when the region corresponding to the block of interest or the block of interest in another frame includes a plurality of subjects having different speeds, that is, if the region includes boundary portions of a plurality of subjects having different speeds, the first In this motion detection processing, the average motion vector, and thus the principal component direction of the data of the block of interest, cannot be obtained accurately, and the filtering based on such principal component direction is performed, so that it is recognized by human vision. It may be difficult to obtain appropriate data of only frequency components that can be obtained.

  Therefore, FIG. 72 shows the average motion vector, and thus the principal component direction of the data of the target block, even when the region corresponding to the target block or the target block of another frame includes a plurality of subjects having different velocities. 25 shows an example of the configuration of the principal component direction acquisition unit 31 in FIG. 25 that performs a process for accurately obtaining (hereinafter referred to as a second motion detection process as appropriate).

  In the figure, portions corresponding to those in FIG. 52 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, the principal component direction acquisition unit 31 in FIG. 72 is different from the scaling synthesis unit 104 and the minimum value detection unit 105 in that the scaling synthesis unit 124 and the minimum value detection unit 125 are provided in the case of FIG. It is configured in the same way.

  Similar to the scaling synthesis unit 104 in FIG. 52, the scaling synthesis unit 124 scales the correlation information obtained from each of the correlation calculation target frames supplied from the correlation calculation unit 103 in the spatial directions x and y, Further, the correlation information after scaling is synthesized to obtain synthesized correlation information. However, the scaling combining unit 124 obtains a plurality of pieces of combined correlation information by changing the number of pieces of correlation information after scaling (the number of pieces of correlation information to be combined). Then, the scaling synthesis unit 124 supplies the plurality of synthesis correlation information to the minimum value detection unit 125.

  The minimum value detecting unit 125 obtains maximum combined correlation information that is the combined correlation information having the maximum correlation among the plurality of combined correlation information from the scaling combining unit 124. That is, the minimum value detection unit 125 obtains the minimum value of “value” of each of the plurality of combined correlation information from the scaling combining unit 124. Further, the minimum value detection unit 125 obtains the composite correlation information having the minimum minimum value as the maximum composite correlation information from the plurality of composite correlation information.

Further, the minimum value detection unit 125 sets the position in the spatial direction where the correlation represented by the maximum combined correlation information is maximum, that is, the position in the spatial direction that minimizes the “value” of the maximum combined correlation information as the maximum correlation position. The vector to the maximum correlation position is detected as an average motion vector. Then, the minimum value detection unit 125 obtains a three-dimensional motion vector obtained by adding the frame period t ₀ of the moving image data stored in the buffer unit 21 (FIG. 25) to the average motion vector as a component in the time direction t. The direction orthogonal to the direction of the three-dimensional motion vector is detected and output as the principal component direction of the block of interest.

  Further, the minimum value detection unit 125 represents a range of frames used for calculating the correlation information synthesized to obtain the maximum synthesized correlation information among a plurality of frames used to obtain the synthesized correlation information. The kernel size to be calculated is further obtained as the effective range of the motion vector (average motion vector) of the target block, that is, the effective range of the main component direction of the target block, and is output together with the main component direction of the target block.

  Next, the processing of the filter generation unit 23 in FIG. 25 when the principal component direction acquisition unit 31 is configured as shown in FIG. 72 will be described with reference to the flowcharts in FIGS. 73 to 77.

  Of the processes of steps S151 to S176 according to the flowcharts of FIGS. 73 to 77, the processes of steps S151 to S172 are second motion detection processes and correspond to the process of step S11 of FIG. Further, the processing in steps S173 to S175 corresponds to the processing in step S12 in FIG. 31, and the processing in step S176 corresponds to the processing in step S13 in FIG.

  In the second motion detection process, an area corresponding to the target block among a plurality of frames for calculating correlation information with the target block using the so-called property of a moving image on which a plurality of subjects with different speeds are projected. Only the correlation information obtained for a frame having a frame is combined to obtain combined correlation information, and an average motion vector capable of obtaining an accurate principal component direction of the block of interest is obtained using the combined correlation information.

That is, for example, as shown in FIG. 69 to FIG. 71, seven frames from time t-3t _{0 to} t + 3t ₀ where the moving subject P ₆₉₀₂ exists on the front side of the stationary subject P ₆₉₀₁ are shown. Since the time between them is as short as 6t ₀ , it can be approximated that the speed of the moving subject P ₆₉₀₂ is constant.

Then, when a block having image data at time t is set as a target block, in the case where the moving subject P ₆₉₀₂ moves away from the _target block, a time later than time t ( In the image data at a future time), there is an area corresponding to the block of interest. That is, for example, in FIG. 71, when the block B ₇ is the _target block, the subject P ₆₉₀₂ moves away from the _target block B ₇ , but in this case, the time later than the time t At times t + t ₀ , t + 2t ₀ , and t + 3t ₀ , areas B ₁₁₁ , B ₂₁₁ , and B ₃₁₁ corresponding to the block of interest B ₇ exist.

In addition, in the case where the moving subject P ₆₉₀₂ approaches the block of interest, there is an area corresponding to the block of interest in the image data at a time prior to the time t (past time). To do. That is, for example, in FIG. 71, when the block B ₁₄ is the _target block, the subject P ₆₉₀₂ approaches the _target block B ₁₄ , but in this case, the time before the time t At times tt ₀ , t-2t ₀ , and t-3t ₀ , there are regions B ₋₁₁₂ , B ₋₂₁₂ , and B ₋₃₁₂ corresponding to the block of interest B ₁₄ .

Note that in the case where the subject block includes a stationary subject P ₆₉₀₁ and a moving subject P ₆₉₀₂ , there is no region corresponding to the subject block in the image data at times before and after time t. That is, for example, in FIG. 71, when the block B ₉ is the target block, the target block B ₉ includes a stationary subject P ₆₉₀₁ and a moving subject P _{6902. In} this case, , The block of interest B _{9 at} any of the times tt ₀ , t-2t ₀ , t-3t ₀ before the time t and the times t + t ₀ , t + 2t ₀ , t + 3t ₀ after the time t There is no corresponding area.

  In the second motion detection process, a process using the nature of a moving image in which a plurality of subjects with different speeds is projected as described above is performed.

  That is, in the principal component direction acquisition unit 31 in FIG. 72, the moving image data read from the buffer unit 21 (FIG. 25) is supplied to the buffer unit 101, and the buffer unit 101 temporarily stores the moving image data.

  In step S151 in FIG. 73, the block extraction unit 102 divides the moving image data stored in the buffer unit 101 into, for example, blocks of 16 × 16 pixels, as in step S101 in FIG. To the unit 103. In the subsequent processing, the blocks obtained by the block extraction unit 102 are sequentially performed as the target block.

Here, as in FIGS. 64 to 67, the position of the block of interest (for example, the position of the upper left pixel of the block of interest) is represented as (x ₀ , y ₀ ). The frame of the block of interest (frame of interest) is assumed to be a frame at time t.

After the processing in step S151, the process proceeds to step S152, in which the correlation calculation unit 103 reads the image data at time t + t ₀ that is the image data of the frame next to the frame of interest from the buffer unit 101, The function E ₁ (u ₁ , v ₁ ) representing the sum of absolute differences between corresponding pixels with the image data at t + t ₀ is expressed as the position of the image data at time t + t ₀ ( x ₀ + u ₁ , y ₀ + v ₁ ), and the function E ₁ (u ₁ , v ₁ ) is supplied to the scaling synthesis unit 124 as correlation information at time t + t ₀ The process proceeds to S153.

In step S153, the correlation calculation unit 103 reads the image data at time t + 2t ₀ that is the image data of the next frame after the frame of interest from the buffer unit 101, and reads the block of interest and the image at time t + 2t ₀ . The function E ₂ (u ₂ , v ₂ ) representing the sum of absolute differences between corresponding pixels with the data is expressed as the position of the image data at the time t + 2t ₀ (x ₀ + u ₂ , y ₀ + v ₂ ) is changed and the function E ₂ (u ₂ , v ₂ ) is supplied as correlation information at time t + 2t _{0 to} the scaling synthesis unit 124, and the process proceeds to step S154.

In step S154, the correlation calculation unit 103 reads out the image data at time t + 3t ₀ that is temporally preceding image data by three frames of the target frame from the buffer unit 101, and sets the target block and time t + 3t _0. A function E ₃ (u ₃ , v ₃ ) representing the sum of absolute differences between corresponding pixels with the image data of the image data position of the image data at time t + 3t ₀ (x ₀ + u ₃ , y ₀ + v ₃ ), and the function E ₃ (u ₃ , v ₃ ) is supplied to the scaling combiner 124 as correlation information at time t + 3t ₀ , and step S155 in FIG. Proceed to

In step S155, the correlation calculation unit 103 reads the image data at time tt ₀ that is the image data of the frame before the frame of interest from the buffer unit 101, and corresponds the block of interest to the image data at time tt _0. Change the position (x ₀ + r ₁ , y ₀ + s ₁ ) of the image data at the time tt ₀ to associate the block of interest with the function F ₁ (r ₁ , s ₁ ) that represents the sum of absolute differences between pixels. The function F ₁ (r ₁ , s ₁ ) is obtained as correlation information at time tt ₀ and supplied to the scaling synthesizer 124, and the process proceeds to step S156.

In step S156, the correlation calculation unit 103 reads the image data at the time t-2t ₀ that is the image data of the previous frame before the target frame from the buffer unit 101, and reads the target block and the image at the time t-2t ₀ . The function F ₂ (r ₂ , s ₂ ) representing the sum of absolute differences between corresponding pixels with the data is expressed as the position of the image data at the time t-2t ₀ (x ₀ + r ₂ , y ₀ + s ₂ ) is changed, and the function F ₂ (r ₂ , s ₂ ) is supplied as the correlation information at time t−2t _{0 to} the scaling combiner 124, and the process proceeds to step S157.

In step S157, the correlation calculating unit 103, the image data at time t-3t ₀ is image data before in time by three frames of the frame of interest, from the buffer unit 101, and the block of interest, the time t-3t ₀ of the image data, the function _{F 3 (r 3, s 3} ) which represents the sum of the absolute differences of each other corresponding pixel, the position of the image data at time t-3t ₀ to correspond to the block of interest (x ₀ + r ₃ , y ₀ + s ₃ ), and the function F ₃ (r ₃ , s ₃ ) is supplied to the scaling combiner 124 as correlation information at time t-3t ₀ , and the process proceeds to step S158.

In step S <b> 158, the scaling synthesis unit 124 includes six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), the position of the correlation information E ₁ (u ₁ , v ₁ ) Scaling is performed with the scale of (u ₁ , v ₁ ) as a reference, and further, the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (u 3/3, v 3/3) = (- r _{1, -s 1) = (-} r 2/2, -s 2/2) = (- r 3/3, performs scaling with -s _3/3), further, the correlation information after the scaling E ₁ (p, q), E ₂ (2p, 2q), E ₃ (3p, 3q), F ₁ (-p, -q), F ₂ (-2p, -2q), F ₃ (-3p, -3q), the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + F ₁ (-p, -q) + F ₂ (-2p, -2q) + F ₃ (-3p, -3q)).

  Note that the combined correlation information E (p, q) can be obtained by simply adding the correlation information after scaling, or by weighted addition as described above.

As described above, the scaling synthesizer 124, in step S158, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t _0, correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ , correlation information F ₂ (r ₂ , s at time t-2t _{0 2} ), combined correlation information E (p, q) that combines all the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ , that is, after time t of the frame of interest After obtaining the combined correlation information E (p, q), which is a total of 6 correlation information for each of the 3 frames and the previous 3 frames in time, the combined correlation information E (p, q) is obtained. The obtained correlation information is normalized by dividing the combined correlation information E (p, q) by six, which is the number of correlation information combined. It is added to the value L ₃₃ By the final synthesized correlation information E (p, q) (= {E 1 (p, q) + E 2 (2p, 2q) + E 3 (3p, 3q) + F 1 (-p, -q ) + F ₂ (−2p, −2q) + F ₃ (−3p, −3q)} / 6 + L ₃₃ ). Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S158 to S159.

In step S159, the minimum value detecting unit 125 supplies the three frames that are temporally subsequent to the time t of the frame of interest and the three frames that are temporally previous, supplied from the scaling composition unit 124 in step S158. (P ₃₃ , q ₃₃ ), which is the position (p, q) in the spatial direction where the correlation represented by the combined correlation information E (p, q), which is obtained by synthesizing the correlation information, is maximized, that is, the combined correlation information E Detect (p ₃₃ , q ₃₃ ), which is the position (p, q) that minimizes the `` value '' of (p, q), and position (p, q) is the maximum correlation position (p ₃₃ , q ₃₃ ) Stored together with the “value” of the composite correlation information E (p ₃₃ , q ₃₃ ) in some cases, that is, the minimum value E ₃₃ , proceeds to step S160 in FIG.

In step S <b> 160, the scaling synthesis unit 124 includes six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), five correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), correlation information E ₁ (u ₁ , v ₁ ) Is scaled with reference to the scale of the position (u ₁ , v ₁ ), and the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (u 3/3, v 3/3) = (- r _{1, -s 1) = (-} r 2/2, performs scaling with -s _2/2), further, the correlation information E ₁ after the scaling _{(p, q), E 2} (2p, 2q ), E ₃ (3p, 3q), F ₁ (-p, -q), and F ₂ (-2p, -2q) are added to obtain the combined correlation information E (p, q) (= E ₁ (p , q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + F ₁ (−p, −q) + F ₂ (−2p, −2q)).

As described above, the scaling synthesizer 124, in step S160, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t _0, correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ , correlation information F ₂ (r ₂ , s at time t-2t _{0 2} ), the combined correlation information E (p, q), that is, the sum of the three frames after the time t and the two frames before the time t of the frame of interest. The composite correlation information E (p, q) obtained by synthesizing the five correlation information in the above is obtained, and then the composite correlation information E is calculated by 5 which is the number of correlation information synthesized in obtaining the composite correlation information E (p, q). Normalization is performed by dividing (p, q) .Further, by adding a predetermined offset value L ₂₃ to the normalized combined correlation information E (p, q), the final combined correlation information E ( p, q) (= {E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + F ₁ (-p, -q) + F ₂ (-2p, -2q)} / 5 + L ₂₃ ) Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S160 to S161.

In step S161, the minimum value detector 125 supplies each of the three frames that are temporally subsequent to the time t of the frame of interest and the two frames that are temporally previous, supplied from the scaling composition unit 124 in step S160. (P ₂₃ , q ₂₃ ), which is the position (p, q) in the spatial direction at which the correlation represented by the combined correlation information E (p, q), which is obtained by combining the correlation information, is maximized, that is, the combined correlation information E (p ₂₃ , q ₂₃ ), which is the position (p, q) that minimizes the (value) of (p, q), is detected, and the position (p, q) is the maximum correlation position (p ₂₃ , q ₂₃ ) Stored together with the “value” of the composite correlation information E (p ₂₃ , q ₂₃ ) in some cases, that is, the minimum value E ₂₃ , proceeds to step S162.

In step S <b> 162, the scaling combining unit 124 includes six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), four correlation information E ₁ (u ₁ , v ₁ ), E _{2 (u 2, v 2)} , E 3 (u 3, v 3), the _{F 1 (r 1, s 1} ), the position of the correlation information _{E 1 (u 1, v 1} ) (u 1, v 1) Is scaled with reference to the scale, and the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (u 3/3, v 3/3) = (- r ₁ , -s ₁ ), and scaling information E ₁ (p, q), E ₂ (2p, 2q), E ₃ (3p, 3q), F ₁ (- By adding (p, -q), the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + F ₁ (- p, -q)).

As described above, the scaling synthesizer 124, in step S162, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Combined correlation information E (p, q) obtained by combining four pieces of correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t _{0 and} correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ , That is, the combined correlation information E (p, q) obtained by combining a total of four pieces of correlation information for each of the three frames that are temporally subsequent to the time t of the frame of interest and the one frame that is temporally previous. After obtaining, the composite correlation information E (p, q) is normalized by dividing the composite correlation information E (p, q) by 4 which is the number of correlation information synthesized in obtaining the composite correlation information E (p, q). By adding a predetermined offset value L ₁₃ to the composite correlation information E (p, q) of the final composite correlation information E (p, q) (= {E ₁ (p, q) + E ₂ ( 2p, 2q) + E ₃ (3p, 3q) + F ₁ (-p, -q)} / 4 + L ₁₃ ) Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S162 to S163.

In step S163, the minimum value detecting unit 125 supplies the three frames that are temporally subsequent to the time t of the frame of interest and the one frame that is temporally previous supplied from the scaling composition unit 124 in step S162. (P ₁₃ , q ₁₃ ), which is the position (p, q) in the spatial direction at which the correlation represented by the combined correlation information E (p, q), which is obtained by combining the correlation information, is maximized, that is, the combined correlation information E (p ₁₃ , q ₁₃ ), which is the position (p, q) that minimizes the (value) of (p, q), is detected, and the position (p, q) is the maximum correlation position (p ₁₃ , q ₁₃ ) The value is stored together with the “value” of the composite correlation information E (p ₁₃ , q ₁₃ ) in some cases, that is, the minimum value E ₁₃ , and the process proceeds to step S164.

In step S164, the scaling synthesis unit 124 receives the six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), three correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ) and E ₃ (u ₃ , v ₃ ) are scaled based on the scale of the position (u ₁ , v ₁ ) of the correlation information E ₁ (u ₁ , v ₁ ), and Then, the scaled correlation information is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = With _{(u 3/3, v 3} /3) By performing scaling and further adding the correlation information E ₁ (p, q), E ₂ (2p, 2q), E ₃ (3p, 3q) after the scaling, the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q)) is obtained.

As described above, the scaling synthesizer 124, in step S164, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Combined correlation information E (p, q) obtained by combining three pieces of correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ , that is, 3 later in time with respect to time t of the frame of interest This is the number of pieces of correlation information synthesized for obtaining the combined correlation information E (p, q) after obtaining the combined correlation information E (p, q) obtained by combining a total of three pieces of correlation information for each frame. 3 is normalized by dividing the composite correlation information E (p, q) by 3 and, further, by adding a predetermined offset value L ₀₃ to the composite correlation information E (p, q) after normalization, Synthetic correlation information E (p, q) (= {E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q)} / 3 + L ₀₃ ) is obtained. Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S164 to S165.

In step S165, the minimum value detection unit 125 synthesizes the correlation information supplied from the scaling synthesis unit 124 in step S164 and synthesizes the correlation information for each of the three frames after the time t of the frame of interest. (P ₀₃ , q ₀₃ ), which is the position (p, q) in the spatial direction where the correlation represented by E (p, q) is maximum, is detected, that is, the “value” of the combined correlation information E (p, q) (P ₀₃ , q ₀₃ ), which is a position (p, q) that minimizes the position of the signal, is detected, and the combined correlation information E (p (p, q) is the maximum correlation position (p ₀₃ , q ₀₃ ) ₀₃ , q ₀₃ ), ie, the minimum value E ₀₃ , and the process proceeds to step S166 in FIG.

In step S <b> 166, the scaling synthesis unit 124 includes the six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), five correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), correlation information E ₁ (u ₁ , v ₁ ) Is scaled with reference to the scale of the position (u ₁ , v ₁ ), and the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (- r 1, -s 1) = (- r 2 / _{2, -s 2/2) =} (- r 3/3, performs scaling with -s _3/3), further, the correlation information E ₁ after the scaling _{(p, q), E 2} (2p , 2q), F ₁ (−p, −q), F ₂ (−2p, −2q), and F ₃ (−3p, −3q), the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + F ₁ (−p, −q) + F ₂ (−2p, −2q) + F ₃ (−3p, −3q)) is obtained.

As described above, the scaling synthesizer 124, in step S166, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Correlation information F ₁ (r ₁ , s ₁ ) at time tt _0, correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ , correlation information F ₃ (r ₃ , s at time t-3t _{0 3} ) combined correlation information E (p, q), that is, a total of five for each of the two frames later in time with respect to the time t of the frame of interest and the three frames earlier in time. The composite correlation information E (p, q) obtained by synthesizing the correlation information is obtained, and then the composite correlation information E (p, q) is obtained by 5 which is the number of correlation information synthesized in obtaining the composite correlation information E (p, q). q) is divided by normalization, and the final combined correlation information E (p, q is added by adding a predetermined offset value L ₃₂ to the normalized combined correlation information E (p, q). ) (= {E ₁ (p, q ) + E ₂ (2p, 2q) + F ₁ (−p, −q) + F ₂ (−2p, −2q) + F ₃ (−3p, −3q)} / 5 + L ₃₂ ). Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S166 to S167.

In step S167, the minimum value detection unit 125 supplies each of the two frames that are temporally subsequent to the time t of the frame of interest and the three frames that are temporally previous supplied from the scaling composition unit 124 in step S166. (P ₃₂ , q ₃₂ ), which is the position (p, q) in the spatial direction where the correlation represented by the combined correlation information E (p, q) that is obtained by combining the correlation information of Detect (p ₃₂ , q ₃₂ ), which is the position (p, q) that minimizes the `` value '' of (p, q), and position (p, q) is the maximum correlation position (p ₃₂ , q ₃₂ ) Stored together with the “value” of the composite correlation information E (p ₃₂ , q ₃₂ ) in some cases, that is, the minimum value E ₃₂ , proceeds to step S168.

In step S168, the scaling synthesis unit 124 receives the six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), four correlation information E ₁ (u ₁ , v ₁ ), F _{1 (r 1, s 1)} , F 2 (r 2, s 2), F 3 (r 3, s 3) for the correlation information E ₁ (u _1, v ₁₎ the position of the (u _1, v ₁₎ Is scaled with reference to the scale, and the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (- r 1, -s 1) = (- r 2/2, -s 2/2) = (- r _3/3 performs scaling with -s _3/3), further, its the scaled correlation information _{E 1 (p, q),} F 1 (-p, -q), F 2 (-2p, -2q) and F ₃ (-3p, -3q) are added to obtain the combined correlation information E (p, q) (= E ₁ (p, q) + F ₁ (-p, -q) + F ₂ (-2p, -2q) + F ₃ (-3p, -3q)) is obtained.

As described above, the scaling synthesizer 124, in step S168, the correlation at time t + t ₀ information _{E 1 (u 1, v 1} ), the correlation information at time _{tt 0 F 1 (r 1,} s 1), the time Synthetic correlation information E (p, q) obtained by synthesizing correlation information F ₂ (r ₂ , s ₂ ) at t-2t _{0 and} correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ , that is, attention After obtaining combined correlation information E (p, q), which is a total of four pieces of correlation information for one frame later in time with respect to time t of the frame and three frames earlier in time. Then, the composite correlation information E (p, q) is normalized by dividing the composite correlation information E (p, q) by 4 which is the number of correlation information synthesized in obtaining the composite correlation information E (p, q). By adding a predetermined offset value L ₃₁ to the information E (p, q), the final combined correlation information E (p, q) (= {E ₁ (p, q) + F ₁ (−p, -q) + F ₂ (-2p, -2q) + F ₃ (-3p, -3q)} / 4 + L ₃₁ ) obtain. Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S168 to S169.

In step S169, the minimum value detecting unit 125 supplies each of one frame that is temporally subsequent to the time t of the frame of interest and the three frames that are temporally previous, supplied from the scaling composition unit 124 in step S168. (P ₃₁ , q ₃₁ ), which is the position (p, q) in the spatial direction at which the correlation represented by the combined correlation information E (p, q), which is obtained by combining the correlation information, is maximized, that is, the combined correlation information E (p ₃₁ , q ₃₁ ), which is the position (p, q) that minimizes the (value) of (p, q), is detected, and the position (p, q) is the maximum correlation position (p ₃₁ , q ₃₁ ) Stored together with the “value” of the composite correlation information E (p ₃₁ , q ₃₁ ) in some cases, that is, the minimum value E ₃₁ , proceeds to step S170.

In step S <b> 170, the scaling composition unit 124 includes the six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), three correlation information F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ) and F ₃ (r ₃ , s ₃ ) are scaled with reference to the scale of the position (u ₁ , v ₁ ) of the correlation information E ₁ (u ₁ , v ₁ ), Then, the scaled correlation information is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (- r 1, -s 1) = (- r 2/2, -s 2/2) = (- r 3/3, -s 3 / Scaling is performed by 3), and the correlation information F ₁ (-p, -q), F ₂ (-2p, -2q), and F ₃ (-3p, -3q) after the scaling is added. Thus, the composite correlation information E (p, q) (= F ₁ (−p, −q) + F ₂ (−2p, −2q) + F ₃ (−3p, −3q)) is obtained.

As described above, in step S170, the scaling composition unit 124 calculates the correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ , the correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ , and the time The combined correlation information E (p, q) obtained by synthesizing the correlation information F ₃ (r ₃ , s ₃ ) at t-3t ₀ , that is, for each of the three frames temporally prior to the time t of the frame of interest After obtaining the combined correlation information E (p, q) obtained by combining the three pieces of correlation information in total, the number of pieces of correlation information combined in determining the combined correlation information E (p, q) is 3 to determine the combined correlation information. normalized by dividing the information E (p, q), further synthesized correlation information E after the normalization (p, q) in, by adding a predetermined offset value L _30, final synthesized correlation information E (p, q) (= {F ₁ (−p, −q) + F ₂ (−2p, −2q) + F ₃ (−3p, −3q)} / 3 + L ₃₀ ) is obtained. Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S170 to S171.

In step S171, the minimum value detection unit 125 synthesizes the correlation information supplied from the scaling synthesis unit 124 in step S170 and synthesizes the correlation information for each of the previous three frames with respect to time t of the frame of interest. (P ₃₀ , q ₃₀ ) is detected as the position (p, q) in the spatial direction where the correlation represented by E (p, q) is maximum, that is, the `` value '' of the combined correlation information E (p, q) (P ₃₀ , q ₃₀ ), which is the position (p, q) that minimizes, and the position (p, q) is the maximum correlation position (p ₃₀ , q ₃₀ ). ₃₀ , q ₃₀ ), ie, the minimum value E ₃₀ , and the process proceeds to step S 172 in FIG. 77.

In step S <b> 172, the minimum value detection unit 125 obtains the maximum combined correlation information that is the combined correlation information with the maximum correlation among the plurality of combined correlation information from the scaling combining unit 124. In other words, the minimum value detection unit 125 synthesizes the combined correlation information E (p, q) obtained by synthesizing the six correlation information of the three frames later in time and the previous three frames with respect to the time t of the frame of interest. The minimum value E ₃₃ (step S159), the combined correlation information E (p, q) obtained by synthesizing the five correlation information of the temporally subsequent three frames and the temporally previous two frames with respect to the time t of the frame of interest. The minimum value E ₂₃ (step S161), the combined correlation information E (p, q) obtained by synthesizing the four correlation information of the temporally subsequent three frames and the temporally previous one frame with respect to the time t of the frame of interest. Minimum value E ₁₃ (step S163), minimum value E _{03 of} combined correlation information E (p, q) obtained by synthesizing three pieces of correlation information of three frames later in time with respect to time t of the frame of interest (step S165) , 2 frames later in time and 3 frames earlier in time with respect to time t of the frame of interest The minimum value E ₃₂ (step S167) of the combined correlation information E (p, q) obtained by synthesizing the five pieces of correlation information of the first frame, one frame later in time with respect to the time t of the frame of interest, and the previous three in time The minimum value E _{31 of} the combined correlation information E (p, q) obtained by combining the four pieces of correlation information of the frame (step S169), and the three correlation information of the previous three frames with respect to the time t of the target frame are combined. The combined correlation information that gives the minimum value of the minimum values E ₃₀ (step S171) of the combined correlation information E (p, q) is obtained as the maximum combined correlation information.

Here, in order to obtain the combined correlation information that has become the maximum combined correlation information, the sum of the correlation information of the k frame temporally subsequent to the time t of the target frame and the temporally previous h frame is k + If the correlation information of h is synthesized (here, k, h = 0,1,2,3), among the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ Can be expressed as E _hk .

In step S172, the minimum value detecting unit 125 further minimizes the “value” of the maximum combined correlation information, that is, the position in the spatial direction that gives the minimum value E _hk of the maximum combined correlation information (p _hk , q _hk ), together with the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ , the stored positions (p ₃₃ , q ₃₃ ), (p ₂₃ , q ₂₃ ), ( p ₁₃ , q ₁₃ ), (p ₀₃ , q ₀₃ ), (p ₃₂ , q ₃₂ ), (p ₃₁ , q ₃₁ ), (p ₃₀ , q ₃₀ ) Let it be the position (p ₀ , q ₀ ).

Then, the minimum value detection unit 125 sets the maximum correlation position (p ₀ , q ₀ ) as the average motion vector (p ₀ , q ₀ ), and uses the original motion picture data as the average motion vector (p ₀ , q ₀ ). The three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) obtained by adding the frame period t ₀ of the current as a component in the time direction t is obtained, and the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) Assuming that the direction orthogonal to the direction is the principal component direction, a three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) is supplied to the filter information supply unit 32 (FIG. 25), and the maximum combined correlation information The minimum value E _hk is supplied to the filter information supply unit 32. Further, the minimum value detection unit 125 uses (h, k), which is the suffix of the minimum value E _hk of the maximum combined correlation information, and the range of frames used for calculating the correlation information combined to obtain the maximum combined correlation information. Is supplied to the filter information supply unit 32, and the process proceeds from step S172 to S173.

Here, the offset values L ₃₃ (step S158), L ₂₃ (step S160), L ₁₃ (step S162), L ₀₃ (step S164), L ₃₂ (step S166), L ₃₁ (step S168), L described above. ₃₀ (step S170), the formula _{L 33 ≦ L 32 ≦ L 31} ≦ L 30, and a small constant satisfying the formula _{L 33 ≦ L 23 ≦ L 13} ≦ L 03.

In step S173, the filter information supply unit 32 (FIG. 25) determines that the minimum value E _hk of the maximum combined correlation information from the principal component direction acquisition unit 31 (its minimum value detection unit 125) in FIG. It is determined whether it is (less than). Here, the threshold value ε used in step S173 is larger than the offset values L ₃₃ , L ₂₃ , L ₁₃ , L ₀₃ , L ₃₂ , L ₃₁ , and L _{30 and} can be determined by, for example, simulation. . Further, the threshold value ε can be set according to a user operation.

If it is determined in step S173 that the minimum value E _hk of the maximum combined correlation information is equal to or smaller than the predetermined threshold ε, that is, the minimum value E _hk of the maximum combined correlation information is small to some extent, and therefore corresponds to the block of interest. When the region exists in the frame before or after time t, the process proceeds to step S174, and the filter information supply unit 32 performs the principal component from the principal component direction acquisition unit 31 from the origin (0, 0) in the frequency domain. A region extending in the direction, having a width of 2π / (4 × t ₀ ) in the T direction, and the X and Y directions being − (π / r ₀ ) to + (π / r ₀ ), and the T direction Are in the range of − (π / t ₀ ) to + (π / t ₀ ), that is, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region in FIG. R ₁₃₀₁ is determined as the pass band of the filter, and the minimum value detection of the principal component direction acquisition unit 31 in FIG. (H, k) from the unit 125) is directly determined as the kernel size, and the process proceeds to step S176.

On the other hand, if it is determined in step S173 that the minimum value E _hk of the maximum combined correlation information is not less than or equal to the predetermined threshold ε, that is, the minimum value E _hk of the maximum combined correlation information is large, and therefore corresponds to the target block. When the region does not exist in any of the frames before and after time t, the process proceeds to step S175, and the filter information supply unit 32 starts from the origin (0, 0) in the frequency domain as in step S174. , A region extending in the principal component direction from the principal component direction acquisition unit 31 and having a width of 2π / (4 × t ₀ ) in the T direction, and the X and Y directions being − (π / r ₀ ) to A region where + (π / r ₀ ) and the T direction is in the range of − (π / t ₀ ) to + (π / t ₀ ) is determined as the passband of the filter. Further, in step S175, the filter information supply unit 32 replaces (h, k) from the principal component direction acquisition unit 31 (the minimum value detection unit 125 thereof) in FIG. 72 with (h, k) = (0, 0) is determined as the kernel size, and the process proceeds to step S176.

In step S176, the filter information supply unit 32 uses, for example, a three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) as filter information indicating the pass band of the filter determined in step S174 or S175, and the filter unit 22. (FIG. 25) and the kernel size (h, k) are supplied to the filter unit 22 and the process is terminated.

According to the processing according to the flowcharts shown in FIG. 73 to FIG. 77, the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) as the following filter information and the kernel size (h , k).

For example, in the moving image data shown in FIG. 70, when the block B ₃ is the target block, as described above, the times t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t−2t at _0, respectively image data in t-3t _0, block of interest B ₃ with the corresponding region _{B 101, B 201, B 301} , B -101, B -201, B -301 is present, the time t + t ₀ Correlation information E ₁ (u ₁ , v ₁ ), correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ , correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ , time tt correlation information in ₀ F ₁ of (r _1, s _1), the correlation information F ₂ at time _{t-2t 0 (r 2,} s 2), the correlation information F ₃ at time _{t-3t 0 (r 3,} s 3) The values for all are (u ₂ , v ₂ ), (u ₃ , v ₃ ), (r ₁ , s ₁ ), (r ₂ , s ₂ ), (r ₃ , s ₃ ) are (0, 0) is the smallest and almost zero.

In this case, the position (p ₃₃ , q ₃₃ ) that gives the minimum value E ₃₃ of the combined correlation function E (p, q) obtained in step S159 of FIG. 74 is (0,0), and the minimum value E ₃₃ is approximately equal to the offset value L _33. Similarly, the position (p ₂₃ , q ₂₃ ) that gives the minimum value E ₂₃ of the combined correlation function E (p, q) obtained in step S161 in FIG. 75 is (0,0), and the minimum value E ₂₃ is approximately equal to the offset value L _23. In addition, the position (p ₁₃ , q ₁₃ ) that gives the minimum value E ₁₃ of the combined correlation function E (p, q) obtained in step S163 in FIG. 75 is (0,0), and the minimum value E ₁₃ is almost equal to equal to the offset value L _13. Further, the position (p ₀₃ , q ₀₃ ) that gives the minimum value E ₀₃ of the combined correlation function E (p, q), which is obtained in step S165 of FIG. 75, is (0, 0), and the minimum value E ₀₃ is almost equal to equal to the offset value L _03. In addition, the position (p ₃₂ , q ₃₂ ) that gives the minimum value E ₃₂ of the combined correlation function E (p, q) obtained in step S167 of FIG. 76 is (0,0), and the minimum value E ₃₂ is almost equal to equal to the offset value L _32. Further, the position (p ₃₁ , q ₃₁ ) that gives the minimum value E ₃₁ of the combined correlation function E (p, q) obtained in step S169 in FIG. 76 is (0,0), and the minimum value E ₃₁ is almost equal to equal to the offset value L _31. In addition, the position (p ₃₀ , q ₃₀ ) that gives the minimum value E ₃₀ of the combined correlation function E (p, q) obtained in step S171 in FIG. 76 is (0, 0), and the minimum value E ₃₀ is almost equal to equal to the offset value L _30.

As described above, the offset values L ₃₃ , L ₂₃ , L ₁₃ , L ₀₃ , L ₃₂ , L ₃₁ , and L ₃₀ are the expressions L ₃₃ ≦ L ₃₂ ≦ L ₃₁ ≦ L ₃₀ and the expressions L ₃₃ ≦ L _23. Since ≦ L ₁₃ ≦ L ₀₃ is satisfied, for the target block B ₃ , the minimum value E _{33 that} is substantially the offset value L ₃₃ is basically the minimum value E ₃₃ , E ₂₃ , E in step S172 of FIG. ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ are detected as the smallest one, and ( ₀ , 0) is obtained as the average motion vector (p ₀ , q ₀ ). Further, in step S172, (0,0, t ₀ ) is obtained as the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and (3,3) is obtained as the kernel size (h, k). .

Further, the offset value L _33, as described above, since the smaller the threshold ε in step S173 (or less is), the minimum value E ₃₃ of substantially the offset value L ₃₃ is the minimum value E _33, E _23, E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ regarding the target block B ₃ detected as the smallest one in step S 174 of FIG. 77, the three-dimensional motion vector (p ₀ , q _0, t ₀₎ as (0, 0, with t ₀₎ is obtained, the kernel size (h, k) as the (3,3) is obtained.

This time t-ht ₀ from t + kt _0, that is, between time t-3t ₀ of t + 3t _0, the block of interest B ₃ is (0,0, t ₀₎ of the speed (time t ₀ It is shown that it is moving at a speed (moving by (0,0)) in the spatial direction.

Accordingly, the kernel size (h, k) is determined from the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and thus the target block obtained from the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ). It can be said that it represents the effective range of the principal component direction.

Then, for example, in the moving image data shown in FIG. 70, when the block B ₁₀ is the target block, as described above, the time _{t + t 0, t + 2t} 0, t + 3t 0, tt 0, in each image data in the _{t-2t 0, t-3t} 0, region B ₁₀₂ corresponding to a target block _{B 3, B 202, B 302} , B -102, B -202, is B _-302 exist, time t + The _value of the correlation information E ₁ (u ₁ , v ₁ ) at t ₀ is the correlation information E ₂ (u ₂ , u at time t + 2t ₀ when the position (u ₁ , v ₁ ) is (g, 0). v is the value of _2), the value of the position (u _2, v ₂₎ is (2 g, 0 when), correlation information at time _{t + 3t 0 E 3 (u} 3, v 3) , the position (u ₃ , v ₃ ) is (3g, 0), the _value of correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ is when position (r ₁ , s ₁ ) is (-g, 0) Further, the _value of the correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ is the correlation information at time t-3t ₀ when the position (r ₂ , s ₂ ) is (-2g, 0). the value of _{F 3 (r 3, s 3} ) , the position _{(r 3, s 3),} - when the (3 g, 0) , Each becomes a minimum, it is substantially zero.

In this case, the position (p ₃₃ , q ₃₃ ) giving the minimum value E ₃₃ of the combined correlation function E (p, q) obtained in step S159 of FIG. 74 is (g, 0), and the minimum value E ₃₃ is approximately equal to the offset value L _33. Similarly, the position (p ₂₃ , q ₂₃ ) that gives the minimum value E ₂₃ of the combined correlation function E (p, q) obtained in step S161 in FIG. 75 is (g, 0), and the minimum value E ₂₃ is approximately equal to the offset value L _23. In addition, the position (p ₁₃ , q ₁₃ ) that gives the minimum value E ₁₃ of the combined correlation function E (p, q) obtained in step S163 in FIG. 75 is (g, 0), and the minimum value E ₁₃ is almost equal to equal to the offset value L _13. Furthermore, the position (p ₀₃ , q ₀₃ ) that gives the minimum value E ₀₃ of the combined correlation function E (p, q), which is obtained in step S165 of FIG. 75, is (g, 0), and the minimum value E ₀₃ is almost equal to the offset value L _03. In addition, the position (p ₃₂ , q ₃₂ ) that gives the minimum value E ₃₂ of the combined correlation function E (p, q) obtained in step S167 of FIG. 76 is (g, 0), and the minimum value E ₃₂ is almost equal to equal to the offset value L _32. Further, the position (p ₃₁ , q ₃₁ ) that gives the minimum value E ₃₁ of the combined correlation function E (p, q) obtained in step S169 in FIG. 76 is (g, 0), and the minimum value E ₃₁ is almost equal to equal to the offset value L _31. In addition, the position (p ₃₀ , q ₃₀ ) that gives the minimum value E ₃₀ of the combined correlation function E (p, q) obtained in step S171 in FIG. 76 is (g, 0), and the minimum value E ₃₀ is almost equal to equal to the offset value L _30.

As described above, the offset values L ₃₃ , L ₂₃ , L ₁₃ , L ₀₃ , L ₃₂ , L ₃₁ , and L ₃₀ are the expressions L ₃₃ ≦ L ₃₂ ≦ L ₃₁ ≦ L ₃₀ and the expressions L ₃₃ ≦ L _23. since satisfy ≦ L ₁₃ ≦ L _03, for the block of interest B _10, in step S172 of FIG. 77, basically, the minimum value E _33, which is substantially offset value L ₃₃ is the minimum value E _33, E _23, E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ are detected as the smallest one, and (g, 0) is obtained as the average motion vector (p ₀ , q ₀ ). In step S172, (g, 0, t ₀ ) is obtained as a three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and (3, 3) is obtained as a kernel size (h, k). .

Further, as described above, since the offset value L ₃₃ is smaller than the threshold value ε in step S173, the minimum value E _{33 that} is substantially the offset value L ₃₃ becomes the minimum value E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E _With respect to the target block B ₁₀ detected as the smallest of ₃₂ , E ₃₁ , and E ₃₀ , in step S174 in FIG. 77, the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) serving as filter information is obtained. (G, 0, t ₀ ) is obtained, and (3, 3) is obtained as the kernel size (h, k).

This time t-ht ₀ from t + kt _0, that is, between time t-3t ₀ of t + 3t _0, the block of interest B ₁₀ is (g, 0, t ₀₎ of the speed (time t ₀ It is shown that it is moving at a speed (moving by (g, 0)) in the space direction.

Next, for example, in the moving image data shown in FIG. 71, when the block B ₇ is the target block, as described above, the times t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , in each image data in the t-2t _0, region B _111, B ₂₁₁ corresponding to a target block _{B 7, B 311, B -111} , and B _-211 is present, the correlation information at time t + t ₀ E ₁ (u ₁ , v ₁ ) is the position (u ₁ , v ₁ ) is (0,0) and the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is the position ( When u ₂ , v ₂ ) is (0,0), the _value of the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is the position (u ₃ , v ₃ ) at (0,0 ), The _value of the correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ is the correlation information at time t-2t ₀ when the position (r ₁ , s ₁ ) is (0,0). The value of F ₂ (r ₂ , s ₂ ) is minimized when the position (r ₂ , s ₂ ) is (0, 0), and is almost zero.

However, in the image data at time t-3t _0, a region corresponding to the block of interest B _7, which may originally, but it should be region B _-311 at the same position as the block of interest B _7, a moving subject P ₆₉₀₂ It is hidden behind and does not exist. Therefore, the _value of the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ is minimized when the position (r ₃ , s ₃ ) is (0,0). The value of the correlation information F ₃ (r ₃ , s ₃ ) is not always the minimum when the position (r ₃ , s ₃ ) is (0,0). That is, the value of the correlation information F ₃ (r ₃ , s ₃ ) is minimum at a certain position (r ₃ , s ₃ ) that is not necessarily (0, 0). Become.

In this case, the minimum value E _{33 of the} combined correlation function E (p, q) obtained using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ in step S158 in FIG. The position (p ₃₃ , q ₃₃ ) that gives is a position that is not necessarily (0, 0), and the minimum value E ₃₃ is considerably larger than the offset value L ₃₃ . Similarly, the minimum value E _{32 of the} combined correlation function E (p, q) obtained using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ in step S166 of FIG. The position (p ₃₂ , q ₃₂ ) that gives is a position that is not necessarily (0, 0), and the minimum value E ₃₂ is considerably larger than the offset value L ₃₂ . Further, in step S168 of FIG. 76, the minimum value E ₃₁ is also set for the combined correlation function E (p, q) obtained using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ as well. The given position (p ₃₁ , q ₃₁ ) is a position that is not necessarily (0, 0), and the minimum value E ₃₁ is considerably larger than the offset value L ₃₁ . Further, in step S170 of FIG. 76, the time t-3t ₀ correlation information F ₃ in (r _3, s ₃₎ is also a synthetic correlation function E obtained (p, q) for even the minimum value E ₃₀ The given position (p ₃₀ , q ₃₀ ) is a certain position that is not necessarily (0, 0), and the minimum value E ₃₀ is considerably larger than the offset value L ₃₀ .

On the other hand, in step S160 of FIG. 75, for a time t-3t correlation in ₀ information _{F 3 (r 3, s 3} ) determined without using the synthesized correlation function E (p, q), the minimum value E ₂₃ The given position (p ₂₃ , q ₂₃ ) is (0, 0), and the minimum value E ₂₃ is substantially equal to the offset value L ₂₃ . Similarly, the minimum value E _{13 of the} combined correlation function E (p, q) obtained without using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ in step S162 in FIG. The position (p ₁₃ , q ₁₃ ) that gives is (0, 0), and the minimum value E ₁₃ is substantially equal to the offset value L ₁₃ . Further, in step S164 of FIG. 75, the minimum value _E03 is also set for the combined correlation function E (p, q) obtained without using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ . The given position (p ₀₃ , q ₀₃ ) is (0, 0), and the minimum value E ₀₃ is substantially equal to the offset value L ₀₃ .

Since the offset values L ₂₃ , L ₁₃ , and L ₀₃ satisfy the formula L ₂₃ ≦ L ₁₃ ≦ L ₀₃ from the above description, the target block B ₇ is basically substantially offset in step S172 of FIG. The minimum value E ₂₃ having the value L ₂₃ is detected as the minimum of the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ and the average motion vector (p ₀ , (0,0) is obtained as q ₀ ). Further, in step S172, (0,0, t ₀ ) is obtained as the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and (2,3) is obtained as the kernel size (h, k). .

Further, the offset value L _23, as described above, is smaller than the threshold value ε in step S173, the minimum value E _23, which is substantially offset value L ₂₃ is the minimum value _{E 33, E 23, E 13} , E 03, E _With respect to the target block B ₇ detected as the smallest of ₃₂ , E ₃₁ , and E _30, the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) that becomes the filter information in step S174 of FIG. (0,0, t ₀ ) is obtained, and (2,3) is obtained as the kernel size (h, k).

This is because the target block B ₇ moves at a speed of (0,0, t ₀ ) from time t-ht ₀ to t + kt ₀ , that is, from time t-2t ₀ to t + 3t _0. It shows that there is.

Then, for example, in the moving image data shown in FIG. 71, when the block B ₁₄ is the target block, as described above, at time _{t + t 0, tt 0,} t-2t 0, t-3t 0 In each image data, there are regions B ₁₁₂ , B ₋₁₁₂ , B ₋₂₁₂ , and B ₋₃₁₂ corresponding to the block of interest B _14, and the _value of the correlation information E ₁ (u ₁ , v ₁ ) at time t + t ₀ When the position (u ₁ , v ₁ ) is (0,0), the _value of the correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ is the position (r ₁ , s ₁ ) is (0 , 0), the _value of the correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ is the time t-3t when the position (r ₂ , s ₂ ) is (0,0). _{The value} of the correlation information F ₃ (r ₃ , s ₃ ) at 0 is minimum and almost 0 when the position (r ₃ , s ₃ ) is (0,0).

However, in the image data at time t + 2t _0, a region corresponding to the block of interest B ₁₄ is originally, but it should be region B ₂₁₂ of the block of interest B ₁₄ at the same position, moving the subject P ₆₉₀₂ are It is hidden behind and does not exist. For this reason, the _value of the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is minimized when the position (u ₂ , v ₂ ) is (0,0). The value of the correlation information E ₂ (u ₂ , v ₂ ) is not always the minimum when the position (u ₂ , v ₂ ) is (0,0). That is, the value of the correlation information E ₂ (u ₂ , v ₂ ) is minimum at a certain position (u ₂ , v ₂ ) that is not necessarily (0,0). Become.

Further, in the image data at time t + 3t _0, a region corresponding to the block of interest B ₁₄ is originally but should a region B ₃₁₂ of the block of interest B ₁₄ at the same position, moving the subject P ₆₉₀₂ are It is hidden behind and does not exist. Therefore, the _value of the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is minimized when the position (u ₃ , v ₃ ) is (0,0). The value of the correlation information E ₃ (u ₃ , v ₃ ) is not always the minimum when the position (u ₃ , v ₃ ) is (0,0). That is, the value of the correlation information E ₃ (u ₃ , v ₃ ) is minimum at a certain position (u ₃ , v ₃ ) that is not necessarily (0,0). Become.

In this case, in step S158 of FIG. 74, the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are also obtained. For the resultant correlation function E (p, q), the position (p ₃₃ , q ₃₃ ) giving the minimum value E ₃₃ is not necessarily (0,0), and the minimum value E ₃₃ is offset. It becomes considerably to a value greater than the value L _33. Similarly, in step S160 in FIG. 75, the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are also used. The position (p ₂₃ , q ₂₃ ) that gives the minimum value E ₂₃ is also a position that is not necessarily (0,0), and the minimum value E ₂₃ is an offset value. It becomes a much larger value than L _23. 75, the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are also obtained. Also for the composite correlation function E (p, q), the position (p ₁₃ , q ₁₃ ) giving the minimum value E ₁₃ is not always (0,0), and the minimum value E ₁₃ is an offset value. It becomes a much larger value than L _13. Further, in step S164 of FIG. 75, the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are also obtained. Also for the composite correlation function E (p, q), the position (p ₀₃ , q ₀₃ ) giving the minimum value E ₀₃ is not always (0,0), and the minimum value E ₀₃ is an offset value. It is considerably larger than L ₀₃ .

In step S166 of FIG. 76, the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is not used, but the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is used. The position (p ₃₂ , q ₃₂ ) that gives the minimum value E ₃₂ of the composite correlation function E (p, q) obtained by using the position is not always (0,0), and the minimum value E ₃₂ becomes considerably larger than the offset value L _32.

On the other hand, in step S168 in FIG. 76, neither correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ nor correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is used. The position (p ₃₁ , q ₃₁ ) giving the minimum value E ₃₁ is (0,0), and the minimum value E ₃₁ is almost equal to the offset value L ₃₁ . Will be equal. Similarly, in step S170 of FIG. 76, both correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are used. The position (p ₃₀ , q ₃₀ ) that gives the minimum value E ₃₀ is (0, 0), and the minimum value E ₃₀ is almost equal to the offset value L _30. Is equal to

Since the offset values L ₃₁ and L ₃₀ satisfy the expression L ₃₁ ≦ L ₃₀ from the above description, the block B ₁₄ of interest is basically the minimum that substantially becomes the offset value L ₃₁ in step S172 of FIG. The value E ₃₁ is detected as the smallest of the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ , and ( ₀ as the average motion vector (p ₀ , q ₀ )) , 0) is required. Further, in step S172, (0,0, t ₀ ) is obtained as the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and (3,1) is obtained as the kernel size (h, k). .

Further, the offset value L _31, as described above, since less than the threshold value ε in step S173, the minimum value E _31, which is substantially offset value L ₃₁ is the minimum value _{E 33, E 23, E 13} , E 03, E For the target block B ₁₄ detected as the smallest of ₃₂ , E ₃₁ , and E ₃₀ , in step S174 of FIG. 77, the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) serving as filter information is obtained. (0,0, t ₀ ) is obtained, and (3,1) is obtained as the kernel size (h, k).

This is because the target block B ₁₄ moves at a speed of (0,0, t ₀ ) from time t-ht ₀ to t + kt ₀ , that is, from time t-3t ₀ to t + t _0. It shows that there is.

Next, for example, in the moving image data shown in FIG. 71, when the block B ₉ is the target block, as described above, the target block B ₉ moves at a position where the subject P ₆₉₀₁ is stationary. Since the stationary subject P ₆₉₀₂ hides the stationary subject P ₆₉₀₁ , the time t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t−2t ₀ , t− There is no region corresponding to the target block B _{9 in} any of the image data at 3t ₀ .

Therefore, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , t at time t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) The position (u ₁ , v ₁ ), (u ₂ ) of the image data at t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t _{0 that} is most similar to the target block B ₉ , v ₂ ), (u ₃ , v ₃ ), (r ₁ , s ₁ ), (r ₂ , s ₂ ), (r ₃ , s ₃ ) are the smallest, but the value is somewhat large Value.

In this case, for the combined correlation function E (p, q) obtained in step S158 of FIG. 74, the position (p ₃₃ , q ₃₃ ) giving the minimum value E ₃₃ is a certain position, and the minimum value E ₃₃ is It becomes considerably larger than the offset value L _33. Similarly, for the combined correlation function E (p, q) obtained in step S160 of FIG. 75, the position (p ₂₃ , q ₂₃ ) giving the minimum value E ₂₃ is a certain position, and the minimum value E ₂₃ is an offset. It becomes a much larger value than the value L _23. Also, with respect to the composite correlation function E (p, q) obtained in step S162 in FIG. 75, the position (p ₁₃ , q ₁₃ ) giving the minimum value E ₁₃ is a certain position, and the minimum value E ₁₃ is the offset value. It becomes a much larger value than L _13. Further, with respect to the composite correlation function E (p, q) obtained in step S164 of FIG. 75, the position (p ₀₃ , q ₀₃ ) giving the minimum value E ₀₃ is a certain position, and the minimum value E ₀₃ is an offset value. It is considerably larger than L ₀₃ . Also, for the combined correlation function E (p, q) obtained in step S166 of FIG. 76, the position (p ₃₂ , q ₃₂ ) giving the minimum value E ₃₂ is a certain position, and the minimum value E ₃₂ is the offset value. It becomes considerably to a value greater than L _32. Further, with respect to the composite correlation function E (p, q) obtained in step S168 in FIG. 76, the position (p ₃₁ , q ₃₁ ) that gives the minimum value E ₃₁ is a certain position, and the minimum value E ₃₁ Much larger than L ₃₁ . Also, with respect to the composite correlation function E (p, q) obtained in step S170 of FIG. 76, the position (p ₃₀ , q ₃₀ ) that gives the minimum value E ₃₀ is a certain position, and the minimum value E ₃₀ It becomes considerably to a value greater than L _30.

Accordingly, the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , and E ₃₀ are all large values, and any of them is selected as the minimum in step S172 of FIG. even if they are, larger than the threshold ε in step S173 (it becomes higher), for the block of interest B _9, in step S175 of FIG. 77, three-dimensional motion vectors, which are filter information _{(p 0, q 0, t} 0 ) Is obtained as (p ₀ ′, q ₀ ′, t ₀ ), and (0,0) is obtained as the kernel size (h, k).

This means that the target block B ₉ is moving at a speed of (p ₀ ′, q ₀ ′, t ₀ ) only from time t-ht ₀ to t + kt ₀ , that is, at the timing of time t. Is shown. In other words, the kernel size (h, k) being (0, 0) means that the area corresponding to the target block B ₉ does not exist in the image data at any other time.

Note that (p ₀ ', q ₀ ') is a position (p, q) that gives the smallest one of the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E _30. To express. Further, (p, q) when the kernel size (h, k) is (0,0) is dummy data and is not used.

As described above, according to the processing according to the flowcharts shown in FIGS. 73 to 77, the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) as the filter information and the kernel size ( h, k) is obtained, and as a result, for the block of interest, the block of interest is moving at a speed of (p ₀ , q ₀ , t ₀ ) from time t-ht ₀ to t + kt ₀ Can be obtained.

The offset values L ₃₃ , L ₂₃ , L ₁₃ , L ₀₃ , L ₃₂ , L ₃₁ , and L ₃₀ are the expressions L ₃₃ ≦ L ₃₂ ≦ L ₃₁ ≦ L ₃₀ and the expressions L ₃₃ ≦ L ₂₃ ≦ L _13. By satisfying ≦ L ₀₃ , basically, in step S 172 of FIG. 77, h or k of the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ A large minimum value E _hk is easily selected. As a result, even if there is a slight difference in pixel values between the block of interest and the region corresponding to the block of interest in another frame, the block of interest will be (p ₀ , q ₀ , t ₀ ) can be prevented from underestimating the range (time) of moving at a speed of (t ₀ ).

Next, the filter generation unit 23 in which the principal component direction acquisition unit 31 of FIG. 25 is configured as shown in FIG. 72 performs the processing according to FIGS. When the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) and the kernel size (h, k) as filter information are supplied from the information supply unit 32) to the filter unit 22, the filter unit 22 Then, the process according to the flowchart shown in FIG. 78 is performed as the process of step S3 of FIG.

That is, in step S191, the filter unit 22 receives the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) and the kernel size (h, k) supplied as filter information from the filter information supply unit 32. The process proceeds to step S192.

In step S192, the filter unit 22 calculates the pixel value C (t, x, y) of the pixel at the position (x, y) in the target block of the frame at time t, for example, from time t-3t _{0 to} t + 3t. pixel values of the pixels in the _{0 D (t-3t 0,} x-3p 0, y-3q 0), D (t-2t 0, x-2p 0, y-2q 0), D (tt 0, xp 0, yq ₀ ), D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), D Of (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), filtering is performed by calculation using pixel values in a time (frame) range corresponding to the kernel size (h, k). Note that D (t, x, y) represents a pixel value at a position (x, y) at time t of moving image data used for filtering.

That is, in step S192, the filter unit 22 performs the filtering, that is, the calculation of the pixel value C (t, x, y) of the pixel at the position (x, y) in the target block according to the kernel size (h, k). represented by, carried out using only the pixel values of frames within the effective range of the motion vector _{(p 0, q 0, t} 0).

Specifically, in step S192, when the kernel size (h, k) is (3, 3), the filter unit 22 determines the pixel value D (t−t) of the pixels at times t−3t _{0 to} t + 3t ₀ . 3t ₀ , x-3p ₀ , y-3q ₀ ), D (t-2t ₀ , x-2p ₀ , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ), D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), for example, the pixel value C (t, x, y) of the block of interest is calculated according to equation (1). In this case, in the same manner as in step S132 of FIG. 68, filtering is performed using a filter that extends in the principal component direction of the block of interest and has a region with a width in the T direction of 2π / (4t ₀ ).

・・・（１）

... (1)

In addition, when the kernel size (h, k) is (2,3), the filter unit 22 has a pixel value D (t-3t ₀ , x-3p ₀ , y-3q ₀ ) at time t-3t ₀ . ) Without using the pixel values D (t-2t ₀ , x-2p ₀ , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ) at times t-2t _{0 to} t + 3t ₀ , D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), for example, the pixel value C (t, x, y) of the block of interest is calculated according to equation (2).

・・・（２）

... (2)

Further, when the kernel size (h, k) is (1,3), the filter unit 22 outputs the pixel value D (t-3t ₀ , x-3p ₀ , y-3q ₀ ) of the pixel at time t-3t ₀ . ), and time t-2t pixel value of the pixel at _{0 D (t-2t 0,} x-2p 0, y-2q 0) is not used, the time tt ₀ to t + 3t pixel value of the pixel at ₀ D ( tt ₀ , xp ₀ , yq ₀ ), D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), for example, according to equation (3), the pixel value C (t, x, y) of the block of interest is calculated Do.

・・・（３）

... (3)

In addition, when the kernel size (h, k) is (0,3), the filter unit 22 has a pixel value D (t-3t ₀ , x-3p ₀ , y-3q ₀ ) at time t-3t ₀ . ), Pixel value D (t-2t ₀ , x-2p ₀ , y-2q ₀ ) of the pixel at time t-2t ₀ , and pixel value D (tt ₀ , xp ₀ , yq ₀ ) of the pixel at time tt ₀ Is not used, and the pixel values D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ ) at times t to t + 3t ₀ are used. , x + 2p ₀ , y + 2q ₀ ), D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), for example, according to equation (4), the pixel value C (t, Calculate x, y).

・・・（４）

... (4)

Similarly, when the kernel size (h, k) is (3, 2), the filter unit 22 uses the pixel value D (t + 3t ₀ , x + 3p ₀ , y + 3q) at the time t + 3t ₀ . ₀ ) is not used, and pixel values D (t-3t ₀ , x-3p ₀ , y-3q ₀ ), D (t-2t ₀ , x-2p ₀ ) at times t-3t _{0 to} t + 2t _{0 0} , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ), D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), for example, the pixel value C (t, x, y) of the block of interest is calculated according to equation (5).

・・・（５）

... (5)

Further, when the kernel size (h, k) is (3, 1), the filter unit 22 uses the pixel value D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ) at time t + 3t ₀ . ), And the pixel value D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) of the pixel at time t + 2t ₀ , and the pixel value of the pixel at time t-3t _{0 to} t + t ₀ D (t-3t ₀ , x-3p ₀ , y-3q ₀ ), D (t-2t ₀ , x-2p ₀ , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ), D ( t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), for example, according to equation (6), the pixel value C (t, x, y) of the block of interest Perform the operation.

・・・（６）

... (6)

In addition, when the kernel size (h, k) is (3,0), the filter unit 22 has a pixel value D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ) at time t + 3t ₀ . ), The pixel value D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) of the pixel at time t + 2t ₀ , and the pixel value D (t + t ₀ , x + of the pixel at time t + t ₀ p _0, y + q ₀₎ without the pixel value D of the pixel at time t-3t ₀ to _{t (t-3t 0, x} -3p 0, y-3q 0), D (t-2t 0, x-2p ₀ , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ), D (t, x, y), for example, according to the equation (7), the pixel value C ( t, x, y) is calculated.

・・・（７）

... (7)

On the other hand, when the kernel size (h, k) is (0,0), the same area (image area) as the target block (area corresponding to the target block) is set to other times t-3t ₀ , t- 2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , and t + 3t ₀ are not present in this case. In this case, temporal filtering results in degradation of image quality. (Frame of interest), that is, using only the pixel value D (t, x, y) of the pixel at time t, for example, the pixel value C (t, x, y) of the block of interest is calculated according to equation (8). Do. Equation (8) is equivalent to not performing filtering.

・・・（８）

... (8)

Here, as a case where the region corresponding to the block of interest does not exist in the images at other times t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t ₀ , For example, when the moving speed of the subject projected in the block of interest is so high that humans cannot recognize it, or the subject projected in the block of interest is in the frame at time t (frame of interest). There are cases where it only appears suddenly.

Finding the minimum value of correlation information or composite correlation information is equivalent to detecting a motion vector by so-called block matching, but the subject projected in the target block exceeds the search range in block matching. Even if it has moved, the area corresponding to the block of interest is displayed in the images at other times t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t ₀ . Applicable when it does not exist.

  In step S192, the filtering as the calculation of the pixel value C (t, x, y) is performed while down-sampling that thins out the number of samples in the time direction to 1/4, that is, the ratio of one frame every four frames. Done in

  After the process of step S192, the process proceeds to step S193, and the filter unit 22 outputs the pixel value C that is the filtering result, that is, the filtering result in step S192 (the calculation result of any one of the expressions (1) to (8)). (t, x, y) is output to the encoding unit 24 as a pixel value at the position (x, y) at time t.

As described above, in the second motion detection process (the processes in steps S151 to S172 of the processes in steps S151 to S176 in FIGS. 73 to 77), the filter passes as in the first motion detection process. By using multiple pieces of correlation information representing the correlation with each of multiple frames corresponding to the width in the T direction of the band, 2π / (4t ₀ ), the average movement from the target block to multiple frames can be said. The average motion vector that represents the average motion of the target block is calculated because the correlation information is not used for the frame in which the region corresponding to the target block does not exist. Furthermore, an accurate principal component direction can be obtained for the block of interest.

  In the second motion detection process, the kernel size (h, k) is output, and only the pixels of the frame in the range corresponding to the kernel size (h, k) are used for filtering in the filter unit 22. In other words, since the pixel of the frame in which the region corresponding to the block of interest does not exist is not used, appropriate data of only frequency components that can be recognized by human vision can be obtained.

Next, for example, in the transmission device 1 shown in FIG. 25, as described above, filtering is performed on moving image data with a frame rate of 1 / t ₀ so as to leave only necessary information in consideration of human visual characteristics. By performing downsampling, the amount of data can be reduced.

Furthermore, in the receiving apparatus 2 shown in FIG. 33, by performing filtering and upsampling on the moving image data of the frame rate 1 / (4t ₀ ) output from the transmitting apparatus 1, ideally, human beings have image quality. It is possible to obtain moving image data with a frame rate of 1 / t ₀ that does not feel deterioration of the image.

By the way, in the receiving device 2, by up-sampling the moving image data at the frame rate 1 / (4t ₀ ), 3 frames are interpolated between the two frames of the moving image data, whereby the frame rate 1 / t. ₀ video data is obtained. Since image data of a frame to be interpolated by this upsampling (hereinafter referred to as an interpolated frame as appropriate) is obtained only from moving image data at a frame rate of 1 / (4t ₀ ), in reality, some image quality degradation occurs. Sometimes.

Therefore, in the transmission apparatus 1, in addition to the moving image data at the frame rate 1 / (4t ₀ ), the supplemental data that compensates for the deterioration of the image quality of the interpolation frame interpolated between the two frames of the moving image data is obtained. Can be output.

Figure 79 is the video data of the frame rate 1 / t _0, and shows the video data of the frame rate 1 / (4t _0), a configuration example of a transmitting apparatus 1 of FIG. 24 which outputs to obtain the supplemental data.

In the following, the frame period t ₀ is, for example, 1/240 seconds. In this case, the moving image data with the frame rate 1 / t ₀ supplied (input) to the transmission device 1 is moving image data with a frame rate of 240 fps. In addition, moving image data with a frame rate of 240 fps is hereinafter referred to as 240 fps moving image data as appropriate.

  In the transmission device 1 of FIG. 79, 240 fps moving image data is input to the input terminal 211. The input terminal 211 is connected to the band limiting filter unit 212, and the 240 fps moving image data input to the input terminal 211 is supplied to the band limiting filter unit 212.

The band limiting filter unit 212 includes, for example, the buffer unit 21, the filter unit 22, and the filter generation unit 23 illustrated in FIG. In the band limiting filter unit 212, filtering is performed on the 240 fps moving image data from the input terminal 211 so as to leave only necessary information in consideration of human visual characteristics as described above. However, 1/4 downsampling is not performed in the filter unit 22 (FIG. 25) constituting the band limiting filter unit 212. Therefore, the moving image data output from the band limiting filter unit 212 is not the moving image data (60 fps moving image data) of the frame rate 1 / (4t ₀ ) in which the frame rate of the 240 fps moving image data is 1/4, but the 240 fps moving image data. It is.

Note that the principal component direction acquisition unit 31 of the filter generation unit 23 (FIG. 25) constituting the band limiting filter unit 212 can be configured as shown in FIG. 72, for example. In this case, the filter generation unit 23 performs the processing described with reference to FIGS. 73 to 77, and provides the filter unit 22 (FIG. 25) constituting the band-limiting filter unit 212 with a three-dimensional motion vector as filter information. (p ₀ , q ₀ , t ₀ ) and the kernel size (h, k) are output (step S176 in FIG. 77). Then, as described with reference to FIG. 78, filtering according to the kernel size (h, k) is performed in the filter unit 22 constituting the band limiting filter unit 212. However, as described above, downsampling is not performed in the filter unit 22 constituting the band limiting filter unit 212. That is, one frame is not filtered for every four frames of 240 fps moving image data, but filtering is performed for each frame (each pixel) of 240 fps moving image data.

  Here, the processing in the band limiting filter unit 212 is to remove frequency components that cannot be recognized by humans from the 240 fps moving image data from the input terminal 211, so to speak, before the 240 fps moving image data is compressed. It can also be thought of as stage processing (preprocessing). The band limiting filter unit 212 is not an indispensable block in the transmission device 1 of FIG. 79, and therefore the transmission device 1 of FIG. 79 can be configured without providing the band limiting filter unit 212. In this case, the 240 fps moving image data from the input terminal 211 is directly input to the separation circuit 213 provided at the subsequent stage of the band limiting filter unit 212.

  In FIG. 79, the 240 fps moving image data output from the band limiting filter unit 212 is supplied to the subsequent separation circuit 213.

  The separation circuit 213 converts the 240 fps moving image data (first moving image data) from the band limiting filter unit 212 to a frame rate lower than the frame rate of the 240 fps moving image data, for example, 60 fps moving image data (second moving image data). And the remaining video data (third video data) obtained by removing the 60 fps video data from the 240 fps video data. That is, the separation circuit 213 selects one frame for every four frames of 240 fps moving image data from the band limiting filter unit 212, thereby separating 60 fps moving image data from the 240 fps moving image data. That is, the separation circuit 213 obtains 60 fps moving image data similar to that obtained as a result of 1/4 downsampling in the filter unit 22 of FIG. Further, the separation circuit 213 obtains the remaining moving image data obtained by separating the 60 fps moving image data from the 240 fps moving image data from the band limiting filter unit 212.

  Here, the remaining moving image data obtained by separating 60 fps moving image data from 240 fps moving image data is hereinafter referred to as 240-60 fps moving image data as appropriate.

  The 60 fps moving image data obtained in the separation circuit 213 is supplied to the compression circuit 214. The 240-60 fps moving image data obtained in the separation circuit 213 is supplied to the difference information extraction unit 217 as a target image as an image to be compressed.

  The compression circuit 214 encodes (compresses) the 60 fps moving image data supplied from the separation circuit 213 by, for example, a known encoding method such as MPEG, or any other encoding method, and outputs a bit stream obtained as a result. This bit stream is supplied to the decompression circuit 216 and the output terminal 215, whereby a bit stream as a result of encoding (compression) of 60 fps moving image data is output from the output terminal 215. Here, the compression circuit 214 corresponds to the encoding unit 24 in FIG. 25, and therefore the bit stream output from the compression circuit 214 corresponds to encoded data.

  The decompression circuit 216 decodes (decompresses) (decompresses) the bit stream from the compression circuit 214, that is, performs local decoding, and references the 60 fps moving image data obtained as a result of the local decoding in compressing the target image. ) The image is supplied to the difference information extraction unit 217 as an image.

  When the compression circuit 214 performs encoding by an encoding method that performs local decoding in encoding data, such as MPEG encoding, for example, the 60 fps moving image data obtained as a result of local decoding performed in the compression circuit 214 is used as a reference. An image can be supplied from the compression circuit 214 to the difference information extraction unit 217. In this case, the decompression circuit 216 need not be provided.

  The difference information extraction unit 217 compresses the 240-60 fps moving image data, which is the target image from the separation circuit 213, using the 60 fps moving image data, which is the reference image from the decompression circuit 216, and data obtained by the compression (hereinafter, referred to as the following) (Referred to as differentially compressed data as appropriate) is supplied to the output terminal 218. That is, the difference information extraction unit 217 obtains an estimated value of the target image from the reference image, and compresses the target image using the estimated value. Specifically, the difference information extraction unit 217 compresses the target image by taking the difference between the target image and the estimated value. Then, by taking the difference between the target image and its estimated value, the difference compressed data that is data obtained by compressing the target image is supplied from the difference information extraction unit 217 to the output terminal 218. Thereby, the differential compression data as a compression result of 240-60fps moving image data is output from the output terminal 218.

  Here, as described above, in the transmission device 1 of FIG. 79, the bit stream as the result of encoding (compression) of 60 fps moving picture data and the differential compressed data as the compression result of 240-60 fps moving picture data are output terminal 215. And 218 are output independently. The bit stream and the differentially compressed data are separately separately or multiplexed and recorded on the recording medium 11 of FIG. 24 or transmitted via the transmission medium 12.

  The differentially compressed data output from the output terminal 218 as the compression result of the 240-60 fps moving image data is supplementary data that compensates for the above-described degradation of the image quality of the interpolation frame.

  Next, FIG. 80 shows 240 fps moving image data input to the separation circuit 213, 60 fps moving image data and 240-60 fps moving image data output from the separation circuit 213. In FIG. 80, the horizontal axis represents time.

The top frames f ₁ , f ₂ ,..., F ₁₃ in FIG. 80 show 240 fps moving image data input to the separation circuit 213 in FIG. In 240fps video data, the time interval between the frames f _{i + 1} following the given frame f _i of a 1/240 seconds (i is an integer).

Although the frame continues before the frame f ₁ and after the frame f ₁₃ , the illustration is omitted in FIG.

  The separation circuit 213 of FIG. 79 converts the 240 fps moving image data shown at the top of FIG. 80 into the 60 fps moving image data shown second from the top of FIG. 80 and the 240-60 fps moving image data shown at the bottom of FIG. To separate.

That is, the separation circuit 213 selects, for example, one frame f _{4i + 1} for every four frames f _{4i + 1} , f _{4i + 2} , f _{4i + 3} , and f _{4i + 4} of the 240 fps moving image data. 60 fps moving image data consisting of frames..., F ₁ , f ₅ , f ₉ , f ₁₃ ,.

Further, the separation circuit 213 removes the 60 fps moving image data from the 240 fps moving image data, and the remaining frames shown in the bottom of FIG. 80..., F ₂ , f ₃ , f ₄ , f ₆ , f ₇ , F ₈ , f ₁₀ , f ₁₁ , f ₁₂ ,...

  In 60 fps video data, the time interval between one frame and the next is 1/60 second. Further, 240-60fps moving image data is moving image data with 240-60 = 180 frames per second.

The second 60 fps moving image data from the top of FIG. 80 is supplied as a reference image to the difference information extraction unit 217 of FIG. 79, and the 240-60 fps moving image data shown at the bottom of FIG. 80 is supplied as a target image. Is done. In difference information extraction unit 217, a frame ... of 240-60fps video data which is the target _{image, f 2, f 3, f} 4, f 6, f 7, f 8, f 10, f 11, f 12, · .. (Image data) is compressed using frames of 60 fps moving image data as reference images,..., F ₁ , f ₅ , f ₉ , f ₁₃ ,.

In the compression circuit 214 of FIG. 79, when the 60 fps moving image data from the separation circuit 213 is completely reversibly encoded, the difference between the 60 fps moving image data obtained by the separation circuit 213 and the decompression circuit 216 is obtained. The 60 fps moving image data as the reference image supplied to the information extraction unit 217 matches. However, in the compression circuit 214, when the encoding performed on the 60 fps moving image data from the separation circuit 213 is not completely reversible, the 60 fps moving image data obtained by the separation circuit 213 and the difference information extraction unit 217 from the decompression circuit 216 are obtained. Strictly speaking, there is a difference from the 60 fps moving image data serving as a reference image supplied to. However, since it is not necessary to consider the difference here, in the following, the 60 fps moving image data obtained by the separation circuit 213 and the 60 fps moving image data supplied from the decompression circuit 216 to the difference information extraction unit 217 match. The explanation will be given on the assumption that That is, strictly speaking,..., F ₁ , f ₅ , f ₉ , f ₁₃ ,... (Image data) are encoded (encoded by the compression circuit 214 in FIG. 79) and decoded (decompression circuit 216). The reference image has been subjected to (decoding by). However, here, the reference image is simply the frame of the 60 fps moving image data shown in the second part from the top of FIG. 80..., F ₁ , f ₅ , f ₉ , f ₁₃ ,. Will be described.

  Next, FIG. 81 shows a configuration example of the difference information extraction unit 217 of FIG.

  The difference information extraction unit 217 is roughly composed of a target storage unit 221, a reference storage unit 222, and a data processing unit 223.

  The target storage unit 221 is supplied with 240-60 fps moving image data as a target image from the separation circuit 213 (FIG. 79). The target storage unit 221 temporarily stores 240-60 fps moving image data as a target image from the separation circuit 213.

  Each frame (image data) of the target image stored in the target storage unit 221 is divided into blocks. Each block obtained by the block division is sequentially set as a target block, and the target block (image data thereof) is read from the target storage unit 221 and input to the data processing unit 223 from the input terminal 233. . Note that, for example, 16 × 16 pixels can be adopted as the block size, as in the case described above. In addition, the number of pixels constituting the block may be a plurality of pixels or one pixel.

  The reference storage unit 222 is supplied with 60 fps moving image data as a reference image from the decompression circuit 216 (FIG. 79). The reference storage unit 222 temporarily stores 60 fps moving image data as a reference image from the decompression circuit 216.

  Of the frames of the reference image stored in the reference storage unit 222, the frame immediately before the frame of the target block is read as a past reference image, and the data processing unit is input from the input terminal 231. 223 is input. In addition, a frame (image data) immediately after the frame of the target block among the frames of the reference image stored in the reference storage unit 222 is read as a future reference image, and data is input from the input terminal 232. Input to the processing unit 223.

Here, the target storage unit 221 stores frames f ₂ , f ₃ , f ₄ , f ₆ , f ₇ , f ₈ , f _{10 of} 240-60 fps moving image data, which is the target image shown at the bottom of FIG. , F ₁₁ , f ₁₂ etc. are temporarily stored. Further, the reference storage unit 222 temporarily stores frames f ₁ , f ₅ , f ₉ , f _{13 and the} like of 60 fps moving image data, which is the second reference image from the top in FIG.

For example, when the block of one of the frames f ₂ , f ₃ , and f ₄ of the target image stored in the target storage unit 221 is the target block, the frames f ₁ and f of the reference image _5, of f _9, f _13, the frame f ₁ and f ₅ immediately before and after the frame f _2, f _3, f ₄ of the target image, as a past reference image and Futuresse reference image respectively, the reference storage unit 222 Read from.

Further, for example, when the block of one of the frames f ₆ , f ₇ , and f ₈ of the target image stored in the target storage unit 221 is the target block, the frames f ₁ and f of the reference image Among the frames f ₅ , f ₉ , and f ₁₃ , the frames f ₅ and f ₉ immediately before and after the frames f ₆ , f ₇ , and f ₈ of the target image are used as a past reference image and a feature reference image, respectively, as a reference storage unit 222. Read from.

Further, for example, when the block of one of the frames f ₁₀ , f ₁₁ , and f ₁₂ of the target image stored in the target storage unit 221 is the target block, the frames f ₁ and f of the reference image _5, of f _9, f _13, frame f ₉ and f ₁₃ of immediately before and after the frame f _10, f _11, f ₁₂ of the target image, as a past reference image and Futuresse reference image respectively, the reference storage unit 222 Read from.

  The data processing unit 223 includes input terminals 231 to 233, six difference data calculation units 234 to 239, a selection circuit 240, and an output terminal 241. The target block input from the input terminal 233 is input from the input terminal 231. The compressed past reference image or the feature reference image input from the input terminal 232 is compressed, and the differential compressed data obtained as a result is output from the output terminal 241 to the output terminal 218 of FIG.

  That is, the data processing unit 223 compresses the target image in units of blocks.

  Specifically, in the data processing unit 223, the past reference image input from the input terminal 231 is supplied to four difference data calculation units 234 to 237 among the six difference data calculation units 234 to 239. The feature reference image input from the input terminal 232 is supplied to four difference data calculation units 234 to 236 and 238 among the six difference data calculation units 234 to 239. Further, the target block input from the input terminal 233 is supplied to all of the six difference data calculation units 234 to 239.

  The difference data calculation unit 234 compresses the block of interest from the input terminal 233 using the past reference image from the input terminal 231 and the feature reference image from the input terminal 232, and the first result to be described later is the compression result. Difference data (compressed data) is supplied to the selection circuit 240.

  The difference data calculation unit 235 also compresses the block of interest from the input terminal 233 using the past reference image from the input terminal 231 and the feature reference image from the input terminal 232, and a second result to be described later is obtained as a result of the compression. Difference data (other compressed data) is supplied to the selection circuit 240.

  The difference data calculation unit 236 also compresses the block of interest from the input terminal 233 using the past reference image from the input terminal 231 and the feature reference image from the input terminal 232, and a third result to be described later is obtained as a result of the compression. Are supplied to the selection circuit 240.

  The difference data calculation unit 237 compresses the block of interest from the input terminal 233 using the past reference image from the input terminal 231, and supplies fourth difference data, which will be described later, to the selection circuit 240 as the compression result. .

  The difference data calculation unit 238 compresses the target block from the input terminal 233 using the feature reference image from the input terminal 232, and supplies fifth difference data, which will be described later, to the selection circuit 240 as a compression result. .

  The difference data calculation unit 239 compresses the target block from the input terminal 233, and supplies sixth difference data, which will be described later, as the compression result to the selection circuit 240.

  The selection circuit 240 selects the optimum compression result of the block of interest from the first to sixth difference data supplied from the six difference data calculation units 234 to 239, respectively. Here, of the first to sixth difference data, the one selected by the selection circuit 240 is hereinafter referred to as selection difference data as appropriate.

  The selection circuit 240 adds a case ID (Identification), which will be described later, as identification information for identifying which of the first to sixth difference data the selection difference data is to the selection difference data. Then, it is output from the output terminal 241 as differential compressed data that is the compression result of the block of interest.

  Next, a method of compressing and restoring the target block in each of the difference data calculation units 234 to 239 and first to sixth difference data obtained by compression of the target block will be described.

FIG. 82 shows a target image V _T , a past reference image V _P , and a future reference image V _F (frames thereof).

Now, s is as a variable which takes one of three values 1, 2, 3, the time interval between the target image V _T and Futuresse reference image V _F, when expressed as s / 240 seconds, the target image time interval between V _T and Pasto reference image V _P can be expressed as (4-s) / 240 seconds.

Here, if the target image V _T is, for example, the frame f ₂ of the 240-60 fps moving image data shown at the bottom of FIG. 80, the past reference image V _P and the future reference image V _F are respectively shown in FIG. The frames f ₁ and f ₅ of the 60 fps moving image data shown second from the top of 80 are shown. In this case, the variable s is 3.

Further, if the target image V _T is, for example, the frame f ₃ of the 240-60 fps moving image data shown at the bottom of FIG. 80, the past reference image V _P and the feature reference image V _F are respectively shown in FIG. The frames f ₁ and f ₅ of the 60 fps moving image data shown second from the top, in this case, the variable s is 2.

Further, assuming that the target image V _T is, for example, the frame f ₄ of the 240-60 fps moving image data shown at the bottom of FIG. 80, the past reference image V _P and the feature reference image V _F are respectively shown in FIG. The frames f ₁ and f ₅ of the 60 fps moving image data shown second from the top are shown. In this case, the variable s is 1. Similarly, in other cases, the variable s is one of 1, 2, and 3.

The target image V _T, if the Pasto reference image V _P described above and Futuresse reference image V _F is present, the block of interest of the target image V _T is the Past reference image V _P and Futuresse reference image V _F, so to speak can be compressed as a hint, can be carried out similarly restoration of the compressed result, the Past reference image V _P and Futuresse reference image V _F as a hint.

That is, in the PAST reference image V _P, the position of the target block of the target image V _T (e.g., the position of the upper left of the target block) from the position shifted by vector mv _8201, a region R ₈₂₀₁ of the target block and the same size If it is similar to the target block (image data thereof), the difference between the pixel value (image data) of each pixel of the target block and the pixel value of the corresponding pixel in the region R ₈₂₀₁ is obtained. Since the difference between the pixel values is almost 0, by adopting the difference that is almost 0 as the image data of the block of interest, the data amount of the image data of the block of interest is reduced, that is, the block of interest is compressed. be able to.

Similarly, in Futuresse reference image V _F, when the position of the target block of the target image V _T, the position shifted by vector mv _8202, region R ₈₂₀₂ of the target block and the same size, which are similar to the block of interest Since the difference between the pixel value of each pixel of the target block and the pixel value of the corresponding pixel in the region R ₈₂₀₂ is almost zero, the difference between the pixel values of each pixel is almost zero. By adopting a difference that is almost zero as the image data, it is possible to reduce the amount of image data of the block of interest, that is, to compress the block of interest.

Differential data calculation unit 234 through 239 (FIG. 81) are basically as described above, the block of interest, the method of taking the difference between a past reference image V _P or Futuresse reference image V _F, compressing the block of interest To do.

  Here, in the following, as the 240 fps moving image data, for example, moving image data on which a subject shown in FIG. 83 is projected is considered.

  Note that the horizontal axis in FIG. 83 represents the position x, y in the spatial direction. That is, in FIG. 83, the two spatial directions x and y are represented by one axis in order to avoid making the diagram complicated. The same applies to FIGS. 84 to 93 described later.

In FIG. 83, the upper side shows the waveform of the subject P _{8301 as} the background, and the lower side shows the waveform of the subject P _{8302 as} the foreground.

The subject P ₈₃₀₁ shown at the upper side of FIG. 83 is stationary, and the subject P ₈₃₀₂ shown at the lower side of FIG. 83 is moving in a certain spatial direction.

FIG. 84 shows the target image V _T , past reference image V _P , and feature reference image V _F (frames) onto which the subjects P ₈₃₀₁ and P ₈₃₀₂ shown in FIG. 83 are projected.

In FIG. 84, for example, the subject P ₈₃₀₂ is has moved slowly at a constant speed, the motion vector of the subject P _8302, i.e., per 1/240 second is a frame period in 240fps video data, the subject P ₈₃₀₂ It is assumed that the vector representing the movement amount in the spatial direction x, y is (U, V).

As described above, the time interval between a past reference image V _P and the target image V _T, (4-s) was / 240 seconds, the time interval between the target image V _T and Futuresse reference image V _F, s / 240 seconds.

When the subject P ₈₃₀₂ is slowly moving at a constant speed, the subject P ₈₃₀₂ exists in all of the target image V _T , the past reference image V _P , and the future reference image V _F , and The subject P ₈₃₀₂ on the past reference image V _P is at a position shifted from the subject P ₈₃₀₂ on the target image V _{T by} a vector (-(4-s) U,-(4-s) V). subject P ₈₃₀₂ on the image V _F from the subject P ₈₃₀₂ on a target image V _T, is the position shifted by vector (sU, sV).

Accordingly, in the target image V _T, for example, a certain block B ₈₄₀₁ including only the subject P _8302, when attention as the subject block, in Past reference image V _P, from the target block B ₈₄₀₁ vector (- (4-s) U , The image data of the region R ₈₄₀₁ having the same size as the _target block B _{8401 at} a position shifted _by- (4-s) V) is (ideally) the same as the _target block B ₈₄₀₁ . Also in Futuresse reference image V _F, a vector from the target block B ₈₄₀₁ (sU, sV) of position shifted by the image data of the target block B ₈₄₀₁ same size as the region R ₈₄₀₂ is (ideally) of interest It is the same as block B ₈₄₀₁ .

That is, the image data of Pasto reference image V _P of region R _8401, the block of interest B _8401, and the region R ₈₄₀₂ of Futuresse reference image V _F is made the same (ideally).

  The difference data calculation unit 234 (FIG. 81) obtains first difference data suitable for the case shown in FIG. 84 as described above.

That is, the difference data calculating unit 234 (FIG. 81), for example, each pixel in the region R ₈₄₀₁ Past reference image V _P, the average value of the image data of the area R ₈₄₀₂ of Futuresse reference image V _F (region R ₈₄₀₁ and pixel value, the average value between the pixel value of the corresponding pixel in the region R ₈₄₀₂₎ is calculated, the difference of the average value, as estimated value of the block of interest B _8401, the block of interest B ₈₄₀₁ and its estimated value (The difference between the pixel value of each pixel of the _target block B ₈₄₀₁ and the estimated value of the pixel value of the pixel) is calculated. Since this difference is almost 0, the data amount of the target block B ₈₄₀₁ can be reduced. The difference data calculation unit 234 outputs the difference between the target block B ₈₄₀₁ and its estimated value as the first difference data that is the compression result of the target block B ₈₄₀₁ .

Incidentally, the first difference data, to restore to the original block of interest B ₈₄₀₁ is a motion vector from the target block B ₈₄₀₁ to Pasto reference image V _P vector (- (4-s) U , - (4 and -s) V), the motion vector from the target block B ₈₄₀₁ to Futuresse reference image V _F vector (sU, two motion vectors with sV) is required.

  However, when two motion vectors (-(4-s) U,-(4-s) V) and (sU, sV) are added to the first difference data, the two motion vectors ( -(4-s) U,-(4-s) V) and (sU, sV) increase the amount of data.

Therefore, the difference data calculation unit 234 performs restoration without a motion vector by _obtaining regions R ₈₄₀₁ and R ₈₄₀₂ (to which motion vectors are used) used to obtain the estimated value of the block of interest B ₈₄₀₁ as follows. First differential data that can be obtained is obtained.

That is, in this case, since it is assumed that the subject P ₈₃₀₂ is moving at a constant speed (including the case of speed = 0), the motion vector from the target block _{B 8401 (- (4-s} ) U, - Region R ₈₄₀₁ on past reference image V _P shifted by (4-s) V), region R ₈₄₀₂ on feature reference image V _F shifted by motion vector (sU, sV) from target block B ₈₄₀₁ , and target block As shown in FIG. 84, B ₈₄₀₁ (corresponding pixels) are arranged in a straight line in space-time.

Therefore, the motion vector from the target block _{B 8401 (- (4-s} ) U, - (4-s) V) a region R ₈₄₀₁ on which shifted by PAST reference image V _P, the motion vector from the target block B ₈₄₀₁ (sU , sV), the region R ₈₄₀₂ on the feature reference image V _{F is located} on a straight line passing through the block of interest B ₈₄₀₁ in time and space.

Then, a region R ₈₄₀₁ on Pasto reference image V _P, diffuser image data of a region R ₈₄₀₂ on tea reference image V _F, as described above, (ideally) Since identical, and the area R ₈₄₀₁ correlation (image data) of the R _8402, in time and space, and the region on the Pasto reference image V _P that exists on a straight line passing through the block of interest B _8401, in a correlation with Futuresse reference image V _F on the region And get the highest.

Accordingly, in the past reference image V _P and the future reference image V _F , if the positional relationship of the region having the highest correlation is detected from the region on the space-time straight line passing through the _target block B ₈₄₀₁ , the past relationship image having the highest correlation is detected. a region of the reference image V _P, and the area of Futuresse reference image V _F, the respective regions R ₈₄₀₁ and R _8402.

That is, in the PAST reference image V _P and Futuresse reference image V _F, by detecting the positional relationship of the correlation is the highest region, the motion vector from the target block B ₈₄₀₁ to the area _{R 8401 (- (4-s} ) U, -(4-s) V) and the regions R ₈₄₀₁ and R ₈₄₀₂ can be obtained without the motion vector (sU, sV) from the target block B ₈₄₀₁ to the region R ₈₄₀₂ .

  The difference data calculation unit 234 (FIG. 81) obtains first difference data that can be restored without a motion vector by using the principle as described above.

Specifically, the difference data calculation unit 234 correlates the correlation information e ₁ (representing the correlation between the region Ip of the past reference image V _P and the region If of the future reference image V _F on a space-time straight line passing through the block of interest. U ′, V ′) is obtained, for example, according to equation (9).

・・・（９）

... (9)

Here, in equation (9), (x, y ) represents the position of the pixel of the target block in the target image V _T. Ip (x- (4-s) U ′, y- (4-s) V ′) is a position (x- (4-s) U ′, y- (4-s) in the past reference image V _P. ) Represents the pixel value of the pixel in V ′). Furthermore, If (x + sU ', y + sV') represents the position (x + sU ', y + sV') pixel value of the pixel in the Futuresse reference image V _F. Also, Σ in equation (9) represents summation for all the pixels constituting the block of interest.

Incidentally, the correlation information e ₁ of the formula (9) (U ', V ') is an sum of absolute differences of the pixel values of the region Ip Past reference image V _P, a region If the Futuresse reference image V _F , therefore, the higher the correlation between the region If region Ip and Futuresse reference image V _F of Pasto reference image V _P, the "value" of the correlation information _{e 1 (U ', V'} ) is reduced.

The difference data calculation unit 234 calculates the correlation information e ₁ (U ′, V ′) in Expression (9) for all possible (U ′, V ′). Here, all the possible (U ′, V ′) are, for example, positions (x- (4-s) U ′, y- (4-s) V ′) on the past reference image V _P. and a position (x + sU ', y + sV') If is on Futuresse reference image V _F of (range), (U ', V' ). In addition, the calculation of the correlation information e ₁ (U ′, V ′) in Expression (9) may be performed only for (U ′, V ′) within a predetermined range.

The difference data calculation unit 234 detects the minimum value from the correlation information e ₁ (U ′, V ′) calculated for each value (U ′, V ′), and the correlation information e ₁ (U ′, V ′). gives the minimum value of (U ', V'), and in Past reference image V _P and Futuresse reference image V _F, is detected as a vector (U, V) representing the positional relationship of the correlation is the highest region.

Here, in the past reference image V _P and the feature reference image V _F as described above, the vector (U, V) representing the positional relationship of the region having the highest correlation is appropriately referred to as the positional relationship vector (U, V). ).

Differential data calculation unit 234, the positional relationship vector (U, V) after the detection of, from the image data with its positional relation vector (U, V) Past reference image V _P and Futuresse reference image V _F in the positional relationship indicated by the Then, the estimated value of the target block is obtained.

That is, the difference data calculation unit 234 has an area of the same size as the target block on the past reference image V _P shifted from the target block by a vector (− (4-s) U, − (4-s) V), from the target block vector (sU, sV) shifted on Futuresse reference image V _F was obtains a region of the block of interest and the same size, the image of the region of the Pasto reference image V _P, and Futuresse reference image V _F of the area The average value of the data is obtained as an estimated value of the block of interest.

  Then, the difference data calculation unit 234 calculates the difference between the block of interest and its estimated value, and outputs it as first difference data that is the compression result of the block of interest.

Such first difference data, the motion vector from the target block to Pasto reference image _{V P (- (4-s} ) U, - (4-s) V) and, from the target block to Futuresse reference image V _F Even if there is no motion vector (sU, sV) (even if no motion vector is added to the first difference data), the original block of interest can be restored.

That is, for example, as shown in FIG. 85, the block B ₈₅₀₁ with a top target image V _T as the target block, when restoring the block of interest B ₈₅₀₁ is attached to the block of interest B _8501, formula (9) Correlation information e ₁ (U ′, V ′) is calculated for all possible (U ′, V ′).

Specifically, for example, as shown in FIG. 85, correlation information e ₁ (U ₁ ′, V ₁ ′) is calculated as (U ′, V ′) = (U ₁ ′, V ₁ ′). Further, for example, as shown in FIG. 86, correlation information e ₁ (U ₂ ′, V ₂ ′) is calculated as (U ′, V ′) = (U ₂ ′, V ₂ ′).

Here, in FIG. 85, by setting (U ′, V ′) = (U ₁ ′, V ₁ ′), the past reference image that is a target of calculation of the correlation information e ₁ (U ₁ ′, V ₁ ′). A region Ip of V _P is a region R ₈₅₀₁ in a part of the moving subject P ₈₃₀₂ . Further, in FIG. 85, the correlation information _{e 1 (U 1 ', V} 1') area If the Futuresse reference image V _F as a calculation target of, again, the region R ₈₅₀₂ of the same portion of the subject P ₈₃₀₂ in motion It has become. Since the correlation between the regions R ₈₅₀₁ and R ₈₅₀₂ is high, the value of the correlation information e ₁ (U ₁ ′, V ₁ ′) is small. In FIG. 85, the region R ₈₅₀₁ of the past reference image V _P is shifted from the _target block B ₈₅₀₁ by a vector (− (4-s) U ₁ ′, − (4-s) V ₁ ′). _Yes , the pixel value Ip (x- (4-s) U ₁ ′, y- (4-s) V ₁ ′) of the pixel in the region R ₈₅₀₁ is the correlation information e ₁ (U ₁ ) in Expression (9). ', V ₁ '). Further, the region R ₈₅₀₂ of the future reference image V _F is located at a position shifted from the target block B ₈₅₀₁ by the vector (sU ₁ ′, sV ₁ ′), and the pixel value If (x + sU ₁ in the region R ₈₅₀₂ ', Y + sV ₁ ') is used to calculate the correlation information e ₁ (U ₁ ', V ₁ ') in equation (9).

On the other hand, in FIG. 86, by setting (U ′, V ′) = (U ₂ ′, V ₂ ′), the past reference image V that is the calculation target of the correlation information e ₁ (U ₂ ′, V ₂ ′). _P regions Ip has been in the region R ₈₆₀₁ of the portion of the subject P ₈₃₀₁ which is stationary. Further, in FIG. 86, the correlation information _{e 1 (U 2 ', V} 2') regions If the Futuresse reference image V _F which is an object of calculation of the area R ₈₆₀₂ of the other part of the subject P ₈₃₀₁ at rest It has become. The correlation between the regions R ₈₆₀₁ and R ₈₆₀₂ is not so high, and the value of the correlation information e ₁ (U ₂ ′, V ₂ ′) is not so small. In FIG. 86, the region R ₈₆₀₁ of the past reference image V _P is shifted from the _target block B ₈₅₀₁ by a vector (− (4-s) U ₂ ′, − (4-s) V ₂ ′). _Yes , the pixel value Ip (x- (4-s) U ₂ ′, y- (4-s) V ₂ ′) of the pixel in the region R ₈₆₀₁ is the correlation information e ₁ (U ₂ ) in Expression (9). ', V ₂ '). Further, the region R ₈₆₀₂ of the future reference image V _F is located at a position shifted from the _target block B ₈₅₀₁ by the vector (sU ₂ ′, sV ₂ ′), and the pixel value If (x + sU ₂ in the region R ₈₆₀₂ ', Y + sV ₂ ') is used to calculate the correlation information e ₁ (U ₂ ', V ₂ ') in equation (9).

Now, (U ′, V ′) that minimizes the correlation information e ₁ (U ′, V ′) calculated for the target block B ₈₅₀₁ is (U ₁ ′, V ₁ ′) shown in FIG. Suppose. In this case, if the target block B ₈₅₀₁ is the block B ₈₄₀₁ shown in FIG. 84, the regions R ₈₅₀₁ and R _{8502 in} FIG. 85 match the regions R ₈₄₀₁ and R ₈₄₀₂ in FIG. 84, respectively.

Therefore, the estimated value used when obtaining the first difference data can be obtained from the regions R ₈₅₀₁ and R _{8502 in} FIG. 85 which are the same as the regions R ₈₄₀₁ and R ₈₄₀₂ in FIG. 84, respectively. The block of interest B ₈₅₀₁ (B ₈₄₀₁ ) can be restored by adding to the first difference data.

  Here, the method of restoring the first difference data to the original block as described above is a completely new method that has not been conventionally used in the field of image compression. For example, a low frame rate such as 60 fps moving image data is used. It can be used to restore high frame rate video data such as 240 fps video data from the video data. The following summarizes the restoration method.

  That is, attention is focused on one (frame) image in the high frame rate moving image data to be restored. A plurality of (for example, two) “images constituting moving image data with a low frame rate” that are close in time to the noticed “image to be restored” are extracted, and the correlation between the extracted images is correlated. A portion having a high positional relationship is obtained, and the “image to be restored” focused on from that portion is restored.

  Note that the method of restoring the first difference data is different from, for example, decoding of a B picture used for MPEG2 or the like.

  That is, moving image data composed only of I and P pictures is considered as “low frame rate moving image data”, and moving image data added with a B picture is considered as “high frame rate moving image data”. The method of restoring the difference data of 1 seems to be the same as the method of decoding a B picture.

  However, a B picture in MPEG2 indicates, for each block (macroblock) in the B picture, in which part of the I picture or P picture a texture similar to that block exists, a “motion vector” This information is explicitly held.

On the other hand, the first difference data does not hold such “motion vector” information. Furthermore, restoration of the first difference data, a plurality of frames of moving image data of a low frame rate, i.e., for example, in the Past reference image V _P and Futuresse reference image V _F, searches the positional relationship between the high correlation region (detection) However, it is performed using the region in the positional relationship.

  As described above, the method of restoring the first difference data detects the positional relationship between the areas that do not hold the “motion vector” information and the areas having high correlation in the plurality of frames of the low-frame-rate moving image data, The method is greatly different from the B picture decoding method in that the processing is performed using regions in the positional relationship.

  Next, the second difference data obtained by the difference data calculation unit 235 in FIG. 81 will be described.

In the difference data calculating section 235, for the target block, is detected one motion vector representing the correlation is high positional relationship between the target blocks in the PAST reference image V _P and Futuresse reference image V _F, the positional relationship between the target block, a positional relationship obtained from one motion vector that, from the image data of the region of Pasto reference image V _P and Futuresse reference image V _F, estimate of the block of interest is determined. Then, the difference data calculation unit 235 outputs the difference between the block of interest and its estimated value as second difference data.

That is, FIG. 87 shows the target image V _T , past reference image V _P , and feature reference image V _F (frames) onto which the subjects P ₈₃₀₁ and P ₈₃₀₂ shown in FIG. 83 are projected.

In FIG. 87, for example, the subject P ₈₃₀₂ is moving at a constant speed as in the case of FIG. However, in FIG. 87, the subject P ₈₃₀₂ is moving at a slightly higher speed than in the case of FIG. Also in FIG. 87, the motion vector of the subject P _8302, i.e., per 1/240 second is a frame period in 240fps video data, the spatial direction x of the object P _8302, a vector representing the movement amount of y (U, V ).

Subject P ₈₃₀₂ is also moving at a slightly higher constant rate than in FIG. 84, the target image V _T, Past reference image V _P, and all Futuresse reference image V _F, there is a subject P ₈₃₀₂ Furthermore, the subject P ₈₃₀₂ on the past reference image V _P is at a position shifted from the subject P ₈₃₀₂ on the target image V _{T by} a vector (− (4-s) U, − (4-s) V). subject P ₈₃₀₂ on Futuresse reference image V _F from the subject P ₈₃₀₂ on a target image V _T, is the position shifted by vector (sU, sV).

Accordingly, in the target image V _T, for example, a certain block B ₈₇₀₁ including only the subject P _8302, when attention as the subject block, in Past reference image V _P, from the target block B ₈₇₀₁ vector (- (4-s) U , The image data of the region R ₈₇₀₁ having the same size as the _target block B _{8701 at} a position shifted _by- (4-s) V) is (ideally) the same as the _target block B ₈₇₀₁ . Also in Futuresse reference image V _F, a vector from the target block B ₈₇₀₁ (sU, sV) of position shifted by the image data of the target block B ₈₇₀₁ same size as the region R ₈₇₀₂ is (ideally) of interest It is the same as block B ₈₇₀₁ .

That is, the image data of Pasto reference image V _P of region R _8701, the block of interest B _8701, and the region R ₈₇₀₂ of Futuresse reference image V _F is made the same (ideally).

  The difference data calculation unit 235 (FIG. 81) obtains second difference data suitable for the case shown in FIG. 87 as described above.

That is, even in the difference data calculation unit 235 (FIG. 81), for example, as in the case of the difference data calculation unit 234, image data of the region R ₈₇₀₁ of the past reference image V _P and the region R ₈₇₀₂ of the future reference image V _F the average value of is calculated, the average value, as estimated value of the block of interest B _8701, the difference between the target block B ₈₇₀₁ and its estimated value is calculated. Since this difference is almost zero, the data amount of the target block B ₈₇₀₁ can be reduced. The difference data calculation unit 235 outputs the difference between the attention block B ₈₇₀₁ and the estimated value as described above as second difference data that is a compression result of the attention block B ₈₇₀₁ .

Meanwhile, the second difference data, to restore to the original block of interest B _8701, as in the case of restoring the first differential data described above, the motion vector from the target block B ₈₇₀₁ to Pasto reference image V _P Two motion vectors, a vector (-(4-s) U,-(4-s) V) and a vector (sU, sV) that is a motion vector from the target block B ₈₇₀₁ to the future reference image V _F Is required.

  However, when two motion vectors (-(4-s) U,-(4-s) V) and (sU, sV) are added to the second difference data, the two motion vectors ( -(4-s) U,-(4-s) V) and (sU, sV) increase the amount of data.

Therefore, the difference data calculation unit 235 obtains the regions R ₈₇₀₁ and R ₈₇₀₂ (motion vectors to) used for _obtaining the estimated value of the block of interest B ₈₇₀₁ as follows, by using only one motion vector. Second differential data that can be restored is obtained.

That is, as described above, in this case, since it is assumed that the subject P ₈₃₀₂ is moving at a constant speed, the motion from the target block B ₈₇₀₁ vector (- (4-s) U , - (4-s ) V) shifted by PAST reference image V region of the _P R _8701, the block of interest B ₈₇₀₁ from the motion vector (sU, sV) shifted by Futuresse reference image V region on _F R _8702, and the block of interest B ₈₇₀₁ (the As shown in FIG. 87, the corresponding pixels are aligned in a straight line in the space-time.

Therefore, the motion vector from the target block _{B 8701 (- (4-s} ) U, - (4-s) V) a region R ₈₇₀₁ on which shifted by PAST reference image V _P, the motion vector from the target block B ₈₇₀₁ (sU , sV), the region R ₈₇₀₂ on the feature reference image V _{F is located} on a straight line passing through the block of interest B ₈₇₀₁ in time and space.

Further, the block of interest B _8701, the image data of the area R ₈₇₀₁ on Pasto reference image V _P, as described above, (ideally) from the same, the block of interest B ₈₇₀₁ and the region R ₈₇₀₁ correlation becomes highest in the correlation between the target block B ₈₇₀₁ and Pasto reference image V _P on the region. Further, the block of interest B _8701, also image data of a region R ₈₇₀₂ on Futuresse reference image V _F, as described above, (ideally) from the same, the block of interest B ₈₇₀₁ and the region R ₈₇₀₂ correlation becomes highest in the correlation between the target block B ₈₇₀₁ and Futuresse reference image V _F on the region.

Therefore, if a region having the highest correlation with the _target block B ₈₇₀₁ is detected in each of the past reference image V _P and the feature reference image V _F , the past reference image V _P region, the future reference image V _F region, _Are regions R ₈₇₀₁ and R ₈₇₀₂ , respectively.

By the way, the region R ₈₇₀₁ on the past reference image V _P is located at a position shifted from the _target block B ₈₇₀₁ by a motion vector (− (4-s) U, − (4-s) V), and the future reference image V _F The upper region R ₈₇₀₂ is located at a position shifted from the target block B ₈₇₀₁ by a motion vector (sU, sV).

Therefore, the position of the region R ₈₇₀₁ on Pasto reference image V _P also, the position of the region R ₈₇₀₂ on Futuresse reference image V _F, can be obtained from one motion vector (U, V).

That is, the motion vector from the target block B ₈₇₀₁ to the area _{R 8701 (- (4-s} ) U, - (4-s) V) and the motion vector from the target block B ₈₇₀₁ to the area R ₈₇₀₂ (sU, sV) Thus, even if there are no two motion vectors in total, the regions R ₈₇₀₁ and R ₈₇₀₂ can be obtained from one motion vector (U, V).

  The difference data calculation unit 235 (FIG. 81) obtains second difference data that can be restored from one motion vector (U, V) using the principle as described above.

Specifically, the difference data calculation unit 235 calculates the correlation between the region Ip of the past reference image V _P or the region If of the future reference image V _F on the straight line in the space-time passing through the block of interest and the block of interest. The correlation information e ₂ (U ′, V ′) to be expressed is obtained according to, for example, the equation (10).

・・・（１０）

... (10)

Here, in the formula (10), (x, y ) represents the position of the pixel of the target block in the target image V _T. Ip (x- (4-s) U ′, y- (4-s) V ′) is a position (x- (4-s) U ′, y- (4-s) in the past reference image V _P. ) Represents the pixel value of the pixel in V ′). Furthermore, If (x + sU ', y + sV') represents the position (x + sU ', y + sV') pixel value of the pixel in the Futuresse reference image V _F. Further, Ic (x, y) represents the position in the target image V _T (x, y) pixel value of the pixel in the (pixel value of the pixel of the target block). Further, Σ in Expression (10) represents summation for all the pixels constituting the block of interest.

Incidentally, the correlation information e ₂ of the formula (10) (U ', V ') is the absolute difference of pixel values between the target block and the area Ip Past reference image V _P, a region If the Futuresse reference image V _F block of interest is the sum of the sum of the difference absolute values of the pixel values and, hence, correlation and the block of interest and the region Ip Past reference image V _P, the area If the block of interest Futuresse reference image V _F The higher the correlation, the smaller the “value” of the correlation information e ₂ (U ′, V ′).

The difference data calculation unit 235 calculates the correlation information e ₂ (U ′, V ′) in Expression (10) for all possible (U ′, V ′). Here, all the possible (U ′, V ′) are, for example, positions (x- (4-s) U ′, y- (4-s) V ′) on the past reference image V _P. and a position (x + sU ', y + sV') when is on Futuresse reference image _{V F, (U ', V} '). In addition, the calculation of the correlation information e ₂ (U ′, V ′) in Expression (10) may be performed only for (U ′, V ′) within a predetermined range.

That is, for example, as shown in FIG. 88, the difference data calculation unit 235 sets (U ′, V ′) = (U ₁ ′, V ₁ ′) to correlate information e ₂ (U ₁ ′) in Expression (10). , V ₁ '). Further, for example, as shown in FIG. 89, the difference data calculation unit 235 sets (U ′, V ′) = (U ₂ ′, V ₂ ′) as the correlation information e ₂ (U ₂ ′) in Expression (10). , V ₂ ').

Here, in FIG. 88, by setting (U ′, V ′) = (U ₁ ′, V ₁ ′), the past reference image that is a target of calculation of the correlation information e ₂ (U ₁ ′, V ₁ ′). area Ip of V _P has become the same region R ₈₈₀₁ and the partial block of interest B ₈₇₀₁ of the subject P ₈₃₀₂ is moving. Further, in FIG. 88, the correlation information _{e 2 (U 1 ', V} 1') area If the Futuresse reference image V _F as a calculation target of, again, a portion in the target block among the object P ₈₃₀₂ in motion It is the same region R ₈₈₀₂ . Accordingly, since the correlation between the block of interest B ₈₇₀₁ and each of the regions R _{8801 and} R ₈₈₀₂ is high, the value of the correlation information e ₂ (U ₁ ′, V ₁ ′) is small. Incidentally, in FIG. 88, the region R ₈₈₀₁ Past reference image V _P from the block of interest B _8701, vector (- (4-s) U 1 ', - (4-s) V 1') to a position shifted by _Yes , the pixel value Ip (x- (4-s) U ₁ ′, y- (4-s) V ₁ ′) of the pixel in the region R ₈₈₀₁ is the correlation information e ₂ (U ₁ ) in Expression (10). ', V ₁ '). Further, the region R ₈₈₀₂ of the future reference image V _F is located at a position shifted from the target block B ₈₇₀₁ by the vector (sU ₁ ′, sV ₁ ′), and the pixel value If (x + sU ₁ in the region R ₈₈₀₂ ', Y + sV ₁ ') is used to calculate the correlation information e ₂ (U ₁ ', V ₁ ') in equation (10).

On the other hand, in FIG. 89, by setting (U ′, V ′) = (U ₂ ′, V ₂ ′), the past reference image V that is the calculation target of the correlation information e ₂ (U ₂ ′, V ₂ ′). _P regions Ip has become a part of the region R ₈₉₀₁ with a subject P ₈₃₀₁ which is stationary. Further, in FIG. 89, the correlation information _{e 2 (U 2 ', V} 2') regions If the calculated subject to Futuresse reference image V _F of, become a region R ₈₉₀₂ of the same portion of the subject P ₈₃₀₁ at rest ing. The attention block B ₈₇₀₁ is a part of the moving subject P _8302. Therefore, the correlation between the attention block B ₈₇₀₁ and each of the regions R _{8901 and} R ₈₉₀₂ is not so high, and the correlation information e ₂ ( The value of U ₂ ', V ₂ ') is not so small. In FIG. 89, the region R ₈₉₀₁ of the past reference image V _P is shifted from the _target block B ₈₇₀₁ by a vector (− (4-s) U ₂ ′, − (4-s) V ₂ ′). _Yes , the pixel value Ip (x- (4-s) U ₂ ′, y- (4-s) V ₂ ′) of the pixel in the region R ₈₉₀₁ is the correlation information e ₂ (U ₂ ) in Expression (10). ', V ₂ '). Further, the region R ₈₉₀₂ of the future reference image V _F is located at a position shifted from the target block B ₈₇₀₁ by the vector (sU ₂ ′, sV ₂ ′), and the pixel value If (x + sU ₂ in the region R ₈₉₀₂ ', Y + sV ₂ ') is used to calculate the correlation information e ₂ (U ₂ ', V ₂ ') in equation (10). However, in FIG. 89, (U ₂ ′, V ₂ ′) = (0, 0).

Now, (U ′, V ′) that minimizes the correlation information e ₂ (U ′, V ′) calculated for the _target block B ₈₇₀₁ is (U ₁ ′, V ₁ ′) shown in FIG. _Assuming that the regions R ₈₈₀₁ and R ₈₈₀₂ in FIG. 88 correspond to the regions R ₈₇₀₁ and R ₈₇₀₂ in FIG. 87, respectively.

Then, as described above, the positions of the regions R ₈₇₀₁ and R ₈₇₀₂ are the motion vector (-(4-s) U,-(4-s) V) from the _target block B ₈₇₀₁ to the region R _8701, and the _target block. The correlation information e ₂ (U ′, V ′) calculated for the target block B ₈₇₀₁ is minimized even if there are no two motion vectors in total with the motion vector (sU, sV) from the B ₈₇₀₁ to the region R ₈₇₀₂ It can be obtained from one motion vector (U, V) which is (U ', V').

Now, assuming that the motion vector (U, V) is a correlation maximum vector (U, V), the difference data calculation unit 235 minimizes the correlation information e ₂ (U ′, V ′) in Expression (10). After detecting the maximum correlation vector (U, V) that is (U ', V'), the past reference image whose positional relationship with the block of interest is the positional relationship obtained from the maximum correlation vector (U, V) from the image data of V _P and Futuresse reference image V _F, we obtain the estimated value of the block of interest.

That is, the difference data calculation unit 235 has an area of the same size as the target block on the past reference image V _P shifted from the target block by a vector (− (4-s) U, − (4-s) V), from the target block vector (sU, sV) shifted on Futuresse reference image V _F was obtains a region of the block of interest and the same size, the image of the region of the Pasto reference image V _P, and Futuresse reference image V _F of the area The average value of the data is obtained as an estimated value of the block of interest.

  Then, the difference data calculation unit 235 calculates a difference between the block of interest and its estimated value, sets the calculated result as second difference data, and sets the correlation maximum vector (U, V) in the second difference data. Append and output.

Such second difference data, the motion vector from the target block to Pasto reference image _{V P (- (4-s} ) U, - (4-s) V) and, from the target block to Futuresse reference image V _F Even if there is no two motion vectors (sU, sV) of (no two motion vectors are added to the second difference data), the maximum correlation that is one motion vector added to it The vector (U, V) can be used to restore the original block of interest.

That is, as in the case of obtaining the second difference data, first, the path relationship between the target block and the target block is the position relationship obtained from the maximum correlation vector (U, V) added to the second difference data. from the image data of the reference image V _P and Futuresse reference image V _F, obtains the estimated value of the block of interest. Specifically, an area of the same size as the target block on the past reference image V _P shifted from the target block by a vector (-(4-s) U,-(4-s) V), and the vector from the target block An area of the same size as the target block on the future reference image V _F shifted by (sU, sV) is obtained, and the average value of the image data of the area of the past reference image V _{P and} the area of the future reference image V _F As the estimated value of the block of interest.

  Then, the block of interest can be restored by adding the second difference data and the estimated value of the block of interest.

As described above, the second difference data, it is possible to one correlation maximum vector (U, V) is restored if the motion vector from the target block to Pasto reference image V _P (- (4-s ) U, - (a 4-s) V), motion vector from the target block to Futuresse reference image V _F (sU, as compared with the case of adding the two motion vectors with sV), an increase in the data amount Can be suppressed.

  For the first and second difference data, only one of them may be obtained and the compression result of the block may be obtained, but both the first and second difference data are obtained and the appropriate one of them is determined. The block compression result has the following advantages.

  That is, with respect to the first and second difference data, since the correlation maximum vector (U, V) is added to the second difference data, no motion vector is added from the viewpoint of the corresponding data amount. The first difference data is superior.

  However, the first differential data may have a large data amount. In this case, even if the data amount of the correlation maximum vector (U, V) is included, the data amount of the second differential data is better. The data amount of the first difference data may be smaller.

That is, for example, the region R ₈₈₀₁ on Pasto reference image V _P shown in FIG. 88, the region R ₈₈₀₂ on Futuresse reference image V _F, but is an image of a part of the subject P ₈₃₀₂ is moving, the area When R ₈₈₀₁ and R ₈₈₀₂ are images of the same part of the moving subject P ₈₃₀₂ , the correlation information e ₁ (U ₁ ′, V ₁ ) calculated for the block of interest B ₈₇₀₁ is calculated. The correlation between the regions R ₈₈₀₁ and R ₈₈₀₂ represented by ') is high.

On the other hand, for example, the past-reference image V _P of region R ₈₉₀₁ shown in FIG. 89, the region R ₈₉₀₂ of Futuresse reference image V _F, but is an image of a part of the subject P ₈₃₀₁ which is stationary, the area R _{Even when 8901} and R ₈₉₀₂ are images of the same portion of the stationary subject P ₈₃₀₁ , the regions R ₈₉₀₁ and R represented by the correlation information e ₁ (U ₂ ′, V ₂ ′) in Expression (9) The correlation with ₈₉₀₂ is high.

That is, the value of the correlation information e ₁ (U ₁ ′, V ₁ ′) in Expression (9) representing the correlation between the regions R ₈₈₀₁ and R ₈₈₀₂ in FIG. 88 and the correlation between the regions R ₈₉₀₁ and R ₈₉₀₂ in FIG. The value of the correlation information e ₁ (U ₂ ′, V ₂ ′) in the equation (9) that represents is ideally 0 and is the same.

However, actually, the value of the correlation information e ₁ (U ₁ ′, V ₁ ′) between the regions R ₈₈₀₁ and R ₈₈₀₂ in FIG. 88 becomes smaller due to the influence of noise or the like. The value of the correlation information e ₁ (U ₂ ′, V ₂ ′) between the regions R ₈₉₀₁ and R ₈₉₀₂ in 89 may be smaller.

Then, when the value of the correlation information e ₁ (U ₂ ′, V ₂ ′) between the regions R ₈₉₀₁ and R ₈₉₀₂ in FIG. 89 becomes smaller, the attention block B from the regions R ₈₉₀₁ and R ₈₉₀₂ An estimated value of ₈₇₀₁ is obtained.

That is, the estimated value of the _target block B ₈₇₀₁ of the moving subject P ₈₃₀₂ is obtained from the regions R ₈₉₀₁ and R ₈₉₀₂ of the portion of the stationary subject P ₈₃₀₁ .

The estimated value of the _target block B ₈₇₀₁ obtained from the regions R ₈₉₀₁ and R ₈₉₀₂ of the portion of the stationary subject P ₈₃₀₁ is significantly different from the _target block B ₈₇₀₁ (image data thereof). Even if the difference between the block B ₈₇₀₁ and the estimated value is taken, the difference, that is, the first difference data does not become 0 or a value close to 0 but becomes a large value, and the data amount is hardly reduced.

  In this case, even if the data amount of the correlation maximum vector (U, V) is included, the data amount of the second difference data is smaller than the data amount of the first difference data.

Further, when the _target block B ₈₇₀₁ is compressed into the first difference data due to the influence of noise or the like, the value of the correlation information e ₁ (U ₁ ′, V ₁ ′) between the regions R ₈₈₀₁ and R ₈₈₀₂ in FIG. 89, the value of one of the correlation information e ₁ (U ₂ ′, V ₂ ′) between the regions R ₈₉₀₁ and R ₈₉₀₂ is smaller, but the first difference data At the time of restoration, the other value may be smaller.

In this case, the first difference data is restored using an estimated value different from the estimated value of the _target block B ₈₇₀₁ used when the _target block B ₈₇₀₁ is compressed into the first difference data. The attention block B ₈₇₀₁ cannot be restored.

Therefore, for the _target block B ₈₇₀₁ , by adopting only the second difference data to which the correlation maximum vector (U, V) is added, of the first and second difference data, the data as described above is obtained. It can be prevented that the amount is hardly reduced and the original attention block B ₈₇₀₁ cannot be restored.

  However, when only the second difference data is employed, the data amount corresponding to the maximum correlation vector (U, V) is necessarily increased.

Therefore, for the target block B _8701, the first and obtains both the second difference data, for example, by selecting the lesser of the data amount of which, as the final compression result of the block of interest B _8701, the amount of data In addition, it is possible to prevent the target block B ₈₇₀₁ from being unable to be restored.

Further, for example, when there is almost no difference between the minimum value of the correlation information e ₁ (U ′, V ′) of Equation (9) calculated when obtaining the first difference data and the second smallest value (For example, when it is below a certain threshold), even when the amount of the first difference data is small, the second difference data is selected as the final compression result of the block of interest B _8701. It is possible to more firmly prevent the attention block B ₈₇₀₁ from being unable to be restored.

  Next, the 3rd difference data calculated | required by the difference data calculation part 236 of FIG. 81 is demonstrated.

In the difference data calculating section 236, for the target block, a motion representing the correlation is high positional relationship between a motion vector representing the correlation is high positional relationship between the target blocks in the PAST reference image V _P, the target block in Futuresse reference image V _F In total, two motion vectors with the vector are determined.

Here, hereinafter, the motion vector representing the correlation is high positional relationship between the target blocks in the PAST reference image V _P, before called motion vector, also highly correlated positions of the target block in Futuresse reference image V _F A motion vector representing the relationship is referred to as a post-motion vector.

In the difference data calculating section 236, the image data of the region of Pasto reference image V _P in the positional relationship in which the positional relationship between the block of interest is expressed in the previous motion vector is expressed positional relationship between the target block in backward motion vector from the image data of the region of Futuresse reference image V _F in a positional relationship that, estimate of the block of interest is determined. Then, the difference data calculation unit 236 outputs the difference between the block of interest and its estimated value as third difference data.

90 shows the target image V _T , past reference image V _P , and feature reference image V _F (frames) onto which the subjects P ₈₃₀₁ and P ₈₃₀₂ shown in FIG. 83 are projected.

In FIG. 90, for example, the subject P ₈₃₀₂ is moving at a non-constant speed. That is, in FIG. 90, the object P ₈₃₀₂ is subjected the time of the target image V _T from Pasto reference image V _P, moves at a slower rate, from the target image V _T to a time of Futuresse reference image V _F, at a faster rate Has moved.

Now, in the target image V _T, for example, a certain block B ₉₀₀₁ including only the subject P _8302, when attention as the subject block, in Past reference image V _P of FIG. 90, the vector (-U ₁ from the target block B _9001, - image data V ₁₎ of the shifted position, the block of interest B ₉₀₀₁ and the region R ₉₀₀₁ of the same size is made equal to the (ideally) the block of interest B _9001. In the future reference image V _F , the image data of the region R ₉₀₀₂ having the same size as the _target block B _{9001 at} the position shifted from the _target block B ₉₀₀₁ by the vector (U ₂ , V ₂ ) is (ideally) It is the same as the attention block B ₉₀₀₁ . Therefore, the vector (−U ₁ , −V ₁ ) is a front motion vector, and the vector (U ₂ , V ₂ ) is a back motion vector.

  The difference data calculation unit 236 (FIG. 81) obtains third difference data suitable for the case shown in FIG. 90 as described above.

That is, the difference data calculating unit 236 (FIG. 81), for example, the region R ₉₀₀₁ Past reference image V _P, the average value of the image data of the area R ₉₀₀₂ of Futuresse reference image V _F is calculated, the average value as an estimate of the block of interest B _9001, the difference between the target block B ₉₀₀₁ and its estimated value is calculated. Since this difference is almost 0, the data amount of the target block B ₉₀₀₁ can be reduced. The difference data calculation unit 236 outputs the difference between the block of interest B ₉₀₀₁ as described above and its estimated value as third difference data that is the compression result of the block of interest B ₉₀₀₁ .

Here, the difference data calculation unit 236 uses the previous motion vector (−U ₁ , −V ₁ ) representing the position of the region R ₉₀₀₁ of the same (most similar) image data as the target block B ₉₀₀₁ in the past reference image V _P. ) And the back motion vector (U ₂ , V ₂ ) representing the position of the same (most similar) image data area R _{9002 as} the target block B ₉₀₀₁ in the future reference image V _F as follows. .

That is, the difference data calculation unit 236 obtains the correlation information e ₃ (U ′, V ′) representing the correlation between the block of interest and the region Ip of the past reference image V _P according to, for example, Expression (11).

・・・（１１）

(11)

Here, in Expression (11), (x, y) represents the pixel position of the target block in the target image V _T , and Ic (x, y) represents the position (x, y) in the target image V _T. This represents the pixel value of the pixel (pixel value of the pixel of the target block). Also, Ip (x + U ', y + V') denotes the pixel value of the pixel at the position in Pasto reference image _{V P (x + U ',} y + V'). Further, Σ in equation (11) represents summation for all the pixels constituting the block of interest.

Note that the correlation information e ₃ (U ′, V ′) in Expression (11) is the sum of absolute differences of pixel values between the block of interest and the region Ip of the past reference image V _P , and accordingly, the block of interest and the past the higher the correlation between the area Ip of the reference image V _P, the "value" of the correlation information _{e 3 (U ', V'} ) is reduced.

Further, the difference data calculation unit 236, a block of interest, the correlation information e ₄ representing the correlation between the region If the Futuresse reference image V _F a (U ', V'), for example, determined according to the equation (12).

・・・（１２）

(12)

Here, in Expression (12), (x, y) represents the position of the pixel of the target block in the target image V _T , and Ic (x, y) represents the position (x, y) in the target image V _T. This represents the pixel value of the pixel (pixel value of the pixel of the target block). Also, If (x + U ', y + V') denotes the pixel value of the pixel at the position (x + U ', y + V') in Futuresse reference image V _F. Further, Σ in Expression (12) represents summation for all the pixels constituting the block of interest.

Incidentally, formula (12) correlation information _{e 4 (U ', V'} ) of a total sum of the difference absolute value of pixel values of a region If the block of interest and Futuresse reference image V _F, therefore, the block of interest and Fuyucha the higher the correlation between the region If the reference image V _F, the "value" of the correlation information _{e 4 (U ', V'} ) is reduced.

Differential data calculation unit 236, correlation information e ₃ of the formula (11) (U ', V') and the formula (12) correlation information e ₄ (U ', V') of a, all possible (U ', V'). Here, all the possible (U ′, V ′) are the positions (x + U ′, y + V ′) of the correlation information e ₃ (U ′, V ′) in Expression (11). (U ′, V ′) in the case of being on the reference image V _P , and the position (x + U ′, y + V ′) for the correlation information e ₄ (U ′, V ′) in Expression (12). there is a case that is on Futuresse reference image _{V F (U ', V'} ). The calculation equation correlation information e ₃ of (11) (U ', V') and the correlation information e ₄ (U ', V') of the formula (12), other, of a predetermined range (U It may be performed only for ', V').

The difference data calculation unit 236 detects the minimum value from the correlation information e ₃ (U ′, V ′) calculated for (U ′, V ′) of each value, and the correlation information e ₃ (U ′, V ′). (U ′, V ′) that gives the minimum value is detected as the previous motion vector (−U ₁ , −V ₁ ). Further, the difference data calculating section 236, for each value (U ', V') correlation information calculated for e ₄ (U ', V') detects a minimum value from the correlation information e ₄ (U ', V (U ', V') that gives the minimum value of ') is detected as a rear motion vector (U ₂ , V ₂ ).

In the difference data calculation unit 236, the image data of the region of the past reference image V _{P having} the positional relationship represented by the previous motion vector (−U ₁ , −V ₁ ) and the positional relationship between the target block and the target block from the image data of the region of Futuresse reference image V _F in a positional relationship shown by the positional relationship backward motion vector (U _2, V _2), an estimate of the block of interest is determined. Then, in the difference data calculation unit 236, the difference between the block of interest and its estimated value is used as third difference data, the third difference data is added to the previous motion vector (−U ₁ , −V ₁ ), and the following A motion vector (U ₂ , V ₂ ) is added and output.

Such third difference data is restored to the original block of interest using the front motion vector (−U ₁ , −V ₁ ) and the back motion vector (U ₂ , V ₂ ) added thereto. can do.

That is, as in the case of obtaining the third difference data, first, the positional relationship with the block of interest is the previous motion vector (−U ₁ , −V ₁ ) added to the third difference data and the subsequent motion vector. from the image data of the Past reference image V _P and Futuresse reference image V _F at the position relationship represented by (U _2, V _2), we obtain the estimated value of the block of interest. That is, before the motion vector from the target block (-U _1, -V ₁₎ shifted on Pasto reference image V _P has a region of the block of interest and the same size, backward motion vector from the target block (U _2, V ₂₎ An area of the same size as the target block on the future reference image V _{F that} is shifted by a certain amount is obtained, and the average value of the image data of the past reference image V _P area and the future reference image V _F area Obtained as an estimated value.

  Then, the block of interest can be restored by adding the third difference data and the estimated value of the block of interest.

  Next, the fourth difference data obtained by the difference data calculation unit 237 in FIG. 81 will be described.

In the difference data calculation unit 237, only the motion vector representing the positional relationship having a high correlation with the target block in the past reference image V _P , that is, the previous motion vector (−U ₁ , −V ₁ ) is obtained for the target block ( motion vector correlation represents a high positional relationship of the target block and the Futuresse reference image V _F, that is, backward motion vector (U _2, V ₂₎ is not required).

The difference data calculation unit 237 estimates the target block from the image data of the region of the past reference image V _P in which the positional relationship with the target block is the positional relationship represented by the previous motion vector (−U ₁ , −V ₁ ). A value is determined. Then, the difference data calculation unit 237 outputs the difference between the block of interest and its estimated value as fourth difference data.

That is, FIG. 91 shows the target image V _T , past reference image V _P , and feature reference image V _F (frames) onto which the subjects P ₈₃₀₁ and P ₈₃₀₂ shown in FIG. 83 are projected.

In FIG. 91, for example, the moving subject P ₈₃₀₂ suddenly disappears. That is, in FIG. 91, the object P ₈₃₀₂ that is moving, although there are from Pasto reference image V _P to a time of the target image V _T, between the target image V _T of time of Futuresse reference image V _F It has disappeared.

Now, in the target image V _T, for example, a certain block B ₉₁₀₁ including only the subject P _8302, when attention as the subject block, in Past reference image V _P of FIG. 91, the vector (-U ₁ from the target block B _9101, - image data V ₁₎ of the shifted position, the block of interest B ₉₁₀₁ and the same size of the area R ₉₁₀₁ is made identical to the (ideally) the block of interest B _9101. In the Futuresse reference image V _F, the block of interest B same (similar) and ₉₁₀₁ regions are not present.

  The difference data calculation unit 237 (FIG. 81) obtains fourth difference data suitable for the case shown in FIG. 91 as described above.

That is, the difference data calculating unit 237 (FIG. 81), for example, the image data of the area R ₉₁₀₁ Past reference image V _P, it is, as inferred values of the target block B _9101, of the block of interest B ₉₁₀₁ and its estimated value The difference is calculated. Since this difference is almost zero, the data amount of the target block B ₉₁₀₁ can be reduced. The difference data calculation unit 237 outputs the difference between the attention block B ₉₁₀₁ as described above and the estimated value thereof as fourth difference data that is a compression result of the attention block B ₉₁₀₁ .

Here, the difference data calculating section 237 in a past reference image V _P, the block of interest B identical (most similar) and ₉₁₀₁ before representing the location of the region R ₉₁₀₁ of the image data a motion vector (-U _1, -V ₁ ) Is obtained as follows.

That is, the difference data calculation unit 237 obtains the correlation information e ₃ (U ′, V ′) representing the correlation between the block of interest and the region Ip of the past reference image V _{P according} to, for example, the above equation (11). .

The difference data calculation unit 237 calculates the correlation information e ₃ (U ′, V ′) in Expression (11) for all possible (U ′, V ′). Further, the difference data calculating section 237, for each value (U ', V') correlation information e ₃ (U ', V') calculated for detecting a minimum value from the correlation information e ₃ (U ', V (U ', V') that gives the minimum value of ') is detected as the previous motion vector (-U ₁ , -V ₁ ).

The difference data calculation unit 237 estimates the target block from the image data of the past reference image V _P region in which the positional relationship with the target block is the positional relationship represented by the previous motion vector (−U ₁ , −V ₁ ). A value is determined. Then, the difference data calculation unit 237 adds the difference between the block of interest and its estimated value as the fourth difference data, and adds the previous motion vector (−U ₁ , −V ₁ ) to the fourth difference data. Is output.

Such fourth difference data can be restored to the original block of interest using the previous motion vector (−U ₁ , −V ₁ ) added thereto.

That is, as in the case of obtaining the fourth difference data, first, the positional relationship with the block of interest is represented by the previous motion vector (−U ₁ , −V ₁ ) added to the fourth difference data. from the image data of Pasto reference image V _P in the positional relationship, obtaining the estimated value of the block of interest. That is, an area of the same size as the target block on the past reference image V _P shifted from the target block by the previous motion vector (−U ₁ , −V ₁ ) is obtained, and image data of the area of the past reference image V _P is obtained. As it is, it is obtained as an estimated value of the target block.

  The block of interest can be restored by adding the fourth difference data and the estimated value of the block of interest.

  Next, the fifth difference data obtained by the difference data calculation unit 238 in FIG. 81 will be described.

In the difference data calculation unit 238, only the motion vector representing the positional relationship having a high correlation with the target block in the feature reference image V _F , that is, the rear motion vector (U ₂ , V ₂ ) is obtained for the target block (pasto reference). motion vector correlation represents a high positional relationship between the target block in the image V _P, that is, before the motion vector (-U _1, -V ₁₎ is not required).

In the difference data calculation unit 238, from the image data of the area of the feature reference image V _F in a positional relationship in which the positional relationship between the block of interest is represented by backward motion vector (U _2, V _2), an estimate of the block of interest is Desired. Then, the difference data calculation unit 238 outputs the difference between the block of interest and its estimated value as fifth difference data.

That is, FIG. 92 shows the target image V _T , past reference image V _P , and feature reference image V _F (frames) onto which the subjects P ₈₃₀₁ and P ₈₃₀₂ shown in FIG. 83 are projected.

In FIG. 92, for example, the moving subject P ₈₃₀₂ suddenly appears. That is, in FIG. 92, the moving subject P ₈₃₀₂ does not exist at the time of the past reference image V _P , but appears after that until the time of the target image V _T , and the time of the future reference image V _F But it exists.

Now, in the target image V _T, for example, a certain block B ₉₂₀₁ including only the subject P _8302, when attention as the subject block, in the feature reference image V _F of Figure 92, the vector from the target block B ₉₂₀₁ (U _2, V ₂ The image data of the region R ₉₂₀₂ of the same size as the _target block B _{9201 at} a position shifted by () is (ideally) the same as the _target block B ₉₂₀₁ . Further, in the past reference image V _P , there is no same (similar) area as the target block B ₉₂₀₁ .

  The difference data calculation unit 238 (FIG. 81) obtains fifth difference data suitable for the case shown in FIG. 92 as described above.

That is, the difference data calculating unit 238 (FIG. 81), for example, the image data of the area R ₉₂₀₂ of the feature reference image V _F, it is, as inferred values of the target block B _9201, of the block of interest B ₉₂₀₁ and its estimated value The difference is calculated. Since this difference is almost 0, the data amount of the target block B ₉₂₀₁ can be reduced. The difference data calculation unit 238 outputs the difference between the attention block B ₉₂₀₁ and the estimated value as described above as fifth difference data that is a compression result of the attention block B ₉₂₀₁ .

Here, the difference data calculation unit 238, the feature reference image V _F, identical to the block of interest B ₉₂₀₁ (the most similar) motion vector after indicating the position of region R ₉₂₀₂ of the image data (U _2, V ₂₎ Find it as follows.

That is, the difference data calculation unit 238, a block of interest, the correlation information e ₄ representing the correlation between the region If the feature reference image _{V F (U ', V'} ), for example, determined according to the above Expression (12) .

The difference data calculation unit 238 calculates the correlation information e ₄ (U ′, V ′) in Expression (12) for all possible (U ′, V ′). Further, the difference data calculating unit 238 of each value (U ', V') correlation information calculated for e ₄ (U ', V') detects a minimum value from the correlation information e ₄ (U ', V (U ', V') that gives the minimum value of ') is detected as a rear motion vector (U ₂ , V ₂ ).

In the difference data calculation unit 238, from the image data of the area of the feature reference image V _F in a positional relationship in which the positional relationship between the block of interest is represented by backward motion vector (U _2, V _2), an estimate of the block of interest is Desired. Then, the difference data calculation unit 238 outputs the difference between the block of interest and its estimated value as fifth difference data, and adds the rear motion vector (U ₂ , V ₂ ) to the fifth difference data. Is done.

Such fifth difference data can be restored to the original block of interest using the post-motion vector (U ₂ , V ₂ ) added thereto.

That is, as in the case of obtaining the fifth difference data, first, the positional relationship with the target block is the positional relationship represented by the rear motion vector (U ₂ , V ₂ ) added to the fifth differential data. from the image data of the feature reference image V _F in, obtaining the estimated value of the block of interest. That is, on the feature reference image V _F offset by backward motion vector (U _2, V ₂₎ from the block of interest, determine the area of the block of interest and the same size, the image data of the area of the feature reference image V _F, as As an estimated value of the target block.

  Then, the block of interest can be restored by adding the fifth difference data and the estimated value of the block of interest.

  Next, the sixth difference data obtained by the difference data calculation unit 239 in FIG. 81 will be described.

  In the difference data calculation unit 239, assuming that the estimated value of the target block is 0, the difference between the target block and the estimated value is output as sixth difference data.

That is, FIG. 93 shows the target image V _T , past reference image V _P , and feature reference image V _F (frames) onto which the subjects P ₈₃₀₁ and P ₈₃₀₂ shown in FIG. 83 are projected.

In FIG. 93, for example, the moving subject P ₈₃₀₂ suddenly appears and then disappears suddenly. That is, in FIG. 93, the moving subject P ₈₃₀₂ does not exist at the time of the past reference image V _P , but appears thereafter until the time of the target image V _T , and further, the feature reference image V _F It disappears by the time of.

Now, in the target image V _T , for example, when a block B ₉₃₀₁ including only the subject P ₈₃₀₂ is focused as a focused block, the focused block B is included in both the past reference image V _P and the feature reference image V _F in FIG. There is no region similar to (similar to) ₉₃₀₁ .

  The difference data calculation unit 239 (FIG. 81) obtains sixth difference data suitable for the case shown in FIG. 93 as described above.

In other words, in any of Pasto reference image V _P and feature reference image V _F, when the target block B the same (similar) and ₉₃₀₁ there is no region of Pasto reference image V _P and feature reference image V _F, also be determined estimate of the block of interest B _9301, the estimate has a value far from the image data of the target block B _9301. Therefore, even if the difference between the _target block B ₉₃₀₁ and such an estimated value is taken, the difference is almost zero and becomes a large value, and therefore the data amount of the target block B ₉₃₀₁ cannot be reduced. .

Therefore, the difference data calculation unit 239 outputs the target block B ₉₃₀₁ as sixth difference data without _obtaining the estimated value of the target block B ₉₃₀₁ . Here, as described above, the sixth difference data can be regarded as a result of taking the difference between the target block and the estimated value, assuming that the estimated value of the target block is 0.

  Of the above first to sixth difference data, the third to sixth difference data have the same concept as the B picture in MPEG.

  In the above case, the sum of absolute difference values is used as the correlation information representing the correlation in order to reduce the amount of calculation. However, a correlation coefficient or the like can also be used as the correlation information.

  Next, FIG. 94 shows a configuration example of the difference data calculation unit 234 in FIG. 81 for obtaining the first difference data.

In FIG. 94, the target block (image data) from the input terminal 233 of FIG. 81 is input to the input terminal 251. In addition, the past reference image V _P from the input terminal 231 in FIG. 81 is input to the input terminal 252, and the future reference image V _F from the input terminal 232 in FIG. 81 is input to the input terminal 253.

  The block of interest input from the input terminal 251 is supplied to the subtraction unit 256.

The past reference image V _P input from the input terminal 252 is supplied to the maximum correlation position detection unit 254 and the average value calculation unit 255. Futuresse reference image V _F which is input from the input terminal 253 is also supplied to the maximum correlation position detection section 254 and the average value calculation section 255.

Maximum correlation position detection section 254, for the target block from the input terminal 251, and Past reference image V _P from the input terminal 252, in the Futuresse reference image V _F from the input terminal 253, detects the correlation is high positional relationship . That is, the maximum correlation position detection unit 254 calculates the correlation information e ₁ (U ′, V ′) of Expression (9) for the block of interest, and calculates the correlation represented by the correlation information e ₁ (U ′, V ′). It is detected to maximize (U ′, V ′), that is, minimize (U ′, V ′) the value of correlation information e ₁ (U ′, V ′). Then, the maximum correlation position detection section 254, correlation information _{e 1 (U ', V'} ) to minimize the value of the (U ', V'), in the PAST reference image V _P and Futuresse reference image V _F, correlation Is supplied to the average value calculation unit 255 as a positional relationship vector (U, V) representing the positional relationship of the highest region.

The average value calculation unit 255 calculates the target block from the image data of the past reference image V _P and the feature reference image V _{F in} the positional relationship represented by the positional relationship vector (U, V) from the correlation maximum position detection unit 254. Is obtained and supplied to the subtracting unit 256.

That is, the average value calculation unit 255 determines the region at a position shifted from the target block by the vector (− (4-s) U, − (4-s) V) in the past reference image V _P and the feature reference image V _F. , For example, the weighted average value of the image data of the region at a position shifted from the target block by the vector (sU, sV) is calculated according to the equation (13), and the weighted average value is calculated as the estimated value of the target block. Obtained as Pre (x, y) and supplied to the subtracting unit 256.

・・・（１３）

(13)

Here, in Expression (13), (x, y) represents the position of each pixel of the target block in the target image V _T. Also, Ip (x- (4-s ) U, y- (4-s) V) is located in the Past reference image _{V P (x- (4-s} ) U, y- (4-s) V) Represents the pixel value of the pixel. Furthermore, If (x + sU, y + sV) represents the pixel value of the pixel position (x + sU, y + sV ) in Futuresse reference image V _F. W can be a weight, for example, 1/2.

The weight W can be set, for example, according to a user operation input. Further, the weight W is, for example, (4-s) / 240 seconds, which is a time difference between the target image V _T and the past reference image V _P shown in FIG. 82, and the target image V _T and the feature reference image V. It may be determined in consideration of the ratio with s / 240 seconds, which is the time difference from _F.

That is, when the time difference (4-s) / 240 seconds between the target image V _T and the past reference image V _P is shorter than s / 240 seconds, which is the time difference between the target image V _T and the feature reference image V _P. the estimated value of the block of interest, who found by emphasis on Past reference image V _P is, it is possible to obtain a higher accuracy values. On the other hand, when the time difference between the target image V _T and the future reference image V _P is shorter than (4-s) / 240 seconds, which is the time difference between the target image V _T and the past reference image V _P the estimated value of the block of interest, who found by emphasis on Futuresse reference image V _F is, it is possible to obtain a higher accuracy values. Therefore, the weight W can be a value that depends on the variable s representing the above-described time difference, that is, for example, s / 4.

  The subtracting unit 256 uses the estimated value Pre (x, y) of the target block supplied from the average value calculating unit 255 to calculate the target block (image data) Ic (x, y) supplied from the input terminal 251. Compress. That is, the subtracting unit 256 corresponds to the pixel corresponding to the estimated value Pre (x, y) of the target block supplied from the average value calculating unit 255 and the target block Ic (x, y) supplied from the input terminal 251. The difference value Sub (x, y) = Ic (x, y) −Pre (x, y) obtained as a result is supplied to the conversion unit 257.

  The conversion unit 257 converts the difference value Sub (x, y) of the target block from the subtraction unit 256 into data on the frequency space. That is, the transform unit 257 performs, for example, DCT (Discrete Cosine Transform) transform on the difference value Sub (x, y) of the block of interest, and supplies the DCT coefficient obtained as a result to the quantizer 258.

  The quantization unit 258 quantizes the DCT coefficient of the block of interest from the transform unit 257 and supplies the quantized data obtained as a result to the variable length coding unit 259.

  The variable length coding unit 259 performs variable length coding on the quantized data of the block of interest from the quantization unit 258, and uses the resulting variable length code as the first difference data from the output terminal 260 to the selection circuit 240. (FIG. 81). When the block of interest (image data) Ic (x, y) and its estimated value Pre (x, y) completely match, the variable length coding unit 259 converts the quantized data of the block of interest into variable length The variable length code obtained by encoding represents NULL.

  Next, the operation of the difference data calculation unit 234 in FIG. 94 will be described with reference to the flowchart in FIG.

First, in step S211, the block of interest, past reference image V _P , and future reference image V _F are input. That is, the block of interest is input from the input terminal 251 to the subtraction unit 256, and the past reference image V _P is input from the input terminal 252 to the maximum correlation position detection unit 254 and the average value calculation unit 255. Furthermore, Futuresse reference image V _F is, from the input terminal 253 is input to the maximum correlation position detection section 254 and the average value calculation section 255.

Then, the process proceeds from step S211 to step S212, and the maximum correlation position detection unit 254 uses the past reference image V _P from the input terminal 252 and the future reference image V _F from the input terminal 253 for the block of interest from the input terminal 251. , Detecting a positional relationship with high correlation. That is, the maximum correlation position detection unit 254 calculates the correlation information e ₁ (U ′, V ′) in Expression (9) for all possible (U ′, V ′) values for the block of interest, and The correlation represented by the correlation information e ₁ (U ′, V ′) is maximized (U ′, V ′), that is, the value of the correlation information e ₁ (U ′, V ′) is minimized (U ′, V ′). ') Is detected. Then, the maximum correlation position detection section 254, correlation information _{e 1 (U ', V'} ) to minimize the value of the (U ', V'), in the PAST reference image V _P and Futuresse reference image V _F, correlation Is supplied to the average value calculation unit 255 as a positional relationship vector (U, V) representing the positional relationship of the highest region, and the process proceeds from step S212 to S213.

In step S213, the average value calculating unit 255, follows the equation (13), positional relation vector (U, V) from the maximum correlation position detection section 254 are in the positional relationship indicated by, Past reference image V _P and Futuresse reference image from the image data of the region of the V _F, we obtain the estimated value Pre block of interest (x, y), a is supplied to the subtraction unit 256, processing proceeds to step S214.

  In step S214, the subtracting unit 256 corresponds the estimated value Pre (x, y) of the target block supplied from the average value calculating unit 255 and the target block Ic (x, y) supplied from the input terminal 251. The difference between the pixel values of the pixels to be calculated is calculated, and the difference value Sub (x, y) obtained as a result is supplied to the conversion unit 257, and the process proceeds to step S215.

  In step S215, the converting unit 257 DCT-transforms the difference value Sub (x, y) of the target block from the subtracting unit 256, and supplies the resulting DCT coefficient to the quantizing unit 258, and then proceeds to step S216. move on. In step S216, the quantization unit 258 quantizes the DCT coefficient of the block of interest from the transform unit 257, supplies the quantized data obtained as a result to the variable length coding unit 259, and proceeds to step S217. In step S217, the variable length coding unit 259 performs variable length coding on the quantized data of the block of interest from the quantization unit 258, proceeds to step S218, and obtains the variable length code obtained as a result of the variable length coding. 1 is output from the output terminal 260 to the selection circuit 240 (FIG. 81). Accordingly, the first difference data does not include the positional relationship vector (U, V).

As described above, the difference data calculation unit 234 compresses the target image V _T to the first difference data in units of blocks.

  Next, FIG. 96 shows a configuration example of the difference data calculation unit 235 of FIG. 81 for obtaining the second difference data.

96, the target block (image data) from the input terminal 233 in FIG. 81 is input to the input terminal 271. In addition, the past reference image V _P from the input terminal 231 in FIG. 81 is input to the input terminal 272, and the future reference image V _F from the input terminal 232 in FIG. 81 is input to the input terminal 273.

  The block of interest input from the input terminal 271 is supplied to the maximum correlation position detection unit 274 and the subtraction unit 276.

The past reference image V _P input from the input terminal 272 is supplied to the correlation maximum position detection unit 274 and the average value calculation unit 275. The feature reference image V _F input from the input terminal 273 is also supplied to the correlation maximum position detection unit 274 and the average value calculation unit 275.

The correlation maximum position detection unit 274 determines the block of interest from the input terminal 271 in the past reference image V _P from the input terminal 272 and the feature reference image V _F from the input terminal 273 for the block of interest from the input terminal 271. A positional relationship having a high correlation is detected. That is, the maximum correlation position detection unit 274 calculates the correlation information e ₂ (U ′, V ′) of Expression (10) for the block of interest, and calculates the correlation represented by the correlation information e ₂ (U ′, V ′). It is detected to maximize (U ′, V ′), that is, minimize (U ′, V ′) the value of the correlation information e ₂ (U ′, V ′). Then, the maximum correlation position detection section 274, correlation information _{e 2 (U ', V'} ) to minimize the value of the (U ', V'), in the PAST reference image V _P and Futuresse reference image V _F, attention This is supplied to the average value calculation unit 275 and the variable length coding unit 279 as the maximum correlation vector (U, V) representing the positional relationship of the region having the highest correlation with the block.

The average value calculation unit 275 has a positional relationship with the block of interest in a positional relationship obtained from the correlation maximum vector (U, V) from the correlation maximum position detection unit 274, and the past reference image V _P and the feature reference image V _F The estimated value of the block of interest is obtained from the image data of the area, and supplied to the subtracting unit 276.

That is, the average value calculating unit 275, in Pasto reference image V _P, from the target block vector (- (4-s) U , - (4-s) V) shifted the position of the region, Futuresse reference image V _F , For example, the weighted average value of the image data with the region at a position shifted from the target block by the vector (sU, sV) is calculated according to the above-described equation (13), and the weighted average value is calculated for the target block. The estimated value Pre (x, y) is obtained and supplied to the subtracting unit 276.

  The subtraction unit 276 uses the estimated value Pre (x, y) of the target block supplied from the average value calculation unit 275 to calculate the target block (image data) Ic (x, y) supplied from the input terminal 271. Compress. That is, the subtraction unit 276 corresponds to the pixel corresponding to the estimated value Pre (x, y) of the target block supplied from the average value calculation unit 275 and the target block Ic (x, y) supplied from the input terminal 271. The difference value Sub (x, y) = Ic (x, y) −Pre (x, y) obtained as a result is supplied to the conversion unit 277.

  The conversion unit 277 converts the difference value Sub (x, y) of the target block from the subtraction unit 276 into data on the frequency space. That is, the transform unit 277 performs, for example, DCT transform on the difference value Sub (x, y) of the block of interest, and supplies the resulting DCT coefficient to the quantizer 278.

  The quantization unit 278 quantizes the DCT coefficient of the block of interest from the conversion unit 277 and supplies the quantized data obtained as a result to the variable length coding unit 279.

  The variable length coding unit 279 performs variable length coding on the quantized data of the block of interest from the quantization unit 278 and variable length coding the correlation maximum vector (U, V) from the correlation maximum position detection unit 274. . Further, the variable length coding unit 279 adds the variable length code of the correlation maximum vector (U, V) to the variable length code of the quantized data of the block of interest, and outputs the second difference data from the output terminal 280. It outputs to the selection circuit 240 (FIG. 81). When the block of interest (image data) Ic (x, y) and its estimated value Pre (x, y) completely match, the variable length coding unit 279 converts the quantized data of the block of interest into a variable length. The variable length code obtained by encoding represents NULL, and the second difference data is only the variable length code of the maximum correlation vector (U, V).

  Next, the operation of the difference data calculation unit 235 in FIG. 96 will be described with reference to the flowchart in FIG.

First, in step S221, the block of interest, past reference image V _P , and future reference image V _F are input. That is, the block of interest is input from the input terminal 271 to the maximum correlation position detection unit 274 and subtraction unit 276, and the past reference image V _P is input from the input terminal 272 to the maximum correlation position detection unit 274 and average value calculation unit 275. Is done. Furthermore, Futuresse reference image V _F is, from the input terminal 273 is input to the maximum correlation position detection section 274 and the average value calculation section 275.

Then, the process proceeds from step S221 to step S222, and the maximum correlation position detection unit 274 uses the past reference image V _P from the input terminal 272 and the feature reference image V _F from the input terminal 273 for the block of interest from the input terminal 271. Then, a positional relationship having a high correlation with the block of interest is detected. That is, the maximum correlation position detection unit 274 calculates the correlation information e ₂ (U ′, V ′) in Expression (10) for all possible (U ′, V ′) values for the target block, The correlation represented by the correlation information e ₂ (U ′, V ′) is maximized (U ′, V ′), that is, the value of the correlation information e ₂ (U ′, V ′) is minimized (U ′, V ′). ') Is detected. Then, the maximum correlation position detection section 274, correlation information _{e 2 (U ', V'} ) to minimize the value of the (U ', V'), in the PAST reference image V _P and Futuresse reference image V _F, attention The maximum correlation vector (U, V) representing the positional relationship of the region having the highest correlation with the block is supplied to the average value calculation unit 275 and the variable length coding unit 279, and the process proceeds from step S222 to S223.

In step S223, the average value calculation unit 275 has a positional relationship with the block of interest obtained from the maximum correlation vector (U, V) from the maximum correlation position detection unit 274 according to equation (13). The estimated value Pre (x, y) of the block of interest is obtained from the image data of the past reference image V _P and the feature reference image V _F , supplied to the subtraction unit 276, and the process proceeds to step S224.

  In step S224, the subtraction unit 276 associates the estimated value Pre (x, y) of the target block supplied from the average value calculation unit 275 with the target block Ic (x, y) supplied from the input terminal 271. The difference between the pixel values of the pixels to be calculated is calculated, and the difference value Sub (x, y) obtained as a result is supplied to the conversion unit 277, and the process proceeds to step S225.

  In step S225, the converting unit 277 performs DCT conversion on the difference value Sub (x, y) of the target block from the subtracting unit 276, and supplies the resulting DCT coefficient to the quantizing unit 278, and then proceeds to step S226. move on. In step S226, the quantization unit 278 quantizes the DCT coefficient of the target block from the transform unit 277, supplies the quantized data obtained as a result to the variable length coding unit 279, and proceeds to step S227. In step S227, the variable length coding unit 279 performs variable length coding on the quantized data of the block of interest from the quantization unit 278 and also varies the correlation maximum vector (U, V) from the correlation maximum position detection unit 274. Encode long. Further, in step S227, the variable length coding unit 279 adds the variable length code of the correlation maximum vector (U, V) to the variable length code of the quantized data of the target block, thereby obtaining the second difference data. In step S228, the second difference data is output from the output terminal 280 to the selection circuit 240 (FIG. 81).

As described above, the difference data calculation unit 235 compresses the target image V _T into second difference data in units of blocks.

  Next, FIG. 98 shows a configuration example of the difference data calculation unit 236 of FIG. 81 for obtaining the third difference data.

98, the target block (image data) from the input terminal 233 in FIG. 81 is input to the input terminal 291. In addition, the past reference image V _P from the input terminal 231 in FIG. 81 is input to the input terminal 292, and the future reference image V _F from the input terminal 232 in FIG. 81 is input to the input terminal 293.

  The block of interest input from the input terminal 291 is supplied to the correlation maximum position detection units 294 and 295 and the subtraction unit 297.

The past reference image V _P input from the input terminal 292 is supplied to the correlation maximum position detection unit 294 and the average value calculation unit 296. Futuresse reference image V _F which is input from the input terminal 293 is supplied to a maximum correlation position detection section 295 and the average value calculation section 296.

Maximum correlation position detection section 294, in Pasto reference image V _P from the input terminal 292, maximum correlation position detection section 295 for detecting a correlation is high positional relationship between the target block from the input terminal 291, from the input terminal 293 in Futuresse reference image V _F, it detects the correlation is high positional relationship between the target block from the input terminal 291.

That is, the maximum correlation position detection unit 294 calculates the correlation information e ₃ (U ′, V ′) of Expression (11) for the block of interest, and calculates the correlation represented by the correlation information e ₃ (U ′, V ′). The maximum (U ′, V ′), that is, the minimum (U ′, V ′) of the correlation information e ₃ (U ′, V ′) is detected. Then, the maximum correlation position detection section 294, correlation information _{e 3 (U ', V'} ) value to smallest (U ', V'), in the PAST reference image V _P, the correlation between the target block is the most This is supplied to the average value calculation unit 296 and the variable length coding unit 300 as a previous motion vector (−U ₁ , −V ₁ ) representing the positional relationship of a high region.

In addition, the maximum correlation position detection unit 295 calculates the correlation information e ₄ (U ′, V ′) of Expression (12) for the block of interest, and calculates the correlation represented by the correlation information e ₄ (U ′, V ′). It is detected to maximize (U ′, V ′), that is, to minimize the value of correlation information e ₄ (U ′, V ′). Then, the maximum correlation position detection section 295, correlation information _{e 4 (U ', V'} ) to minimize the value of the (U ', V'), in Futuresse reference image V _F, the correlation between the target block is the most This is supplied to the average value calculation unit 296 and the variable length coding unit 300 as a post motion vector (U ₂ , V ₂ ) representing the positional relationship of the high region.

The average value calculation unit 296 has a positional relationship with the block of interest in the region of the past reference image V _P in which the positional relationship represented by the previous motion vector (−U ₁ , −V ₁ ) from the maximum correlation position detection unit 294 is present. from the image data, the positional relationship between the target block, a positional relationship indicated by the motion vector after the maximum correlation position detection section _{295 (U 2, V 2)} , and the image data of the region of Futuresse reference image V _F, An estimated value of the block of interest is obtained and supplied to the subtraction unit 297.

That is, the average value calculating unit 296, in Pasto reference image V _P, from the target block prior to the motion vector (-U _1, -V ₁₎ and the area of the shifted positions, the Futuresse reference image V _F, the rear vector (U ₂ , V ₂ ), for example, the weighted average value of the image data with the region shifted by the position is calculated according to the equation (14), and the weighted average value is calculated as the estimated value Pre (x, y) of the target block. And supplied to the subtracting unit 297.

・・・（１４）

(14)

Here, in the formula (14), (x, y ) represents the position of each pixel of the block of interest in the target image V _{T. Also, Ip (xU 1, yV 1} ) represents the pixel value of the pixel position (xU _1, yV ₁₎ in the PAST reference image V _{P. Furthermore, If (x + U 2,} y + V 2) represents the pixel value of the pixel position in Futuresse reference image _{V F (x + U 2,} y + V 2). W can be a weight, for example, 1/2.

The weight W can be set, for example, according to a user operation input. Further, the weight W is, for example, (4-s) / 240 seconds, which is a time difference between the target image V _T and the past reference image V _P shown in FIG. 82, and the target image V _T and the feature reference image V. It may be determined in consideration of the ratio with s / 240 seconds, which is the time difference from _F.

That is, when the time difference (4-s) / 240 seconds between the target image V _T and the past reference image V _P is shorter than s / 240 seconds, which is the time difference between the target image V _T and the feature reference image V _P. the estimated value of the block of interest, who found by emphasis on Past reference image V _P is, it is possible to obtain a higher accuracy values. On the other hand, when the time difference between the target image V _T and the future reference image V _P is shorter than (4-s) / 240 seconds, which is the time difference between the target image V _T and the past reference image V _P the estimated value of the block of interest, who found by emphasis on Futuresse reference image V _F is, it is possible to obtain a higher accuracy values. Therefore, the weight W can be a value that depends on the variable s representing the above-described time difference, that is, for example, s / 4.

  The subtraction unit 297 uses the estimated value Pre (x, y) of the target block supplied from the average value calculation unit 296 to calculate the target block (image data) Ic (x, y) supplied from the input terminal 291. Compress. That is, the subtraction unit 297 corresponds to the pixel corresponding to the estimated value Pre (x, y) of the target block supplied from the average value calculation unit 296 and the target block Ic (x, y) supplied from the input terminal 291. The difference value Sub (x, y) = Ic (x, y) −Pre (x, y) obtained as a result is supplied to the conversion unit 298.

  The conversion unit 298 converts the difference value Sub (x, y) of the target block from the subtraction unit 297 into data on the frequency space. That is, the transform unit 298 performs, for example, DCT transform on the difference value Sub (x, y) of the block of interest, and supplies the resulting DCT coefficient to the quantization unit 299.

  The quantization unit 299 quantizes the DCT coefficient of the block of interest from the transform unit 298 and supplies the quantized data obtained as a result to the variable length coding unit 300.

The variable length coding unit 300 performs variable length coding on the quantized data of the block of interest from the quantization unit 299, and the previous motion vector (−U ₁ , −V ₁ ) from the correlation maximum position detection unit 294, The rear motion vector (U ₂ , V ₂ ) from the maximum correlation position detector 295 is variable-length encoded. Furthermore, the variable length coding unit 300 adds variable length codes of the previous motion vector (−U ₁ , −V ₁ ) and the subsequent motion vector (U ₂ , V ₂ ) to the variable length code of the quantized data of the block of interest. Is output from the output terminal 301 to the selection circuit 240 (FIG. 81) as third difference data. When the block of interest (image data) Ic (x, y) and its estimated value Pre (x, y) completely match, the variable-length coding unit 300 converts the quantized data of the block of interest into a variable length The variable length code obtained by encoding represents NULL, and the third difference data is the difference between the previous motion vector (−U ₁ , −V ₁ ) and the subsequent motion vector (U ₂ , V ₂ ). Only variable length codes.

  Next, the operation of the difference data calculation unit 236 in FIG. 98 will be described with reference to the flowchart in FIG.

First, in step S231, the target block, past reference image V _P , and future reference image V _F are input. That is, the block of interest is input from the input terminal 291 to the correlation maximum position detection units 294 and 295 and the subtraction unit 297, and the past reference image V _P is input from the input terminal 292 to the correlation maximum position detection unit 294 and the average value calculation unit. 296. Further, the future reference image V _F is input from the input terminal 293 to the maximum correlation position detection unit 295 and the average value calculation unit 296.

Then, the process proceeds from step S231 to S232, and the maximum correlation position detection unit 294 detects a positional relationship having a high correlation with the target block in the past reference image V _P from the input terminal 292 for the target block from the input terminal 291. . That is, the maximum correlation position detection unit 294 calculates the correlation information e ₃ (U ′, V ′) in Expression (11) for all possible (U ′, V ′) values for the block of interest, The correlation represented by the correlation information e ₃ (U ', V') is maximized (U ', V'), that is, the correlation information e ₃ (U ', V') is minimized (U ', V') ') Is detected. Then, the maximum correlation position detection section 294, correlation information _{e 3 (U ', V'} ) value to smallest (U ', V'), in the PAST reference image V _P, the correlation between the target block is the most The previous motion vector (−U ₁ , −V ₁ ) representing the positional relationship of the high region is supplied to the average value calculation unit 296 and the variable length coding unit 300, and the process proceeds from step S232 to S233.

In step S233, the maximum correlation position detection section 295, for the target block from the input terminal 291, in Futuresse reference image V _F from the input terminal 293, the correlation between the target block is detected with high positional relationship. That is, the maximum correlation position detection unit 295 calculates the correlation information e ₄ (U ′, V ′) in Expression (12) for all possible values of (U ′, V ′) for the block of interest. The correlation represented by the correlation information e ₄ (U ′, V ′) is maximized (U ′, V ′), that is, the value of the correlation information e ₄ (U ′, V ′) is minimized (U ′, V ′). ') Is detected. Then, the maximum correlation position detection section 294, correlation information _{e 4 (U ', V'} ) to minimize the value of the (U ', V'), in Futuresse reference image V _F, the correlation between the target block is the most The post motion vector (U ₂ , V ₂ ) representing the positional relationship of the high region is supplied to the average value calculation unit 296 and the variable length coding unit 300, and the process proceeds from step S233 to S234.

In step S234, the average value calculation unit 296 represents the positional relationship represented by the previous motion vector (−U ₁ , −V ₁ ) from the maximum correlation position detection unit 294 in accordance with the equation (14). in some, the image data of a region of Pasto reference image V _P, positional relationship between the target block is in a positional relationship indicated by the motion vector (U _2, V ₂₎ after the maximum correlation position detection section 295, Futuresse reference from the image data region of the image V _F, we obtain the estimated value Pre block of interest (x, y), a is supplied to the subtraction unit 297, processing proceeds to step S235.

  In step S <b> 235, the subtraction unit 297 corresponds to the estimated value Pre (x, y) of the target block supplied from the average value calculation unit 296 and the target block Ic (x, y) supplied from the input terminal 291. The difference between the pixel values of the pixels to be calculated is calculated, the difference value Sub (x, y) obtained as a result is supplied to the conversion unit 298, and the process proceeds to step S236.

In step S236, the conversion unit 298 DCT-transforms the difference value Sub (x, y) of the target block from the subtraction unit 297, supplies the DCT coefficient obtained as a result to the quantization unit 299, and proceeds to step S237. move on. In step S237, the quantization unit 299 quantizes the DCT coefficient of the block of interest from the conversion unit 298, supplies the quantized data obtained as a result to the variable length coding unit 300, and proceeds to step S238. In step S238, the variable length coding unit 300 performs variable length coding on the quantized data of the block of interest from the quantization unit 299, and the previous motion vector (−U ₁ , −V from the maximum correlation position detection unit 294). ₁ ) and the rear motion vector (U ₂ , V ₂ ) from the maximum correlation position detector 295 are variable-length encoded. Further, in step S238, the variable length coding unit 300 adds the front motion vector (−U ₁ , −V ₁ ) and the back motion vector (U ₂ , V ₂ ) to the variable length code of the quantized data of the block of interest. As the third difference data, the process proceeds to step S239, and the third difference data is output from the output terminal 301 to the selection circuit 240 (FIG. 81).

As described above, the difference data calculating section 236, a target image V _T is in units of blocks, is compressed to a third difference data.

  Next, FIG. 100 shows a configuration example of the difference data calculation unit 237 of FIG. 81 for obtaining the fourth difference data.

In FIG. 100, the target block (image data) from the input terminal 233 in FIG. 81 is input to the input terminal 311, and the past reference image V _P from the input terminal 231 in FIG. 81 is input to the input terminal 312. Is done.

  The block of interest input from the input terminal 311 is supplied to the maximum correlation position detection unit 313 and the subtraction unit 315.

The past reference image V _P input from the input terminal 312 is supplied to the correlation maximum position detection unit 313 and the clipping unit 314.

Maximum correlation position detection section 313, in Pasto reference image V _P from the input terminal 312, detects the correlation is high positional relationship between the target block from the input terminal 311.

That is, the maximum correlation position detection unit 313 calculates the correlation information e ₃ (U ′, V ′) of Expression (11) for the block of interest, and calculates the correlation represented by the correlation information e ₃ (U ′, V ′). The maximum (U ′, V ′), that is, the minimum (U ′, V ′) of the correlation information e ₃ (U ′, V ′) is detected. The maximum correlation position detection section 313, correlation information _{e 3 (U ', V'} ) value to smallest (U ', V'), in the PAST reference image V _P, the correlation between the target block is the most This is supplied to the cutout unit 314 and the variable length coding unit 318 as the previous motion vector (−U ₁ , −V ₁ ) representing the positional relationship of the high region.

The cutout unit 314 has image data of a region of the past reference image V _P in which the positional relationship with the target block is the positional relationship represented by the previous motion vector (−U ₁ , −V ₁ ) from the maximum correlation position detection unit 313. And is supplied to the subtraction unit 315 as the estimated value of the block of interest.

In other words, the cutout unit 314 uses the estimated data Pre (x, y) of the block of interest in the region of the past reference image V _P that is shifted from the block of interest by the previous motion vector (−U ₁ , −V ₁ ). ) Is cut out (obtained) and supplied to the subtractor 315.

  The subtraction unit 315 compresses the target block (image data) Ic (x, y) supplied from the input terminal 311 using the estimated value Pre (x, y) of the target block supplied from the cutout unit 314. . That is, the subtraction unit 315 corresponds to the pixel of the corresponding pixel between the estimated value Pre (x, y) of the target block supplied from the clipping unit 314 and the target block Ic (x, y) supplied from the input terminal 311. The difference between the values is calculated, and the difference value Sub (x, y) = Ic (x, y) −Pre (x, y) obtained as a result is supplied to the conversion unit 316.

  The conversion unit 316 converts the difference value Sub (x, y) of the target block from the subtraction unit 315 into data on the frequency space. That is, the transform unit 316 performs, for example, DCT transform on the difference value Sub (x, y) of the block of interest, and supplies the resulting DCT coefficient to the quantization unit 317.

  The quantization unit 317 quantizes the DCT coefficient of the block of interest from the conversion unit 316 and supplies the quantized data obtained as a result to the variable length encoding unit 318.

The variable length encoding unit 318 performs variable length encoding on the quantized data of the block of interest from the quantization unit 317 and also changes the previous motion vector (−U ₁ , −V ₁ ) from the correlation maximum position detection unit 313. Encode long. Further, the variable length coding unit 318 adds the variable length code of the previous motion vector (−U ₁ , −V ₁ ) to the variable length code of the quantized data of the block of interest, and outputs it as fourth difference data. The signal is output from the terminal 319 to the selection circuit 240 (FIG. 81). When the block of interest (image data) Ic (x, y) and its estimated value Pre (x, y) completely match, the variable length coding unit 318 converts the quantized data of the block of interest into a variable length. The variable length code obtained by encoding represents NULL, and the fourth difference data is only the variable length code of the previous motion vector (−U ₁ , −V ₁ ).

  Next, the operation of the difference data calculation unit 237 in FIG. 100 will be described with reference to the flowchart in FIG.

First, in step S241, the target block and past reference image V _P are input. That is, the block of interest is input from the input terminal 311 to the maximum correlation position detection unit 313 and the subtraction unit 315, and the past reference image V _P is input from the input terminal 312 to the maximum correlation position detection unit 313 and the clipping unit 314. .

Then, the process proceeds from step S241 to S242, and the maximum correlation position detection unit 313 detects the positional relationship of the block of interest from the input terminal 311 having a high correlation with the block of interest in the past reference image V _P from the input terminal 312. . That is, the maximum correlation position detection unit 313 calculates the correlation information e ₃ (U ′, V ′) in Expression (11) for all possible values of (U ′, V ′) for the block of interest. The correlation represented by the correlation information e ₃ (U ', V') is maximized (U ', V'), that is, the correlation information e ₃ (U ', V') is minimized (U ', V') ') Is detected. The maximum correlation position detection section 313, correlation information _{e 3 (U ', V'} ) value to smallest (U ', V'), in the PAST reference image V _P, the correlation between the target block is the most The previous motion vector (−U ₁ , −V ₁ ) representing the positional relationship of the high region is supplied to the cutout unit 314 and the variable length coding unit 318, and the process proceeds from step S242 to S243.

In step S243, the cutout unit 314 has the positional relationship with the target block of the past reference image V _{P in} the positional relationship represented by the previous motion vector (−U ₁ , −V ₁ ) from the correlation maximum position detection unit 313. The image data of the area is cut out as the estimated value Pre (x, y) of the block of interest, supplied to the subtraction unit 315, and the process proceeds to step S244.

  In step S244, the subtracting unit 315 corresponds to the pixel corresponding to the estimated value Pre (x, y) of the target block supplied from the clipping unit 314 and the target block Ic (x, y) supplied from the input terminal 311. The difference value Sub (x, y) obtained as a result is supplied to the conversion unit 316, and the process proceeds to step S245.

In step S245, the conversion unit 316 DCT-transforms the difference value Sub (x, y) of the target block from the subtraction unit 315, supplies the DCT coefficient obtained as a result to the quantization unit 317, and proceeds to step S246. move on. In step S246, the quantization unit 317 quantizes the DCT coefficient of the block of interest from the conversion unit 316, supplies the quantized data obtained as a result to the variable length coding unit 318, and proceeds to step S247. In step S247, the variable length coding unit 318 performs variable length coding on the quantized data of the block of interest from the quantization unit 317, and the previous motion vector (−U ₁ , −V from the correlation maximum position detection unit 313). ₁ ) Variable length coding. Further, in step S247, the variable length coding unit 318 adds the variable length code of the previous motion vector (−U ₁ , −V ₁ ) to the variable length code of the quantized data of the block of interest, thereby In step S248, the fourth difference data is output from the output terminal 319 to the selection circuit 240 (FIG. 81).

As described above, the difference data calculation unit 237 compresses the target image V _T into the fourth difference data in units of blocks.

  Next, FIG. 102 shows a configuration example of the difference data calculation unit 238 in FIG. 81 for obtaining fifth difference data.

102, the target block (image data) from the input terminal 233 in FIG. 81 is input to the input terminal 331, and the future reference image V _F from the input terminal 232 in FIG. 81 is input to the input terminal 332. Is done.

  The block of interest input from the input terminal 331 is supplied to the correlation maximum position detection unit 333 and the subtraction unit 335.

The feature reference image V _F input from the input terminal 332 is supplied to the maximum correlation position detection unit 333 and the cutout unit 334.

Maximum correlation position detection section 333, in Futuresse reference image V _F from the input terminal 332, detects the correlation is high positional relationship between the target block from the input terminal 331.

That is, the maximum correlation position detection unit 333 calculates the correlation information e ₄ (U ′, V ′) in Expression (12) for the block of interest, and calculates the correlation represented by the correlation information e ₄ (U ′, V ′). It is detected to maximize (U ′, V ′), that is, to minimize the value of correlation information e ₄ (U ′, V ′). Then, the maximum correlation position detection section 333, correlation information _{e 4 (U ', V'} ) to minimize the value of the (U ', V'), in Futuresse reference image V _F, the correlation between the target block is the most This is supplied to the cutout unit 334 and the variable length coding unit 338 as a post motion vector (U ₂ , V ₂ ) representing the positional relationship of the high region.

Clipping unit 334, the positional relationship between the block of interest, cut out image data of the motion vector (U _2, V ₂₎ in a positional relationship represented by the area of Futuresse reference image V _F after the maximum correlation position detection section 333 The estimated value of the target block is supplied to the subtracting unit 335.

That is, the cutout portion 334, in Futuresse reference image V _F, the image data of the area of the position shifted by backward motion vector (U _2, V ₂₎ from the target block, as estimated value Pre block of interest (x, y) Cutting out (obtaining) and supplying to the subtracting unit 335.

  The subtraction unit 335 compresses the target block (image data) Ic (x, y) supplied from the input terminal 331 using the estimated value Pre (x, y) of the target block supplied from the clipping unit 334. . That is, the subtraction unit 335 corresponds to the pixel of the corresponding pixel between the estimated value Pre (x, y) of the target block supplied from the cutout unit 334 and the target block Ic (x, y) supplied from the input terminal 331. The difference between the values is calculated, and the difference value Sub (x, y) = Ic (x, y) −Pre (x, y) obtained as a result is supplied to the conversion unit 336.

  The conversion unit 336 converts the difference value Sub (x, y) of the target block from the subtraction unit 335 into data on the frequency space. That is, the transform unit 336 performs, for example, DCT transform on the difference value Sub (x, y) of the block of interest, and supplies the resulting DCT coefficient to the quantization unit 337.

  The quantization unit 337 quantizes the DCT coefficient of the block of interest from the transform unit 336 and supplies the quantized data obtained as a result to the variable length coding unit 338.

The variable length coding unit 338 performs variable length coding on the quantized data of the block of interest from the quantization unit 337 and also uses the variable length code for the back motion vector (U ₂ , V ₂ ) from the maximum correlation position detection unit 333. Turn into. Further, the variable length coding unit 338 adds the variable length code of the back motion vector (U ₂ , V ₂ ) to the variable length code of the quantized data of the block of interest, and outputs it as fifth difference data to the output terminal 339. To the selection circuit 240 (FIG. 81). When the block of interest (image data) Ic (x, y) and its estimated value Pre (x, y) completely match, the variable length coding unit 338 converts the quantized data of the block of interest into a variable length. The variable length code obtained by encoding represents NULL, and the fifth difference data is only the variable length code of the back motion vector (U ₂ , V ₂ ).

  Next, the operation of the difference data calculation unit 238 in FIG. 102 will be described with reference to the flowchart in FIG.

First, in step S251, the block of interest and Futuresse reference image V _F is input. That is, the target block is input from the input terminal 331 to the maximum correlation position detection unit 333 and the subtraction unit 335, and the future reference image V _F is input from the input terminal 332 to the maximum correlation position detection unit 333 and the clipping unit 334. .

Then, the process proceeds from step S251 to S252, the maximum correlation position detection section 333, for the target block from the input terminal 331, in Futuresse reference image V _F from the input terminal 332, the correlation between the target block is detected with high positional relationship . That is, the maximum correlation position detection unit 333 calculates the correlation information e ₄ (U ′, V ′) in Expression (12) for all possible values of (U ′, V ′) for the block of interest. The correlation represented by the correlation information e ₄ (U ′, V ′) is maximized (U ′, V ′), that is, the value of the correlation information e ₄ (U ′, V ′) is minimized (U ′, V ′). ') Is detected. Then, the maximum correlation position detection section 333, correlation information _{e 4 (U ', V'} ) to minimize the value of the (U ', V'), in Futuresse reference image V _F, the correlation between the target block is the most The post motion vector (U ₂ , V ₂ ) representing the positional relationship of the high region is supplied to the cutout unit 334 and the variable length coding unit 338, and the process proceeds from step S252 to S253.

In step S253, the cutout unit 334 has the positional relationship with the block of interest in the region of the future reference image V _F in which the positional relationship represented by the back motion vector (U ₂ , V ₂ ) from the maximum correlation position detection unit 333 is present. The image data is cut out as the estimated value Pre (x, y) of the block of interest, supplied to the subtraction unit 335, and the process proceeds to step S254.

  In step S254, the subtraction unit 335 corresponds to the pixel of interest corresponding to the estimated value Pre (x, y) of the target block supplied from the cutout unit 334 and the target block Ic (x, y) supplied from the input terminal 331. The difference value Sub (x, y) obtained as a result is supplied to the conversion unit 336, and the process proceeds to step S255.

In step S255, the converting unit 336 performs DCT conversion on the difference value Sub (x, y) of the target block from the subtracting unit 335, and supplies the resulting DCT coefficient to the quantizing unit 337, and the process proceeds to step S256. move on. In step S256, the quantization unit 337 quantizes the DCT coefficient of the block of interest from the conversion unit 336, supplies the quantized data obtained as a result to the variable length coding unit 338, and proceeds to step S257. In step S257, the variable length coding unit 338 performs variable length coding on the quantized data of the block of interest from the quantization unit 337, and the back motion vector (U ₂ , V ₂ ) from the maximum correlation position detection unit 333. Is encoded with variable length. Further, in step S257, the variable length coding unit 338 adds the variable length code of the back motion vector (U ₂ , V ₂ ) to the variable length code of the quantized data of the block of interest, thereby obtaining the fifth difference. The process proceeds to step S258 as data, and the fifth difference data is output from the output terminal 339 to the selection circuit 240 (FIG. 81).

As described above, the difference data calculation unit 238 compresses the target image V _T into the fifth difference data in units of blocks.

  Next, FIG. 104 shows a configuration example of the difference data calculation unit 239 in FIG. 81 for obtaining sixth difference data.

  104, the target block (image data thereof) from the input terminal 233 in FIG. 81 is input to the input terminal 351. The block of interest input from the input terminal 351 is supplied to the conversion unit 352.

  The converting unit 352 receives the difference value Sub () between the target block input from the input terminal 351, that is, the target block (image data) Ic (x, y) and 0 as the estimated value Pre (x, y). x, y) = Ic (x, y) -Pre (x, y) = Ic (x, y) is converted into data in the frequency space. That is, the transform unit 352 performs, for example, DCT transform on the difference value Sub (x, y) equal to the image data Ic (x, y) of the block of interest, and supplies the resulting DCT coefficient to the quantization unit 353. To do.

  The quantization unit 353 quantizes the DCT coefficient of the block of interest from the transform unit 352 and supplies the quantized data obtained as a result to the variable length coding unit 354.

  The variable length coding unit 354 performs variable length coding on the quantized data of the block of interest from the quantization unit 353, and outputs the result as sixth difference data from the output terminal 355 to the selection circuit 240 (FIG. 81).

  Next, the operation of the difference data calculation unit 239 in FIG. 104 will be described with reference to the flowchart in FIG.

  First, in step S261, a target block is input. That is, the block of interest is input from the input terminal 351 to the conversion unit 352.

  Then, the process proceeds from step S261 to S262, and the conversion unit 352 receives the target block input from the input terminal 351, that is, the target block (image data) Ic (x, y) and its estimated value Pre (x, y). The difference value from 0 as Sub (x, y) = Ic (x, y) -Pre (x, y) = Ic (x, y) is DCT transformed, and the resulting DCT coefficient is quantized Then, the process proceeds to step S263. In step S263, the quantization unit 353 quantizes the DCT coefficient of the target block from the transform unit 352, supplies the quantized data obtained as a result to the variable length coding unit 354, and proceeds to step S264. In step S264, the variable length coding unit 354 performs variable length coding on the quantized data of the block of interest from the quantization unit 353, and proceeds to step S265 using the resulting variable length code as sixth difference data. The sixth difference data is output from the output terminal 355 to the selection circuit 240 (FIG. 81).

As described above, the difference data calculation unit 239 compresses the target image V _T into the sixth difference data in units of blocks.

  Next, FIG. 106 schematically illustrates a data structure of each of the first to sixth difference data output from the difference data calculation units 234 to 239 of FIG. 81 to the selection circuit 240.

  As shown first in FIG. 106, the first difference data output from the difference data calculation unit 234 to the selection circuit 240 is obtained by subjecting the difference value between the target block and its estimated value to DCT processing and further quantizing the difference data. It consists of variable length codes of quantized data obtained in this way. That is, the first difference data does not include the positional relationship vector (U, V).

  The second difference data output from the difference data calculation unit 235 to the selection circuit 240 is obtained by performing DCT processing on the difference value between the target block and its estimated value as shown in the second from the top in FIG. The variable length code of the quantized data obtained in this way and the variable length code of the maximum correlation vector (U, V).

The third difference data output from the difference data calculation unit 236 to the selection circuit 240 is obtained by performing DCT processing on the difference value between the target block and its estimated value as shown in the third part from the top of FIG. The variable length code of the quantized data obtained in this way, and the variable length code of the previous motion vector (−U ₁ , −V ₁ ) and the subsequent motion vector (U ₂ , V ₂ ).

The fourth difference data output from the difference data calculation unit 237 to the selection circuit 240 is obtained by performing DCT processing on the difference value between the block of interest and its estimated value, as shown in the fourth from the top in FIG. The variable length code of the quantized data obtained in this way and the variable length code of the previous motion vector (−U ₁ , −V ₁ ).

The fifth difference data output from the difference data calculation unit 238 to the selection circuit 240 is subjected to DCT processing on the difference value between the block of interest and its estimated value, as shown in the fifth figure from the top in FIG. The variable length code of the quantized data obtained in this way and the variable length code of the back motion vector (U ₂ , V ₂ ).

  The sixth difference data output from the difference data calculation unit 239 to the selection circuit 240 is the difference value between the target block and 0 as its estimated value, that is, the target block, as shown in the sixth from the top in FIG. It consists of variable length codes of quantized data obtained by DCT processing and further quantization.

  Note that the variable length code of the quantized data obtained by performing DCT processing on the difference value between the target block and its estimated value, and further quantizing it, as described above, if the target block and its estimated value match. , NULL.

  Next, FIG. 107 schematically shows the data structure of first to sixth difference data output from the output terminal 241 as the differentially compressed data by the selection circuit 240 of FIG.

  The selection circuit 240 selects, as selection difference data, one of the first to sixth difference data output from the difference data calculation units 234 to 239 for the target block, and the selection difference data is the first to sixth difference data. A case ID indicating which of the sixth difference data is added, and output as differential compressed data from the output terminal 241.

  In FIG. 107, 1, 2, 3, 4, 5, and 6 are added as case IDs to the first to sixth difference data, respectively.

  Next, processing of the transmission apparatus 1 in FIG. 79 will be described with reference to the flowchart in FIG.

  The fps moving image data is supplied from the input terminal 211 in FIG. 79 to the band limiting filter unit 212.

  In step S281, the band limiting filter unit 212 performs filtering on the 240 fps moving image data from the input terminal 211 to leave only necessary information in consideration of human visual characteristics as described above, and the 240 fps moving image obtained as a result The data is supplied to the separation circuit 213, and the process proceeds to step S282.

  In step S282, the separation circuit 213 separates the 240 fps moving image data from the band limiting filter unit 212 into 60 fps moving image data and the remaining 240-60 fps moving image data obtained by removing 60 fps moving image data from the 240 fps moving image data. Data is supplied to the compression circuit 214 and 240-60 fps moving image data is supplied to the difference information extraction unit 217 as a target image.

  In step S283, the compression circuit 214 encodes (compresses) the 60 fps moving image data supplied from the separation circuit 213, and outputs the resultant bit stream. This bit stream is supplied to the decompression circuit 216 and the output terminal 215.

  Thereafter, the process proceeds to step S284, and the decompression circuit 216 locally decodes the bit stream from the compression circuit 214, and supplies 60 fps moving image data obtained as a result of the local decoding as a reference image to the difference information extraction unit 217, The process proceeds to step S285.

  In step S285, the difference information extraction unit 217 compresses the 240-60 fps moving image data that is the target image from the separation circuit 213 using the 60 fps moving image data that is the reference image from the decompression circuit 216, that is, the reference image ( The difference compressed data relating to the difference between the target image (240-60 fps moving image data) with respect to the 60 fps moving image data) is obtained, the target image is compressed, the difference compressed data is supplied to the output terminal 218, and the process proceeds to step S286.

  In step S286, a bit stream as an encoding result of 60 fps moving image data (reference image) is output from the output terminal 215, and differential compressed data as a compression result of 240-60 fps moving image data is output from the output terminal 218. The

  As described above, in the transmission device 1 of FIG. 79, the bit stream as the encoding result of the 60 fps moving image data and the differential compressed data as the compression result of the 240-60 fps moving image data are separately output independently. In the receiving apparatus 2 (FIG. 24), for example, by receiving only a bit stream as a result of encoding 60 fps moving picture data, the 60 fps moving picture data can be restored. The receiving device 2 can also restore 60 fps moving image data by receiving, for example, a bit stream as a result of encoding 60 fps moving image data and differentially compressed data as a compression result of 240-60 fps moving image data. And 240fps video data can be restored. In other words, so-called temporal scalability can be provided.

  Next, the compression of the target image (240-60 fps moving image data) performed by the difference information extraction unit 217 in step S285 of FIG. 108 will be described with reference to the flowchart of FIG.

  In the difference information extraction unit 217 (FIG. 81), in step S291, the difference data calculation units 234 to 239 explain the first to sixth difference data shown in FIG. Thus, it is obtained and supplied to the selection circuit 240, and the process proceeds to step S292.

  In step S292, the selection circuit 240 selects, as selection difference data, the one with the smallest data amount from among the first to sixth difference data for the target block from the difference data calculation units 234 to 239, Proceed to step S293.

Here, when the data amount of the first difference data among the first to sixth difference data for the block of interest is the minimum, the selection circuit 240 calculates the first difference data. It is determined whether the difference between the minimum value of the correlation information e ₁ (U ′, V ′) in equation (9) and the second smallest value is equal to or less than a certain threshold, and the correlation information in equation (9) When the difference between the minimum value of e ₁ (U ′, V ′) and the second smallest value is equal to or smaller than a certain threshold value, instead of the first difference data, the second to sixth difference data Can be selected as the selected difference data (the data amount of the first to sixth difference data having the second smallest data amount). In this case, as described above, it is possible to more firmly prevent the first difference data from being restored to the original block of interest.

  In step S293, the selection circuit 240 adds a corresponding case ID to the selected difference data, and thereby converts the difference compressed data having one of the data structures shown in FIG. 107 as the compression result of the block of interest. Output from the output terminal 241.

  That is, as a result, the optimum one of the first to sixth difference data is output from the output terminal 241 for each block.

  As described above, the transmission apparatus 1 in FIG. 79 compresses 240 fps moving image data input from the input terminal 211 and outputs two compression results (first and second compression results). The first compression result is a bit stream obtained by compressing 60 fps moving image data separated from 240 fps moving image data by the compression circuit 214. This bit stream is output from the output terminal 215. The second compression result is differentially compressed data obtained by compressing the 240-60 fps moving image data separated from the 240 fps moving image data by taking the difference from the 60 fps moving image data by the difference information extraction unit 217. The differentially compressed data is variable-length encoded in units of blocks and is output from the output terminal 218.

Further, in the transmission device 1 of FIG. 79, the band limiting filter unit 212 performs a passband filter considering human visual characteristics with respect to 240 fps moving image data from the input terminal 211, that is, a low pass of about 1/60 seconds. A filter (however, the filter coefficient (tap coefficient) is adaptively changed in consideration of the movement of the subject) is applied. Due to human visual characteristics, the human eye cannot follow an object that moves irregularly within a range of about 1/60 seconds, and recognizes the pixel value integrated for 1/60 seconds. Therefore, even if such a low-pass filter (band limiting filter) is applied, humans do not feel that the image quality has deteriorated. The moving image data obtained as a result of filtering by the low-pass filter is moving image data in which the subject is moving at a constant speed within a time of about 1/60 seconds (that is, 4/240 seconds). This is because an irregular movement whose speed changes at a high speed in a time shorter than an interval of 1/60 seconds is averaged by the low-pass filter. Such moving image data minimizes the correlation information e ₁ (U ′, V ′) of the equation (9) and the correlation information e ₂ (U ′, V ′) of the equation (10) (U ′, V ′) is suitable for obtaining an estimated value of the target block in the target image. This is because the formula (9) correlation information e ₁ of (U ', V') or Formula (10) correlation information e ₂ of (U ', V') to minimize (U ', V') in the case of obtaining the This is because, as described in FIG. 84 or FIG. 87, it is assumed that the subject is moving at a constant speed.

  Next, FIG. 110 illustrates a configuration example of the reception device 2 in FIG. 24 when the transmission device 1 is configured as illustrated in FIG. 79.

  The receiving device 2 is supplied with the bit stream as the encoding result of the 60 fps moving image data and the differential compression data as the compression result of the 240-60 fps moving image data output from the transmitting device 1 of FIG. The differential compressed data is input from the terminal 361 to the receiving device 2 from the input terminal 362, respectively. Then, the bit stream is supplied from the input terminal 361 to the decompression circuit 363, and the differential compressed data is supplied from the input terminal 362 to the differential information restoring unit 364.

  In addition, when the bit stream as an encoding result of 60fps moving image data and the differential compression data as the compression result of 240-60fps moving image data are multiplexed in the multiplexed data, the bit stream is converted from the multiplexed data. And the differentially compressed data are separated and input to the input terminals 361 and 362, respectively.

  Similar to the decompression circuit 216 in FIG. 79, the decompression circuit 363 decodes (decompresses) the bit stream from the input terminal 361 and restores 60 fps moving image data. The 60 fps moving image data is supplied as a reference image to the difference information restoring unit 364 and also supplied to the synthesizing unit 365.

  The difference information restoring unit 364 restores the differentially compressed data from the input terminal 362 to 240-60 fps moving image data in units of blocks using the 60 fps moving image data as the reference image from the decompression circuit 363. The 240-60 fps moving image data is supplied to the synthesis unit 365.

  The synthesizing unit 365 synthesizes the 60 fps moving image data from the decompression circuit 363 and the 240-60 fps moving image data from the difference information restoring unit 364, thereby restoring the 240 fps moving image data and from the output terminal 366 to the display device 3. (FIG. 24).

  Next, processing of the receiving device 2 in FIG. 110 will be described with reference to the flowchart in FIG.

  A bit stream as an encoding result of 60 fps moving image data is supplied from the input terminal 361 to the decompression circuit 363. Also, the differential information restoring unit 364 is supplied with differential compressed data as a compression result of 240-60 fps moving image data from the input terminal 362.

  In step S301, the decompression circuit 363 decodes the bit stream from the input terminal 361, and supplies the 60 fps moving image data obtained as a result to the difference information restoration unit 364 as a reference image and also to the synthesis unit 365. Then, the process proceeds to step S302.

  In step S302, the difference information restoration unit 364 uses the 60fps moving image data as the reference image from the decompression circuit 363 to restore the differential compressed data from the input terminal 362 to 240-60fps moving image data in units of blocks. This is supplied to the synthesis unit 365.

  Then, the process proceeds from step S302 to S303, and the combining unit 365 combines the 60 fps moving image data from the decompression circuit 363 and the 240-60 fps moving image data from the difference information restoring unit 364, thereby reconstructing the 240 fps moving image data. Then, the data is output from the output terminal 366 to the display device 3 (FIG. 24).

That is, the decompression circuit 363 uses, for example, the frame of 60 fps moving image data shown second from the top in FIG. 80 from the bit stream from the input terminal 361, f ₁ , f ₅ , f ₉ , f ₁₃ ,. (Image data) is restored and supplied to the difference information restoration unit and synthesis unit 365.

Further, the difference information restoration unit 364 uses the 60 fps moving image data from the decompression circuit 363 as a reference image, and converts the difference compressed data from the input terminal 362 into, for example, the 240-60 fps moving image data shown at the bottom of FIG. ..., F ₂ , f ₃ , f ₄ , f ₆ , f ₇ , f ₈ , f ₁₀ , f ₁₁ , f ₁₂ ,. .

  The synthesizing unit 365 repeatedly selects one frame of 60 fps moving image data from the decompression circuit 363, and then repeatedly selects three frames of 240-60 fps moving image data from the difference information restoring unit 364. Output from the output terminal 366 every 1/240 seconds. That is, by this, the 240 fps moving image data as shown at the top of FIG. 80, which is a combination of the 60 fps moving image data and the 240-60 fps moving image data, is output from the output terminal 366.

  Next, with reference to the flowchart of FIG. 112, the decoding of 240-60 fps moving image data performed by the difference information restoring unit 364 of FIG. 110 in step S302 of FIG. 111 will be described.

  In step S311, the difference information restoration unit 364 uses the 60 fps moving image data from the decompression circuit 363 (FIG. 110) as a reference image, and restores the difference compressed data from the input terminal 362 (FIG. 110) to 240-60 fps moving image data. By performing the processing described later, each frame of the 240-60 fps moving image data, that is, the pixel value of the target image is obtained for each block, and the process proceeds to step S312.

  In step S312, the difference information restoration unit 364 obtains image data of each frame of the 240-60 fps moving image data by collecting the blocks whose pixel values are obtained in step S311 and proceeds to step S313. In step S313, the difference information restoration unit 364 outputs the 240-60 fps moving image data obtained in step S312 to the synthesis unit 365 (FIG. 110).

  Next, FIG. 113 shows a configuration example of the difference information restoration unit 364 of FIG.

  The differential data storage unit 371 is supplied with differential compressed data from the input terminal 362 (FIG. 110). The difference data storage unit 371 temporarily stores the difference compressed data from the input terminal 362.

  The reference storage unit 372 is supplied with 60 fps moving image data as a reference image from the decompression circuit 363. The reference storage unit 372 temporarily stores the reference image from the decompression circuit 363.

  The case ID determination unit 373 reads a target image block to be restored from now on as a target block, reads out the differential compressed data of the target block from the differential data storage unit 371, and adds the case ID added to the differential compressed data. (FIG. 107) is determined. Then, the case ID determination unit 373 supplies the case ID determination result for the block of interest to the maximum correlation position detection unit 377.

The variable length decoding unit 374 reads the difference compressed data of the block of interest from the difference data storage unit 371, obtains quantized data by variable length decoding the difference compressed data, and supplies the quantized data to the inverse quantization unit 375. . Furthermore, the variable length decoding unit 374 performs variable length decoding on the differentially compressed data, so that the maximum correlation vector (U, V), the previous motion vector (−U ₁ , −V ₁ ), or the rear motion is used as necessary. A vector (U ₂ , V ₂ ) is obtained and supplied to the maximum correlation position detector 377.

  That is, when the differentially compressed data is the first or sixth differential data, that is, when the case ID of the target block is 1 or 6, as described with reference to FIG. 106, the first or sixth differential data Since only the variable length code of the quantized data is included, the variable length decoding unit 374 decodes the variable length code into quantized data and supplies it to the inverse quantization unit 375.

  When the differentially compressed data is the second differential data, that is, when the case ID of the target block is 2, as described with reference to FIG. 106, the second differential data includes a variable of quantized data. Since the long code and the variable length code of the maximum correlation vector (U, V) are included, the variable length decoding unit 374 decodes the variable length code into the quantized data and the maximum correlation vector (U, V). . Then, the variable length decoding unit 374 supplies the quantized data to the inverse quantization unit 375 and supplies the maximum correlation vector (U, V) to the maximum correlation position detection unit 377.

Furthermore, when the differentially compressed data is the third differential data, that is, when the case ID of the target block is 3, as described with reference to FIG. 106, the third differential data includes a variable of quantized data. Since the long code and the variable length codes of the front motion vector (−U ₁ , −V ₁ ) and the back motion vector (U ₂ , V ₂ ) are included, the variable length decoding unit 374 converts the variable length code into The quantized data is decoded into the front motion vector (−U ₁ , −V ₁ ) and the back motion vector (U ₂ , V ₂ ). Then, the variable length decoding unit 374 supplies the quantized data to the inverse quantization unit 375, and converts the front motion vector (−U ₁ , −V ₁ ) and the back motion vector (U ₂ , V ₂ ) to the maximum correlation. This is supplied to the position detection unit 377.

Further, when the differentially compressed data is the fourth differential data, that is, when the case ID of the target block is 4, as described with reference to FIG. 106, the fourth differential data includes the variable of the quantized data. Since the long code and the variable length code of the previous motion vector (−U ₁ , −V ₁ ) are included, the variable length decoding unit 374 converts the variable length code into the quantized data and the previous motion vector (−U _1). , -V ₁ ). Then, the variable length decoding unit 374 supplies the quantized data to the inverse quantization unit 375 and supplies the previous motion vector (−U ₁ , −V ₁ ) to the correlation maximum position detection unit 377.

Further, when the differentially compressed data is the fifth differential data, that is, when the case ID of the target block is 5, as described with reference to FIG. 106, the fifth differential data includes a variable of quantized data. Since the long code and the variable length code of the back motion vector (U ₂ , V ₂ ) are included, the variable length decoding unit 374 converts the variable length code into the quantized data and the back motion vector (U ₂ , V _2). ) And decrypt. Then, the variable length decoding unit 374 supplies the quantized data to the inverse quantization unit 375 and supplies the back motion vector (U ₂ , V ₂ ) to the correlation maximum position detection unit 377.

  The inverse quantization unit 375 inversely quantizes the quantized data from the variable length decoding unit 374, and supplies, for example, DCT coefficients as data on the frequency space obtained as a result to the conversion unit 376.

  The transform unit 376 performs inverse DCT transform on the DCT coefficient from the inverse quantization unit 375, and the resulting image data (pixel value) Ic (x, y) of the target block and its estimated value Pre (x, y) The difference value Sub (x, y) is supplied to the adding unit 379.

The maximum correlation position detection unit 377 uses the past reference image V _P and the feature reference for the frames immediately before and after the target image V _T that are the frames of the target block among the reference images stored in the reference storage unit 372, respectively. As the image V _F , the case ID determination result of the block of interest in the case ID determination unit 373 or the maximum correlation vector (U, V) and the previous motion vector (−U ₁ , −V ₁ ) supplied from the variable length decoding unit 374 ), based on the backward motion vector (U _2, V _2), the positional relationship and a high correlation in the PAST reference image V _P and Futuresse reference image V _F, the Past reference image V _P or Futuresse reference image V _F, and the block of interest Are detected and supplied to the estimation unit 378.

Estimating unit 378, based on the positional relationship between the maximum correlation position detection section 377, from the Past reference image V _P or Futuresse reference image V _F of the reference image stored in the reference storage unit 372, estimates Pre block of interest (x, y) is obtained and supplied to the adder 379.

  The adding unit 379 adds the difference value Sub (x, y) of the target block from the converting unit 376 and the estimated value Pre (x, y) of the target block from the estimating unit 378, and thereby the target block ( Image data) Ic (x, y) = Sub (x, y) + Pre (x, y) is restored and output to the synthesis unit 365 (FIG. 110).

  Next, processing for obtaining the pixel value (image data) of the block performed in step S311 of FIG. 112 by the difference information restoring unit 364 of FIG. 112 will be described with reference to the flowcharts of FIGS.

First, in step S321, a frame having 240-60fps moving image data to be restored is set as a target image V _T, and a block obtained by dividing the target image V _T into blocks is stored in the difference data storage unit 371 as a target block. Of the difference compressed data, the difference compressed data of the target block is read from the difference data storage unit 371 and input to the case ID determination unit 373 and the variable length decoding unit 374, and the process proceeds to step S322.

In step S322, among the reference images stored in the reference storage unit 372, before and immediately after the frame of the target image V _T are respectively set to Past reference image V _P and Futuresse reference image V _F, the correlation maximum position The data is input to the detection unit 377 and the estimation unit 378, and the process proceeds to step S323.

Here, for example, when one of the blocks f ₂ , f ₃ , and f ₄ of the target image shown at the bottom of FIG. 80 is the target block, the frames f ₁ and f of the reference image Of frames ₅ , f ₉ , and f ₁₃ , the frames f ₁ and f ₅ immediately before and immediately after the frames f ₂ , f ₃ , and f ₄ of the target image are used as a past reference image and a feature reference image, respectively.

Also, for example, when any of the blocks f ₆ , f ₇ , and f ₈ of the target image is the target block, the frames of the reference image frames f ₁ , f ₅ , f ₉ , and f ₁₃ The frames f ₅ and f ₉ immediately before and after the frames f ₆ , f ₇ , and f ₈ of the target image are set as a past reference image and a feature reference image, respectively.

Further, for example, when any of the blocks f ₁₀ , f ₁₁ , and f ₁₂ of the target image is the target block, the frames of the reference image frames f ₁ , f ₅ , f ₉ , and f ₁₃ The frames f ₉ and f ₁₃ immediately before and after the frames f ₁₀ , f ₁₁ , and f ₁₂ of the target image are set as a past reference image and a feature reference image, respectively.

  In step S323, the case ID determination unit 373 analyzes the case ID added to the differential compression data of the block of interest, obtains the value of the case ID, and proceeds to step S324.

  In step S324, the variable length decoding unit 374 analyzes the differential compression data of the block of interest, and decodes the quantized data that is a variable length code in the differential compression data (variable length decoding). The quantized data obtained by performing variable length decoding in the variable length decoding unit 374 is supplied from the variable length decoding unit 374 to the inverse quantization unit 375. Note that there may be no quantized data (in the case of NULL).

  After the process of step S324, the process proceeds to step S325, and the inverse quantization unit 375 inversely quantizes the quantized data from the variable length decoding unit 374, and the resulting data on the frequency space, for example, DCT The coefficient is supplied to the conversion unit 376, and the process proceeds to step S325.

  In step S325, the transform unit 376 performs inverse DCT transform that transforms the DCT coefficient from the inverse quantization unit 375 into data in a two-dimensional space, and the image data (pixel value) of the block of interest and the result thereof. The difference value Sub (x, y) from the estimated value is supplied to the adding unit 379. In the variable length decoding in step S324, when there is no quantized data (in the case of NULL), the converting unit 376 supplies 0 as the difference value Sub (x, y) of the target block to the adding unit 379. .

  After the processing in step S326, the process proceeds to step S327 in FIG. 115, and the case ID determination unit 373 determines whether or not the case ID added to the differential compression data of the block of interest is 1, that is, the differential compression data of the block of interest. Is the first difference data.

  In step S327, when it is determined that the case ID added to the differential compression data of the target block is 1, that is, when the differential compression data of the target block is the first differential data, the case ID determination unit 373 supplies the determination result to the maximum correlation position detection unit 377, and proceeds to step S351 in FIG. 116. As will be described later, first differential data that is differentially compressed data of the target block is restored. The

  If it is determined in step S327 that the case ID added to the differential compression data of the target block is not 1, the process proceeds to step S328, and the case ID determination unit 373 adds the case ID to the differential compression data of the target block. It is determined whether or not the case ID is 2, that is, whether or not the differential compressed data of the block of interest is the second differential data.

  If it is determined in step S328 that the case ID added to the differential compressed data of the block of interest is 2, that is, if the differential compressed data of the block of interest is the second differential data, the case ID determination unit 373 Supplies the determination result to the maximum correlation position detection unit 377, and proceeds to step S361 in FIG. 117. As will be described later, second differential data that is differentially compressed data of the target block is restored. .

  If it is determined in step S328 that the case ID added to the differential compression data of the block of interest is not 2, the process proceeds to step S329, and the case ID determination unit 373 adds the case ID to the differential compression data of the block of interest. It is determined whether or not the case ID is 3, that is, whether or not the differential compressed data of the block of interest is the third differential data.

  If it is determined in step S329 that the case ID added to the differential compressed data of the block of interest is 3, that is, if the differential compressed data of the block of interest is the third differential data, the case ID determination unit 373 Supplies the determination result to the correlation maximum position detection unit 377 and proceeds to step S381 in FIG. 118, and the third difference data, which is the difference compressed data of the block of interest, is restored as will be described later. .

  If it is determined in step S329 that the case ID added to the differential compression data of the target block is not 3, the process proceeds to step S330, and the case ID determination unit 373 adds the case ID to the differential compression data of the target block. It is determined whether or not the case ID is 4, that is, whether or not the differential compressed data of the block of interest is the fourth differential data.

  If it is determined in step S330 that the case ID added to the differential compressed data of the block of interest is 4, that is, if the differential compressed data of the block of interest is the fourth differential data, the case ID determination unit 373 Supplies the determination result to the correlation maximum position detection unit 377, and proceeds to step S391 in FIG. 119. Hereinafter, as will be described later, fourth difference data that is differentially compressed data of the block of interest is restored. .

  If it is determined in step S330 that the case ID added to the differential compression data of the target block is not 4, the process proceeds to step S331, and the case ID determination unit 373 adds the case ID to the differential compression data of the target block. It is determined whether or not the case ID is 5, that is, whether or not the differential compressed data of the block of interest is the fifth differential data.

  If it is determined in step S331 that the case ID added to the differential compressed data of the block of interest is 5, that is, if the differential compressed data of the block of interest is the fifth differential data, the case ID determination unit 373 Supplies the determination result to the maximum correlation position detection unit 377, and proceeds to step S401 in FIG. 120. The fifth differential data, which is differential compressed data of the block of interest, is restored, as will be described later. .

  If it is determined in step S331 that the case ID added to the differential compression data of the block of interest is not 5, that is, the case ID added to the differential compression data of the block of interest is 6, and the block of interest If the difference compressed data is the sixth difference data, the case ID determination unit 373 supplies the determination result to the correlation maximum position detection unit 377 and proceeds to step S411 in FIG. 121, and will be described later. In addition, the sixth differential data that is the differentially compressed data of the block of interest is restored.

  Next, with reference to the flowchart of FIG. 116, restoration of the first difference data when the difference compressed data of the target block is the first difference data will be described.

When the correlation maximum position detection unit 377 receives the determination result indicating that the case ID is 1 from the case ID determination unit 373, the past reference image read from the reference storage unit 372 for the target block in step S351. in the V _P and Futuresse reference image V _F, it detects the correlation is high positional relationship. That is, the maximum correlation position detection unit 377 performs the correlation information of Equation (9) for each possible value of (U ′, V ′) for the target block, as in the correlation maximum position detection unit 254 of FIG. e ₁ (U ′, V ′) is calculated and the correlation represented by the correlation information e ₁ (U ′, V ′) is maximized (U ′, V ′), that is, the correlation information e ₁ (U ′, V ′) (U ′, V ′) that minimizes the value of V ′) is detected. Then, the maximum correlation position detection section 377, correlation information _{e 1 (U ', V'} ) to minimize the value of the (U ', V'), in the PAST reference image V _P and Futuresse reference image V _F, correlation Is supplied to the estimation unit 378 as a positional relationship vector (U, V) representing the positional relationship of the highest region, and the process proceeds from step S351 to S352.

In step S352, the estimation unit 378 calculates from the image data of the past reference image V _P and the feature reference image V _{F in} the positional relationship represented by the positional relationship vector (U, V) from the correlation maximum position detection unit 377. An estimated value of the block of interest is obtained and supplied to the adding unit 379.

That is, the estimating unit 378, the Pasto reference image V _P, from the target block vector and the shifted position, the region of interest blocks the same size, (- (4-s) V - (4-s) U,) in Futuresse reference image V _F, a vector from the target block (sU, sV) shifted by a position of the image data of the block of interest and the region of the same size, for example, a weighted average value, calculated according to equation (13) The weighted average value is obtained as the estimated value Pre (x, y) of the block of interest, supplied to the adding unit 379, and the process proceeds from step S352 to S353.

  In step S353, the addition unit 379 adds the difference value Sub (x, y) of the target block from the conversion unit 376 and the estimated value Pre (x, y) of the target block from the estimation unit 378, thereby The target block (image data) Ic (x, y) is restored, and the process proceeds to step S354, where the target block Ic (x, y) is output to the synthesizing unit 365 (FIG. 110).

  Next, with reference to the flowchart of FIG. 117, the restoration of the second differential data when the differential compressed data of the block of interest is the second differential data will be described.

  When the differential compressed data of the block of interest is the second differential data, the second differential data includes the correlation maximum vector (U, V) as described above. Therefore, in step S361, the variable length decoding unit 374 analyzes the second difference data that is the difference compressed data of the block of interest, and the correlation maximum vector (U, U) that is a variable length code in the second difference data. V) is variable length decoded. The maximum correlation vector (U, V) obtained by performing variable length decoding in the variable length decoding unit 374 is supplied from the variable length decoding unit 374 to the maximum correlation position detection unit 377.

Further, in step S361, the maximum correlation position detection unit 377 receives a determination result indicating that the case ID is 2 from the case ID determination unit 373, and in this case, the target block is read from the reference storage unit 372. in the Past reference image V _P and Futuresse reference image V _F was a correlation between the target block is detected with high positional relationship.

That is, the maximum correlation position detection unit 377 determines that a position having a high correlation with the target block in the past reference image V _P from the maximum correlation vector (U, V) from the variable length decoding unit 374 is a vector (− (4-s) U, - (4-s) and detects that the position shifted by V), in Futuresse reference image V _F, a high correlation position of the target block, from the target block vector (sU , sV) is detected, and vectors (-(4-s) U,-(4-s) V) and (sU, sV) representing the positional relationship are supplied to the estimation unit 378. Then, the process proceeds from step S361 to S362.

In step S362, the estimation unit 378 has the positions from the target block represented by vectors (− (4-s) U, − (4-s) V) and (sU, sV) from the correlation maximum position detection unit 377. An estimated value of the block of interest is obtained from the image data of the past reference image V _P and the feature reference image V _{F at} the position, and is supplied to the adding unit 379.

That is, the estimating unit 378, the Pasto reference image V _P, from the target block vector and the shifted position, the region of interest blocks the same size, (- (4-s) V - (4-s) U,) in Futuresse reference image V _F, a vector from the target block (sU, sV) shifted by a position of the image data of the block of interest and the region of the same size, for example, a weighted average value, calculated according to equation (13) The weighted average value is obtained as the estimated value Pre (x, y) of the target block, supplied to the adding unit 379, and the process proceeds from step S362 to S363.

  In step S363, the addition unit 379 adds the difference value Sub (x, y) of the target block from the conversion unit 376 and the estimated value Pre (x, y) of the target block from the estimation unit 378, thereby The target block Ic (x, y) is restored, and the process proceeds to step S364, where the target block Ic (x, y) is output to the synthesis unit 365 (FIG. 110).

  Next, with reference to the flowchart of FIG. 118, the restoration of the third difference data when the difference compressed data of the block of interest is the third difference data will be described.

When the differential compressed data of the block of interest is the third differential data, the third differential data includes the previous motion vector (−U ₁ , −V ₁ ) and the subsequent motion vector as described above. (U ₂ , V ₂ ) is included. Therefore, in step S381, the variable length decoding unit 374 analyzes the third difference data that is the difference compressed data of the block of interest, and the previous motion vector (-U) that is a variable length code in the third difference data. ₁ , −V ₁ ) and the back motion vector (U ₂ , V ₂ ) are variable length decoded. The front motion vector (−U ₁ , −V ₁ ) and the back motion vector (U ₂ , V ₂ ) obtained by performing variable length decoding in the variable length decoding unit 374 are correlated from the variable length decoding unit 374. The maximum position detection unit 377 is supplied.

Furthermore, in step S381, the correlation maximum position detection unit 377 receives a determination result indicating that the case ID is 3 from the case ID determination unit 373, and in this case, the block of interest is read from the reference storage unit 372. in each of the Past reference image V _P and Futuresse reference image V _F was a correlation between the target block is detected with high positional relationship.

That is, the maximum correlation position detection unit 377 indicates that the position having a high correlation with the target block in the past reference image V _P is a position shifted from the target block by the previous motion vector (−U ₁ , −V ₁ ). , before the motion vector (-U _1, -V ₁₎ from the variable length decoding unit 374 and detects from the Futuresse reference image V _F, a high correlation position of the target block, backward motion vector from the target block (U ₂ , V ₂ ) is detected from the back motion vector (U ₂ , V ₂ ) from the variable length decoding unit 374 and the positional relationship, that is, the front motion from the variable length decoding unit 374 is detected. The vector (−U ₁ , −V ₁ ) and the rear motion vector (U ₂ , V ₂ ) are supplied to the estimation unit 378, and the process proceeds from step S381 to S382.

In step S382, the estimation unit 378 indicates that the position from the target block is represented by the front motion vector (−U ₁ , −V ₁ ) and the back motion vector (U ₂ , V ₂ ) from the maximum correlation position detection unit 377. in some, from the image data of the region of Pasto reference image V _P and Futuresse reference image V _F, we obtain the estimated value of the block of interest, and supplies to the adder 379.

That is, the estimating unit 378, the Pasto reference image V _P, from the target block prior to the motion vector (-U _1, -V ₁₎ of a position shifted by a region of the block of interest and the same size, in Futuresse reference image V _F For example, the weighted average value of the image data of the region of the same size as the target block at the position shifted by the backward motion vector (U ₂ , V ₂ ) from the target block is calculated according to the equation (14), The weighted average value is obtained as the estimated value Pre (x, y) of the block of interest, supplied to the adding unit 379, and the process proceeds from step S382 to S383.

  In step S383, the addition unit 379 adds the difference value Sub (x, y) of the target block from the conversion unit 376 and the estimated value Pre (x, y) of the target block from the estimation unit 378, thereby The target block Ic (x, y) is restored, and the process proceeds to step S384, where the target block Ic (x, y) is output to the synthesis unit 365 (FIG. 110).

  Next, with reference to the flowchart of FIG. 119, description will be given of restoration of the fourth difference data when the difference compressed data of the target block is the fourth difference data.

When the differential compressed data of the block of interest is the fourth differential data, the fourth differential data includes the previous motion vector (−U ₁ , −V ₁ ) as described above. Therefore, in step S391, the variable length decoding unit 374 analyzes the fourth difference data that is the difference compressed data of the block of interest, and the previous motion vector (−U that is a variable length code in the fourth difference data). ₁ , -V ₁ ) is variable length decoded. The previous motion vector (−U ₁ , −V ₁ ) obtained by performing variable length decoding in the variable length decoding unit 374 is supplied from the variable length decoding unit 374 to the maximum correlation position detection unit 377.

Further, in step S391, the maximum correlation position detection unit 377 receives the determination result that the case ID is 4 from the case ID determination unit 373, and in this case, the attention block is read from the reference storage unit 372. in the Past reference image V _P, the correlation between the target block is detected with high positional relationship.

That is, the maximum correlation position detection unit 377 indicates that the position having a high correlation with the target block in the past reference image V _P is a position shifted from the target block by the previous motion vector (−U ₁ , −V ₁ ). , Detected from the previous motion vector (−U ₁ , −V ₁ ) from the variable length decoding unit 374, and the positional relationship, that is, the previous motion vector (−U ₁ , −V ₁ ) from the variable length decoding unit 374 is detected. , The process proceeds to step S392 from step S391.

In step S392, the estimation unit 378 determines the region of the past reference image V _P in which the position from the target block is at the position represented by the previous motion vector (−U ₁ , −V ₁ ) from the correlation maximum position detection unit 377. An estimated value of the target block is obtained from the image data and supplied to the adding unit 379.

That is, the estimating unit 378 cuts out the Past reference image V _P, from the target block prior to the motion vector (-U _1, -V ₁₎ of a position shifted by the image data of a region of the target block and the same size, the block of interest The estimated value Pre (x, y) is supplied to the adding unit 379, and the process proceeds from step S392 to S393.

  In step S393, the addition unit 379 adds the difference value Sub (x, y) of the target block from the conversion unit 376 and the estimated value Pre (x, y) of the target block from the estimation unit 378, thereby The target block Ic (x, y) is restored, and the process proceeds to step S394. The target block Ic (x, y) is output to the synthesis unit 365 (FIG. 110).

  Next, with reference to the flowchart of FIG. 120, restoration of the fifth difference data when the difference compressed data of the target block is the fifth difference data will be described.

When the differential compressed data of the block of interest is the fifth differential data, the fifth differential data includes the back motion vector (U ₂ , V ₂ ) as described above. Therefore, in step S401, the variable length decoding unit 374 analyzes the fifth difference data that is the difference compressed data of the block of interest, and the post motion vector (U ₂₎ that is a variable length code in the fifth difference data. , V ₂ ) is variable length decoded. The post motion vector (U ₂ , V ₂ ) obtained by performing variable length decoding in the variable length decoding unit 374 is supplied from the variable length decoding unit 374 to the maximum correlation position detection unit 377.

Further, in step S401, the maximum correlation position detection unit 377 receives the determination result that the case ID is 5 from the case ID determination unit 373, and in this case, the attention block is read from the reference storage unit 372. in the Futuresse reference image V _F, the correlation between the target block is detected with high positional relationship.

In other words, the maximum correlation position detection section 377, in Futuresse reference image V _F, that high correlation position of the block of interest, a position shifted by backward motion vector (U _2, V ₂₎ from the target block, the variable Detected from the back motion vector (U ₂ , V ₂ ) from the long decoding unit 374, and supplies the positional relationship, that is, the back motion vector (U ₂ , V ₂ ) from the variable length decoding unit 374 to the estimation unit 378. Then, the process proceeds from step S401 to S402.

In step S402, the estimation unit 378 has the image data of the area of the feature reference image V _F in which the position from the target block is at the position represented by the back motion vector (U ₂ , V ₂ ) from the maximum correlation position detection unit 377. Then, an estimated value of the target block is obtained and supplied to the adding unit 379.

That is, the estimating unit 378, the Futuresse reference image V _F, the position shifted by backward motion vector (U _2, V ₂₎ from the target block, the image data of the region of interest blocks the same size cut out, guess of the block of interest The value Pre (x, y) is supplied to the adding unit 379, and the process proceeds from step S402 to S403.

  In step S403, the addition unit 379 adds the difference value Sub (x, y) of the target block from the conversion unit 376 and the estimated value Pre (x, y) of the target block from the estimation unit 378, thereby The target block Ic (x, y) is restored, and the process proceeds to step S404, where the target block Ic (x, y) is output to the synthesis unit 365 (FIG. 110).

  Next, with reference to the flowchart in FIG. 121, restoration of the sixth difference data when the difference compressed data of the block of interest is the sixth difference data will be described.

  When the difference compressed data of the block of interest is the sixth difference data, the sixth difference data sets the estimated value Pre (x, y) of the block of interest to 0, and the block of interest Ic (x, y) And the estimated value Pre (x, y).

  Therefore, when the determination result indicating that the case ID is 6 is supplied from the case ID determination unit 373, the maximum correlation position detection unit 377 supplies the determination result to the estimation unit 378, and the estimation unit 378 When the determination result that the case ID is 6 is supplied from the maximum correlation position detection unit 377, 0 is supplied to the addition unit 379 as the estimated value Pre (x, y) of the block of interest.

  In this case, in step S411, the adding unit 379 calculates the difference value Sub (x, y) of the target block from the conversion unit 376 (here, equal to the image data Ic (x, y) of the target block), The estimated value Pre (x, y) (here, 0) of the target block from the estimation unit 378 is added, thereby restoring the target block Ic (x, y), and the synthesis unit 365 (FIG. 110).

  For example, when the display device 3 (FIG. 24) can only display at a frame rate of 60 fps, the receiving device of FIG. 110 receives only the bit stream from the input terminal 361 in the decompression circuit 363. The restored 60fps moving image data may be output as it is to the display device 3 via the synthesizing unit 365.

  In addition, when the display device 3 (FIG. 24) can only display at a frame rate of 120 fps, for example, the receiving device of FIG. 110 may perform the following processing.

  That is, the decompression circuit 363 restores the bit stream from the input terminal 361 to 60 fps moving image data and supplies it to the synthesis unit 365. Also, the differential information restoring unit 364 restores the differential compressed data of the block of the frame with the variable s described in FIG. 82 out of the differential compressed data from the input terminal 362 and supplies the restored differential data to the synthesizing unit 365.

In this case, for example, the frame of 60 fps moving image data shown second from the top in FIG. 80..., F ₁ , f ₅ , f ₉ , f ₁₃ ,.・ Is supplied. Further, for example, the frame of 240-60 fps moving image data shown at the bottom of FIG. 80..., F ₂ , f ₃ , f ₄ , f ₆ , Of f ₇ , f ₈ , f ₁₀ , f ₁₁ , f ₁₂ ,..., frames..., f ₃ , f ₇ , f ₁₁ ,.

Therefore, in the synthesis unit 365, frames of 60 fps moving image data from the decompression circuit 363,..., F ₁ , f ₅ , f ₉ , f ₁₃ ,. Select f ₃ , f ₇ , f ₁₁ ,... in the order of frame..., f ₁ , f ₃ , f ₅ , f ₇ , f ₉ , f ₁₁ , f ₁₃ ,. By doing so, it is possible to obtain moving image data having a frame rate of 120 fps. The moving image data having a frame rate of 120 fps may be output to the display device 3.

  In this manner, the receiving device 2 in FIG. 110 can obtain moving image data having an appropriate frame rate according to the display capability (frame rate) of the display device 3 (FIG. 24).

  As described above, the transmission apparatus 1 in FIG. 79 highly compresses high frame rate moving picture data such as 240 fps moving picture data, for example, and the receiving apparatus 2 in FIG. Frame rate video data can be restored.

In particular, regarding the first difference data, when restoring a high frame rate (for example, 240 fps) moving image, attention is paid to one image in the high frame rate moving image to be restored. A plurality of (for example, two) “images constituting a low frame rate (for example, the above-described past reference image and feature reference image)” that are close in time to this noticed “image to be restored” (target image) And a portion (for example, correlation information e ₁ (U ′, V ′) in Expression (9)) that has a high correlation in the extracted images is minimized (U ′, V ′). ) To restore the “image you want to restore”. Thereby, there is an advantage that it is not necessary to transmit a motion vector from the compression side (transmission device 1) to the decompression side (reception device 2). This is effective in reducing the amount of data.

Further, the first difference data that does not include a motion vector may fail to be restored when there are a plurality of positional relationships with high correlation in the past reference image and the future reference image. Therefore, in order to deal with such a case, in the transmission apparatus 1, in addition to the first difference data, the second difference data including the correlation maximum vector (U, V), and the previous motion vector (−U ₁ , −V ₁ ) and the back motion vector (U ₂ , V ₂ ), the third difference data, the front motion vector (−U ₁ , −V ₁ ), the fourth difference data, and the back motion vector The fifth difference data including (U ₂ , V ₂ ) and the sixth difference data that can be restored without referring to the reference image are adopted, and the first to sixth difference data are included as necessary. Select one of them. Therefore, it is possible to prevent a failure in restoring the moving image data.

  Further, in the transmission apparatus 1 of FIG. 79, data relating to a low frame rate moving image (the above-described bit stream) and data required for restoring a high frame rate moving image from the low frame rate moving image (the above difference) 110 is received separately by the receiving apparatus 2 in FIG. 110, so that the low frame rate moving image can be restored. In addition, the receiving apparatus 2 receives both “data relating to a low frame rate moving image” and “data necessary for restoring a high frame rate moving image from the low frame rate moving image”, whereby a high frame rate is obtained. Can be restored. In other words, so-called temporal scalability can be provided.

  Next, the series of processes described above can be performed by dedicated hardware or by software. When a series of processing is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

  Therefore, FIG. 122 shows a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.

  A CPU (Central Processing Unit) 401, which is the main controller of the computer, is connected to each unit via a bus 408, and executes the application program under the control of the operating system (OS) to thereby execute the above-described operation. Perform a series of processing. The application program executed by the CPU 401 includes an application program for performing the above-described series of processing. The moving image data downloaded to the HDD (Hard disk Drive) 414 via 407 is processed.

  The memory 402 is a storage device used for storing program codes executed by the CPU 401 and temporarily storing work data being executed. The memory 402 includes both a nonvolatile memory such as a ROM (Read Only Memory) and a volatile memory such as a DRAM (Dynamic Random Access Memory).

  The display controller 403 is a dedicated controller for actually processing a drawing command issued by the CPU 401. The drawing data processed in the display controller 403 is temporarily written in a frame buffer (not shown), for example, and then output on the screen by the display 411. For example, the moving image reproduced from the HDD 414 is displayed on the screen of the display 411 as described above.

  The input device interface 404 is a device for connecting user input devices such as a keyboard 412 and a mouse 413 to a computer. A user can input a command or the like for playing back a moving image via the keyboard 412 or the mouse 413.

  The network interface 405 can connect the computer to a local network such as a LAN (Local Area Network) and further to a wide area network such as the Internet according to a predetermined communication protocol such as Ethernet (registered trademark). On the network, a plurality of host terminals and servers (not shown) are connected in a transparent state, and a distributed computing environment is constructed. On the network, distribution services such as a software program including an application program for executing the above-described series of processing, a data content including the above-described bit stream (encoded data), and differentially compressed data can be provided. In addition, the computer receives, for example, moving image data, the above-described bit stream, and differentially compressed data from the server storing moving images taken by others by the network interface 405 via the network and downloads them to the HDD 414. can do.

  The external device interface 407 is a device for connecting external devices such as the HDD 414 and the media drive 415 to the computer.

  The HDD 414 is a random-accessible external storage device in which a magnetic disk as a storage medium is fixedly mounted, and is superior to other external storage devices in terms of storage capacity and data transfer speed. The HDD 414 is installed with an application program that executes the above-described series of processing. Here, “install” means placing the software program on the HDD 414 in an executable state. The HDD 414 stores the operating system program code to be executed by the CPU 401, application programs, device drivers, and the like in a nonvolatile manner. The application program can be downloaded from the portable medium 416 or via the network interface 405 and installed in the HDD 414. The application program can be installed in the HDD 414 in advance.

  The media drive 415 is a device for loading portable media 416 such as CD (Compact Disc), MO (Magneto-Optical disc), DVD (Digital Versatile Disc), etc., and accessing the data recording surface.

  The portable media 416 is used mainly for the purpose of backing up software programs, data files, and the like as data in a computer-readable format, and for moving them between systems (that is, including sales, distribution, and distribution). . An application program that executes the above-described series of processing and data obtained by executing the application program can be physically distributed and distributed among a plurality of devices by using the portable medium 416. Data can also be distributed via the network interface 405.

  The VTR interface 409 is a device for taking a moving image reproduced from the VTR 410 into the computer.

  Here, as the computer shown in FIG. 122, a compatible computer or a successor of IBM's personal computer “PC / AT (Personal Computer / Advanced Technology)” can be employed. Of course, you may employ | adopt the computer provided with the other architecture.

  Note that although the transmission device 1 in FIG. 79 processes 240 fps moving image data, the frame rate of the moving image data processed by the transmission device 1 is not limited to 240 fps. Further, the method of separating the moving image data in the separation circuit 213 is not limited to the method of separating the moving image data into 60 fps moving image data and the remaining 240-60 fps moving image data. That is, the separation circuit 213 can separate, for example, 240 fps moving image data into 30 fps or 120 fps moving image data and the remaining moving image data.

  Further, “frame” in the above description can be read as “field”.

  Furthermore, in the present embodiment, the estimated value of the target block is obtained using the past reference image or the future reference image. However, the estimated value of the target block is further in time before the past reference image. It is also possible to use a frame or a frame that is temporally after the future reference image.

  79 obtains the first to sixth difference data for each block, and selects the data that is the block compression result from the first to sixth difference data. Alternatively, only the first difference data or only the second difference data may be obtained, and the first or second difference data may be used as the block compression result (difference compressed data) as it is. Further, the transmission device 1 may obtain the first difference data and one or more of the second to sixth difference data, and may select a block compression result from the first difference data. Further, the transmission apparatus 1 obtains the second difference data, the first difference data, and one or more of the third to sixth difference data, and selects the one that becomes the block compression result from the second difference data May be.

It is a figure which shows the moving image data in the three-dimensional space which added the time direction (t) to the two-dimensional plane (x, y). It is a figure which shows the range on the frequency domain of the moving image which can be recognized by human vision. It is a figure which shows the area | region on the frequency domain where the data of the part of the moving image which are still are distributed. Subject is a diagram showing a region on the frequency domain data is distributed in the moving parts at a rate _{(r 0 / t 0) /} 2 approximately. It is a waveform diagram showing a moving subject at a rate _{(r 0 / t 0) /} 2. It is a diagram showing a moving subject at a rate _{(r 0 / t 0) /} 2. It is a diagram for explaining a portion of a frequency domain data of the moving parts at a rate _{(r 0 / t 0) /} 2 approximately. Subject is a diagram showing a region on the frequency domain data is distributed in the moving parts in the order rate r ₀ / t _0. Subject is a diagram showing a region on the frequency domain data is distributed in the moving parts in the order rate 2r ₀ / t _0. It is a figure explaining the area | region on the frequency domain which a human can recognize about the still part of a moving image. It is a diagram for explaining a region of the frequency domain that can recognize human the portion where the subject is moving at a speed _{(r 0 / t 0) /} 2 approximately. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate 2r ₀ / t _0. It is a diagram for explaining a region of the frequency domain that can recognize human the portion where the subject is moving at a speed _{(r 0 / t 0) /} 2 approximately. It is a diagram for explaining a region of the frequency domain that can recognize human the portion where the subject is moving at a speed _{(r 0 / t 0) /} 2 approximately. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate 2r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate 2r ₀ / t _0. It is a figure explaining the area | region on the frequency domain which a human can recognize about the still part of a moving image. It is a diagram for explaining a region of the frequency domain that can recognize human the portion where the subject is moving at a speed _{(r 0 / t 0) /} 2 approximately. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate 2r ₀ / t _0. It is a block diagram which shows the structural example of one Embodiment of the image processing system to which this invention is applied. 3 is a block diagram illustrating a first configuration example of a transmission apparatus 1. FIG. 4 is a flowchart for explaining processing of the transmission device 1. It is a figure explaining the moving image data which the transmitter 1 outputs. It is a figure explaining the moving image data which the transmitter 1 outputs. It is a figure explaining the moving image data which the transmitter 1 outputs. It is a figure explaining the moving image data which the transmitter 1 outputs. 5 is a flowchart for explaining processing of a filter generation unit 23. It is a flowchart explaining the method of acquiring a principal component direction. 3 is a block diagram illustrating a first configuration example of a receiving device 2. FIG. 5 is a flowchart for explaining processing of the reception device 2. FIG. 6 is a diagram for explaining moving image data after upsampling in the receiving apparatus 2; FIG. 6 is a diagram for explaining moving image data after upsampling in the receiving apparatus 2; FIG. 6 is a diagram for explaining moving image data after upsampling in the receiving apparatus 2; FIG. 6 is a diagram for explaining moving image data after upsampling in the receiving apparatus 2; It is a figure explaining the integration by an electronic shutter. It is a figure explaining the pass band of a filter. It is a figure explaining the pass band of a filter. It is a figure explaining the pass band of a filter. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. 3 is a block diagram illustrating a configuration example of a principal component direction acquisition unit 31. FIG. It is a figure which shows correlation information. It is a figure which shows the correlation information after scaling. It is a figure which shows synthetic | combination correlation information. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. 5 is a flowchart illustrating processing of a principal component direction acquisition unit 31. 5 is a flowchart illustrating processing of a principal component direction acquisition unit 31. 5 is a flowchart illustrating processing of a principal component direction acquisition unit 31. 5 is a flowchart illustrating processing of a principal component direction acquisition unit 31. 4 is a flowchart for explaining processing of a filter unit 22; It is a figure which shows the to-be-photographed subject and the to-be-moved subject. FIG. 6 is a waveform diagram showing moving image data in which a stationary subject and a moving subject are projected. FIG. 6 is a waveform diagram showing moving image data in which a stationary subject and a moving subject are projected. 10 is a block diagram illustrating another configuration example of the principal component direction acquisition unit 31. FIG. 5 is a flowchart for explaining processing of a filter generation unit 23. 5 is a flowchart for explaining processing of a filter generation unit 23. 5 is a flowchart for explaining processing of a filter generation unit 23. 5 is a flowchart for explaining processing of a filter generation unit 23. 5 is a flowchart for explaining processing of a filter generation unit 23. 4 is a flowchart for explaining processing of a filter unit 22; 6 is a block diagram illustrating a second configuration example of the transmission device 1. FIG. It is a figure which shows 240fps moving image data, 60fps moving image data, and 240-60fps moving image data. 5 is a block diagram illustrating a configuration example of a difference information extraction unit 217. FIG. FIG. 6 is a diagram showing a target image V _T , a past reference image V _P , and a future reference image V _F. It is a wave form diagram which shows a to-be-photographed object. FIG. 6 is a waveform diagram showing a target image V _T , a past reference image V _P , and a future reference image V _F. FIG. 6 is a waveform diagram showing a past reference image V _P and a feature reference image V _F. FIG. 6 is a waveform diagram showing a past reference image V _P and a feature reference image V _F. FIG. 6 is a waveform diagram showing a target image V _T , a past reference image V _P , and a future reference image V _F. FIG. 6 is a waveform diagram showing a target image V _T , a past reference image V _P , and a future reference image V _F. FIG. 6 is a waveform diagram showing a target image V _T , a past reference image V _P , and a future reference image V _F. FIG. 6 is a waveform diagram showing a target image V _T , a past reference image V _P , and a future reference image V _F. FIG. 6 is a waveform diagram showing a target image V _T , a past reference image V _P , and a future reference image V _F. FIG. 6 is a waveform diagram showing a target image V _T , a past reference image V _P , and a future reference image V _F. FIG. 6 is a waveform diagram showing a target image V _T , a past reference image V _P , and a future reference image V _F. 6 is a block diagram illustrating a configuration example of a difference data calculation unit 234. FIG. 5 is a flowchart for explaining processing of a difference data calculation unit 234. 4 is a block diagram illustrating a configuration example of a difference data calculation unit 235. FIG. 5 is a flowchart for explaining processing of a difference data calculation unit 235. 5 is a block diagram illustrating a configuration example of a difference data calculation unit 236. FIG. 5 is a flowchart for explaining processing of a difference data calculation unit 236. 6 is a block diagram illustrating a configuration example of a difference data calculation unit 237. FIG. It is a flowchart explaining the process of the difference data calculation part 237. 6 is a block diagram illustrating a configuration example of a difference data calculation unit 238. FIG. It is a flowchart explaining the process of the difference data calculation part 238. It is a block diagram which shows the structural example of the difference data calculation part 239. It is a flowchart explaining the process of the difference data calculation part 239. FIG. It is a figure which shows the data structure of the 1st thru | or 6th difference data. It is a figure which shows the data structure of the 1st thru | or 6th difference data to which case ID was added. 4 is a flowchart for explaining processing of the transmission device 1. It is a flowchart explaining compression of the target image which the difference information extraction part 217 performs. 6 is a block diagram illustrating a second configuration example of the reception device 2. FIG. 5 is a flowchart for explaining processing of the reception device 2. It is a flowchart explaining the decoding of 240-60fps moving image data which the difference information decompression | restoration part 364 performs. 5 is a block diagram illustrating a configuration example of a difference information restoration unit 364. FIG. It is a flowchart explaining the process which calculates | requires the pixel value of the block which the difference information decompression | restoration part 364 performs. It is a flowchart explaining the process which calculates | requires the pixel value of the block which the difference information decompression | restoration part 364 performs. It is a flowchart explaining the process which calculates | requires the pixel value of the block which the difference information decompression | restoration part 364 performs. It is a flowchart explaining the process which calculates | requires the pixel value of the block which the difference information decompression | restoration part 364 performs. It is a flowchart explaining the process which calculates | requires the pixel value of the block which the difference information decompression | restoration part 364 performs. It is a flowchart explaining the process which calculates | requires the pixel value of the block which the difference information decompression | restoration part 364 performs. It is a flowchart explaining the process which calculates | requires the pixel value of the block which the difference information decompression | restoration part 364 performs. It is a flowchart explaining the process which calculates | requires the pixel value of the block which the difference information decompression | restoration part 364 performs. It is a block diagram which shows the structural example of one Embodiment of the computer to which this invention is applied.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Transmission apparatus, 2 Reception apparatus, 3 Display apparatus, 11 Recording medium, 12 Transmission medium, 21 Buffer part, 22 Filter part, 23 Filter production | generation part, 24 Encoding part, 31 Main component direction acquisition part, 32 Filter information supply part, 50 Decoding unit, 51 Buffer unit, 52 Filter unit, 53 Filter generation unit, 61 Principal component direction acquisition unit, 62 Filter information supply unit, 101 Buffer unit, 102 Block extraction unit, 103 Correlation calculation unit, 104 Scaling synthesis unit, 105 Minimum value detection unit, 124 synthesis unit, 125 minimum value detection unit, 211 input terminal, 212 band limiting filter unit, 213 separation circuit, 214 compression circuit, 215 output terminal, 216 decompression circuit, 217 difference information extraction unit, 218 output terminal , 22 Reference storage unit, 222 Target storage unit, 223 Data processing unit, 231 to 233 input terminal, 234 to 239 Difference data calculation unit, 240 selection circuit, 241 output terminal, 251 to 253 input terminal, 254 Correlation maximum position detection unit, 255 Average value calculation unit, 256 subtraction unit, 257 conversion unit, 258 quantization unit, 259 variable length coding unit, 260 output terminal, 271 to 273 input terminal, 274 maximum correlation position detection unit, 275 average value calculation unit, 276 subtraction Unit, 277 conversion unit, 278 quantization unit, 279 variable length coding unit, 280 output terminal, 291 to 293 input terminal, 294,295 correlation maximum position detection unit, 296 average value calculation unit, 297 subtraction unit, 298 conversion unit , 299 Quantizer, 00 variable length encoding unit, 301 output terminal, 311, 312 input terminal, 313 maximum correlation position detection unit, 314 clipping unit, 315 subtraction unit, 316 conversion unit, 317 quantization unit, 318 variable length encoding unit, 319 output Terminal, 331, 332 input terminal, 333 maximum correlation position detection unit, 334 extraction unit, 335 subtraction unit, 336 conversion unit, 337 quantization unit, 338 variable length coding unit, 339 output terminal, 351 input terminal, 352 conversion unit , 353 quantization unit, 354 variable length coding unit, 355 output terminal, 361, 362 input terminal, 363 decompression circuit, 364 differential information restoration unit, 365 synthesis unit, 366 output terminal, 371 differential data storage unit, 372 reference storage Part, 373 case ID judgment part, 374 variable length decoding unit, 375 inverse quantization unit, 376 conversion unit, 377 correlation maximum position detection unit, 378 estimation unit, 379 addition unit, 401 CPU, 402 memory, 403 display controller, 404 input device interface, 405 network interface, 407 External device interface, 409 VTR interface, 410 VTR, 411 display, 412 keyboard, 413 mouse, 414 HDD, 415 media drive, 416 portable media

Claims (39)

In an image processing apparatus that processes video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separation means for separating data,
Detecting means for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating means for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected by the detecting means. When,
An image processing apparatus comprising: a compression unit that compresses the third moving image data in units of blocks using the estimated value of the block. The output of the second moving image data and the compression result of the third moving image data by the compression unit are independently output as the compression result of the first moving image data. Image processing device. Other detection means for detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for the block;
Other estimation means for obtaining another estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
Other compression means for compressing the third moving image data in units of blocks using another estimated value of the block;
Selecting either one of the compressed data obtained by compressing the third moving image data by the compressing unit or the compressed data obtained by compressing the third moving image data by the other compressing unit; The image processing apparatus according to claim 1, further comprising: selection means for adding identification information for identifying the compressed data and outputting the identification data. The selection unit has a data amount of compressed data obtained by compression of the third moving image data by the compression unit or compressed data obtained by compression of the third moving image data by the other compression unit. The image processing apparatus according to claim 3, wherein a smaller amount of compressed data is selected. The image processing apparatus according to claim 3, wherein the second moving image data and the compressed data to which the identification information is added are independently output as a compression result of the first moving image data. The estimation unit obtains a weighted average value of image data of a plurality of frames of the second moving image data in the positional relationship detected by the detection unit as an estimation value of the block. Item 8. The image processing apparatus according to Item 1. In the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction, the region extends in the principal component direction that is the direction of the principal component of the first moving image data, and the frequency axis in the time direction Filter means for filtering the first moving image data, with a region having a specific width in the direction as a passband,
The image processing apparatus according to claim 1, wherein the separation unit separates the first moving image data after filtering by the filter unit into the second and third moving image data. In an image processing method for processing video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. A separation step for separating data,
A detection step of detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. When,
A compression step of compressing the third moving image data in units of blocks using the estimated value of the block. Another detection step of detecting one motion vector representing a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data for the block;
Another estimation step for obtaining another estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
Another compression step of compressing the third moving image data in units of blocks using another estimated value of the block;
One of the compressed data obtained by compressing the third moving image data by the compression step or the compressed data obtained by compressing the third moving image data by the other compression step is selected. The image processing method according to claim 8, further comprising: a selection step of adding identification information for identifying the compressed data and outputting the identification data. In a program that causes a computer to process video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. A separation step for separating data,
A detection step of detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. When,
A compression step of compressing the third moving image data in units of blocks using the estimated value of the block. Another detection step of detecting one motion vector representing a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data for the block;
Another estimation step for obtaining another estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
Another compression step of compressing the third moving image data in units of blocks using another estimated value of the block;
One of the compressed data obtained by compressing the third moving image data by the compression step or the compressed data obtained by compressing the third moving image data by the other compression step is selected. The program according to claim 10, further comprising a selection step of adding identification information for identifying the compressed data and outputting the identification data. In a program recording medium in which a program for causing a computer to process video data is recorded,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. A separation step for separating data,
A detection step of detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. When,
And a compressing step of compressing the third moving image data in units of blocks using the estimated value of the block. Another detection step of detecting one motion vector representing a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data for the block;
Another estimation step for obtaining another estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
Another compression step of compressing the third moving image data in units of blocks using another estimated value of the block;
One of the compressed data obtained by compressing the third moving image data by the compression step or the compressed data obtained by compressing the third moving image data by the other compression step is selected. The program recording medium according to claim 12, further comprising: a selection step of adding and outputting identification information for identifying the compressed data. In the data structure of data obtained by compressing video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
Detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data having a high correlation with the correlation,
Compressed data obtained by compressing the block using the estimated value of the block;
A data structure comprising: the second moving image data. For another block of the third moving image data, detect one motion vector representing a positional relationship having a high correlation with the other block in a plurality of frames of the second moving image data;
Obtaining an estimated value of the other block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the other block is a positional relationship obtained from the one motion vector,
Further including other compressed data obtained by compressing the other block using the estimated value of the other block;
The compressed data and the other compressed data include identification information for identifying each,
The data structure according to claim 14, wherein the other compressed data further includes the one motion vector. In a data recording medium on which data obtained by compressing moving image data is recorded,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
Detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data having a high correlation with the correlation,
Compressed data obtained by compressing the block using the estimated value of the block;
A data recording medium on which data is recorded, comprising the second moving image data. For another block of the third moving image data, detect one motion vector representing a positional relationship having a high correlation with the other block in a plurality of frames of the second moving image data;
Obtaining an estimated value of the other block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the other block is a positional relationship obtained from the one motion vector,
Further including other compressed data obtained by compressing the other block using the estimated value of the other block;
The compressed data and the other compressed data include identification information for identifying each,
The data recording medium according to claim 16, wherein the other compressed data further includes the one motion vector. In an image processing apparatus that processes video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
Detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data having a high correlation with the correlation,
First restoration means for restoring compressed data obtained by compressing the block using the estimated value of the block to the third moving image data using the second moving image data;
A second restoring means for synthesizing the second and third moving image data and restoring the first moving image data;
The first restoration means includes
Detecting means for detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating means for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected by the detecting means. When,
An image processing apparatus comprising: a block restoration unit that restores the third moving image data in units of blocks using the estimated value of the block and the compressed data. The block is
For the block, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
Other compressed data obtained by compressing the block using the estimated value of the block,
Or compressed into the compressed data,
The compressed data and the other compressed data include identification information for identifying each,
In the case where the other compressed data further includes the one motion vector,
When the identification information represents the compressed data,
The detecting means detects a positional relationship having a high correlation in a plurality of frames of the second moving image data;
The estimation means estimates a block obtained by dividing a frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected by the detection means. Find the value
The block restoration means restores the block using the estimated value of the block and the compressed data,
When the identification information represents the other compressed data,
The detection means detects a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data for the block from the one motion vector,
The estimation means determines an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship determined from the one motion vector,
The image processing apparatus according to claim 18, wherein the block restoration unit restores the block using an estimated value of the block and the other compressed data. The estimation unit obtains a weighted average value of image data of a plurality of frames of the second moving image data in the positional relationship detected by the detection unit as an estimation value of the block. Item 19. The image processing apparatus according to Item 18. In an image processing method for processing video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
Detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data having a high correlation with the correlation,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving image data using the second moving image data;
Combining the second and third moving image data and restoring the first moving image data;
The first restoration step includes
A detection step of detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. When,
An image processing method comprising: a block restoration step of restoring the third moving image data in units of blocks using the estimated value of the block and the compressed data. The block is
For the block, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
Other compressed data obtained by compressing the block using the estimated value of the block,
Or compressed into the compressed data,
The compressed data and the other compressed data include identification information for identifying each,
In the case where the other compressed data further includes the one motion vector,
When the identification information represents the compressed data,
In the detecting step, a positional relationship having a high correlation is detected in a plurality of frames of the second moving image data,
In the estimating step, a block obtained by dividing a frame of the third moving image data into blocks from a plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. Find the value
In the block restoration step, the block is restored using the estimated value of the block and the compressed data,
When the identification information represents the other compressed data,
In the detection step, a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data is detected for the block from the one motion vector,
In the estimating step, an estimated value of the block is obtained from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
The image processing method according to claim 21, wherein in the block restoration step, the block is restored using the estimated value of the block and the other compressed data. In a program that causes a computer to process video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
Detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data having a high correlation with the correlation,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving image data using the second moving image data;
Combining the second and third moving image data and restoring the first moving image data;
The first restoration step includes
A detection step of detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. When,
A block restoring step of restoring the third moving image data in units of blocks using the estimated value of the block and the compressed data. The block is
For the block, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
Other compressed data obtained by compressing the block using the estimated value of the block,
Or compressed into the compressed data,
The compressed data and the other compressed data include identification information for identifying each,
In the case where the other compressed data further includes the one motion vector,
When the identification information represents the compressed data,
In the detecting step, a positional relationship having a high correlation is detected in a plurality of frames of the second moving image data,
In the estimating step, a block obtained by dividing a frame of the third moving image data into blocks from a plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. Find the value
In the block restoration step, the block is restored using the estimated value of the block and the compressed data,
When the identification information represents the other compressed data,
In the detection step, a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data is detected for the block from the one motion vector,
In the estimating step, an estimated value of the block is obtained from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
The program according to claim 23, wherein in the block restoration step, the block is restored using the estimated value of the block and the other compressed data. In a program recording medium in which a program for causing a computer to process video data is recorded,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
Detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Obtaining an estimated value of a block obtained by dividing a frame of the third moving image data from a plurality of frames of image data of the second moving image data having a high correlation with the correlation,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving image data using the second moving image data;
Combining the second and third moving image data and restoring the first moving image data;
The first restoration step includes
A detection step of detecting a positional relationship having a high correlation in a plurality of frames of the second moving image data;
Estimating step for obtaining an estimated value of a block obtained by dividing the frame of the third moving image data from the plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. When,
A program recording medium on which a program is recorded, comprising: a block restoring step of restoring the third moving image data in units of blocks using the estimated value of the block and the compressed data. The block is
For the block, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
Other compressed data obtained by compressing the block using the estimated value of the block,
Or compressed into the compressed data,
The compressed data and the other compressed data include identification information for identifying each,
In the case where the other compressed data further includes the one motion vector,
When the identification information represents the compressed data,
In the detecting step, a positional relationship having a high correlation is detected in a plurality of frames of the second moving image data,
In the estimating step, a block obtained by dividing a frame of the third moving image data into blocks from a plurality of frames of image data of the second moving image data in the positional relationship detected in the detecting step. Find the value
In the block restoration step, the block is restored using the estimated value of the block and the compressed data,
When the identification information represents the other compressed data,
In the detection step, a positional relationship having a high correlation with the block in the plurality of frames of the second moving image data is detected for the block from the one motion vector,
In the estimating step, an estimated value of the block is obtained from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
26. The program recording medium according to claim 25, wherein in the block restoration step, the block is restored using the estimated value of the block and the other compressed data. In an image processing apparatus that processes video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separation means for separating data,
Detecting means for detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for a block obtained by dividing the frame of the third moving image data into blocks;
Inference means for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
An image processing apparatus comprising: a compression unit that compresses the third moving image data in units of blocks using the estimated value of the block. The said 2nd moving image data and the compression result of the said 3rd moving image data by the said compression means are output independently as a compression result of the said 1st moving image data. Image processing device. The estimation means uses, as an estimated value of the block, a weighted average value of image data of a plurality of frames of the second moving image data in which the positional relationship with the block is a positional relationship obtained from the one motion vector. The image processing apparatus according to claim 27, wherein the image processing apparatus is obtained. In the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction, the region extends in the principal component direction that is the direction of the principal component of the first moving image data, and the frequency axis in the time direction Filter means for filtering the first moving image data, with a region having a specific width in the direction as a passband,
The image processing apparatus according to claim 27, wherein the separating unit separates the first moving image data after filtering by the filtering unit into the second and third moving image data. In an image processing method for processing video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. A separation step for separating data,
A detection step of detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for a block obtained by dividing the frame of the third moving image data.
An estimation step of obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
A compression step of compressing the third moving image data in units of blocks using the estimated value of the block. In a program that causes a computer to process video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. A separation step for separating data,
A detection step of detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for a block obtained by dividing the frame of the third moving image data.
An estimation step of obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
A compression step of compressing the third moving image data in units of blocks using the estimated value of the block. In a program recording medium in which a program for causing a computer to process video data is recorded,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. A separation step for separating data,
A detection step of detecting one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for a block obtained by dividing the frame of the third moving image data.
An estimation step of obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
And a compressing step of compressing the third moving image data in units of blocks using the estimated value of the block. In the data structure of data obtained by compressing video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
For a block obtained by dividing the frame of the third moving image data into blocks, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
Compressed data obtained by compressing the block using the estimated value of the block;
Including the second moving image data,
The compressed data includes the one motion vector. In a data recording medium on which data obtained by compressing moving image data is recorded,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
For a block obtained by dividing the frame of the third moving image data into blocks, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
Compressed data obtained by compressing the block using the estimated value of the block;
Including the second moving image data,
The compressed data includes the one motion vector. A data recording medium on which data is recorded. In an image processing apparatus that processes video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
For a block obtained by dividing the frame of the third moving image data into blocks, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
First restoration means for restoring compressed data obtained by compressing the block using the estimated value of the block to the third moving image data using the second moving image data;
A second restoring means for synthesizing the second and third moving image data and restoring the first moving image data;
The first restoration means includes
Detecting means for obtaining a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data from the one motion vector;
An estimation means for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
An image processing apparatus comprising: a block restoration unit that restores the block using the estimated value of the block and the compressed data. In an image processing method for processing video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
For a block obtained by dividing the frame of the third moving image data into blocks, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving image data using the second moving image data;
Combining the second and third moving image data and restoring the first moving image data;
The first restoration step includes
From the one motion vector, a detection step for obtaining a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for the block;
An estimation step for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
An image processing method comprising: a block restoration step of restoring the block using the estimated value of the block and the compressed data. In a program that causes a computer to process video data,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
For a block obtained by dividing the frame of the third moving image data into blocks, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving image data using the second moving image data;
Combining the second and third moving image data and restoring the first moving image data;
The first restoration step includes
From the one motion vector, a detection step for obtaining a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for the block;
An estimation step for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
A block restoration step of restoring the block using the estimated value of the block and the compressed data. In a program recording medium in which a program for causing a computer to process video data is recorded,
The first moving image data includes the second moving image data having a frame rate lower than the frame rate of the first moving image data, and the remaining third moving image obtained by removing the second moving image data from the first moving image data. Separated into data,
For a block obtained by dividing the frame of the third moving image data into blocks, one motion vector representing a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data is detected;
Obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector,
A first restoration step of restoring compressed data obtained by compressing the block using the estimated value of the block into the third moving image data using the second moving image data;
Combining the second and third moving image data and restoring the first moving image data;
The first restoration step includes
From the one motion vector, a detection step for obtaining a positional relationship having a high correlation with the block in a plurality of frames of the second moving image data for the block;
An estimation step for obtaining an estimated value of the block from image data of a plurality of frames of the second moving image data, wherein the positional relationship with the block is a positional relationship obtained from the one motion vector;
A program recording medium on which a program is recorded, comprising: a block restoring step for restoring the block using the estimated value of the block and the compressed data.

Images