JP2010114474A - Image processing device and image processing method using dynamic image motion information - Google Patents

Abstract

The present invention provides an image processing apparatus using motion information of a moving image, which enables high-precision alignment without complicated calculations using motion information between images recorded in the moving image data. .
An image processing apparatus using motion information between frame images recorded in moving image data, wherein a reference frame is obtained from two or more consecutive frame images obtained by decoding moving image data. And a frame selection processing unit for designating a reference frame, and a motion vector cumulative addition conversion processing unit for calculating a motion vector value from the reference frame to the base frame. The motion vector value from the reference frame to the base frame is calculated based on the motion information recorded in (2) by calculation based on cumulative addition and direction conversion.
[Selection] Figure 1

Description

  The present invention relates to an image processing technique, and more particularly to an image processing apparatus and an image processing method that use motion information of a moving image.

  Conventionally, for example, Patent Document 1 discloses a motion vector conversion technique for each block when bitstream conversion is performed from interlace scanning MPEG2 to progressive scanning MPEG4.

  In the motion vector conversion technique described in Patent Document 1, frame rate conversion is performed at the time of conversion from interlace to progressive, the original MPEG2 frame is discarded, and the frame after the adjacent frame to be discarded is discarded. The motion vector value up to the adjacent previous frame is converted and determined based on the motion vector value of the corresponding block of the discarded frame and the adjacent subsequent frame, and then recorded as the new motion vector value of the corresponding block of the adjacent subsequent frame Technology.

  In Patent Document 1, when there is a motion vector from a frame to be discarded to an adjacent previous frame, a value obtained by cumulatively adding the motion vector value of the adjacent subsequent frame and the motion vector value from the discarded frame to the previous frame is newly added. On the other hand, when there is no motion vector from the discarded frame to the adjacent previous frame, the value converted to a constant multiple considering the time for the discarded frame motion vector is discarded. Record as new motion vector value.

  Conventionally, Patent Document 2 discloses a multi-parameter high-accuracy simultaneous estimation technique in image sub-pixel matching as a motion estimation technique required when performing alignment.

Further, Patent Document 3 discloses a super-resolution processing technique for generating a single high-resolution image from a plurality of low-resolution images, and an alignment process (hereinafter simply referred to as a super-resolution processing) required for the super-resolution processing. For example, a motion estimation technique based on high-precision sub-pixel matching as described in Patent Document 2 can be used.
JP 2002-252854 A International Publication No. 04/063991 Pamphlet International Publication No. 04/0688862 Pamphlet Tanaka / Okutomi, “High-speed algorithm for reconfigurable super-resolution processing”, Computer Vision and Image Media (CVIM), Vol. 2004, No. 113, p.97-104, 2004 Okutomi Masatoshi, et al., “Digital Image Processing”, CG-ARTS Association, 2004

  In recent years, in the image processing field, image processing that requires alignment such as electronic blur correction, noise reduction, and super-resolution processing using moving images such as MPEG (hereinafter, these image processing is simply referred to as “image quality improvement processing”). The need to do this is also increasing.

  However, when using multiple frame images in moving image data in which motion information such as MPEG (Moving Picture Expert Group) is recorded, alignment is performed, image quality improvement processing is performed, and a high-definition image is generated. As an example, for example, when super-resolution processing is performed and a high-resolution image is generated, if the motion estimation technique disclosed in Patent Document 2 is applied to alignment, a plurality of images after MPEG decoding are used. Therefore, there is a problem that the processing time is increased and the hardware scale is increased.

  Further, when the motion vector conversion technique disclosed in Patent Document 1 is applied to the alignment process, the motion compensation method differs depending on each MPEG encoder at the time of recording, and the accuracy of the recorded motion vector value depends on the encoder. Since it is not necessarily high, there is a problem that the alignment fails.

  Therefore, if the alignment fails, for example, even when the super-resolution processing disclosed in Patent Document 3 is used, there is a problem that a high-resolution image cannot be generated.

  The present invention has been made under the circumstances as described above, and an object of the present invention is to use high-precision alignment without complicated calculations using motion information between images recorded in moving image data. It is an object of the present invention to provide an image processing apparatus and an image processing method using motion information of a moving image that enable the above.

  The present invention relates to an image processing apparatus using motion information between frame images recorded in moving image data, and the object of the present invention is to provide two or more consecutive images obtained by decoding the moving image data. A frame selection processing unit for designating a base frame and a reference frame from a frame image; and a motion vector cumulative addition conversion processing unit for calculating a motion vector value from the reference frame to the base frame. The addition conversion processing unit calculates a motion vector value from the reference frame to the base frame by calculation based on cumulative addition and direction conversion based on the motion information recorded in the moving image data, or A base frame and a reference frame are designated from two or more consecutive frame images obtained by decoding the moving image data. A frame selection processing unit, and a motion vector cumulative addition conversion processing unit that calculates a motion vector value from the reference frame to the base frame, wherein the motion vector cumulative addition conversion processing unit adds the motion vector data to the moving image data. The recorded motion information is converted into a motion vector value, the target pixel is tracked from the reference frame to the base frame using the motion vector value, and the motion vector value used at the time of tracking is cumulatively added and direction By converting, the moving image data is MPEG 1/2/4, H.261 / 263/264 moving image data, or the motion information recorded in the moving image data is a frame Converted based on motion vector values obtained by block-based matching processing between images or based on the obtained motion vector values Or a registration processing unit that performs registration based on correspondence information of each pixel between the reference frame and the base frame obtained by the motion vector cumulative addition conversion processing unit. A pixel selection processing unit that selects, for each pixel of the registration image output from the processing unit, a similarity, an estimated amount of misalignment, or the estimated amount of similarity and the misregistration. Providing an image quality improvement processing unit that performs image quality improvement processing on the reference frame using only the pixels selected by the pixel selection processing unit, or adding the motion vector cumulative addition An alignment process that performs alignment based on correspondence information of each pixel between the reference frame and the base frame obtained by the conversion processing unit. And the registration image output from the registration processing unit, the degree of similarity that is not affected by the change in brightness, the estimated amount of the position shift that is not affected by the change in brightness, or the estimation of the degree of similarity and the position shift A pixel selection processing unit that selects the pixel based on the amount and corrects the pixel value of the selected pixel so that the brightness of the selected pixel matches the brightness of the corresponding pixel of the reference image; It is effectively achieved by providing an image quality improvement processing unit that performs image quality improvement processing on the reference frame using pixels that have been selected and lightness corrected by the pixel selection processing unit. .

  The present invention also relates to an image processing method using motion information between frame images recorded in moving image data, and the above object of the present invention is to provide continuous 2 obtained by decoding the moving image data. The step of designating a base frame and a reference frame from the above frame images and the movement from the reference frame to the base frame by the motion vector cumulative addition conversion processing means when aligning the reference frame with the base frame Calculating a vector value, wherein the motion vector cumulative addition conversion processing means is based on the motion information recorded in the moving image data, and is calculated from the reference frame by calculation based on cumulative addition and direction conversion. Obtained by calculating a motion vector value for a reference frame or by decoding the moving image data The step of designating a reference frame and a reference frame from two or more consecutive frame images, and when the reference frame is aligned with the reference frame, the motion vector cumulative addition conversion processing means converts the reference frame from the reference frame to the reference frame. The motion vector cumulative addition conversion processing means converts the motion information recorded in the moving image data into a motion vector value, and applies the reference frame to the target pixel. The motion vector value is tracked from the reference frame to the reference frame, and the motion vector value used at the time of tracking is accumulated and converted, or the moving image data is MPEG1 / 2/4, H.261. / 263/264 moving image data or recorded in the moving image data The motion information obtained is a motion vector value obtained by a block-based matching process between frame images, or a value converted based on the obtained motion vector value, or the motion vector cumulative addition conversion process Alignment based on the correspondence information of each pixel of the reference frame and the base frame obtained by means, for each pixel of the registration image obtained by the alignment, similarity, an estimated amount of displacement, or By selecting the pixel based on the similarity and the estimated amount of positional deviation, or by performing image quality improvement processing on the reference frame using only the selected pixel, or The correspondence information of each pixel between the reference frame and the base frame obtained by the motion vector cumulative addition conversion processing means For each pixel of the registration image obtained by the alignment, the degree of similarity that is not affected by the change in brightness, the estimated amount of misalignment that is not affected by the change in brightness, or the estimated amount of the similarity and the misalignment The pixel value of the selected pixel is corrected so that the lightness of the selected pixel matches the lightness of the corresponding pixel of the reference image, or the selected / lightness is selected. This is achieved more effectively by performing image quality improvement processing on the reference frame using the corrected pixels.

  According to the present invention, the motion information in motion picture data in which motion information such as MPEG (Moving Picture Expert Group) is recorded is used to convert the reference frame image to the reference frame image by simple calculation and conversion. The motion vector value is calculated, the reference frame image and the base frame image are aligned with the calculated motion vector value, and then the pixel selection and brightness correction are performed, so that the motion is so highly accurate by simple calculation and conversion. Even if it is not a vector value, highly accurate alignment processing can be performed, so that a high-resolution image and a high-quality image can be generated, and calculation time can be reduced and hardware scale can be reduced.

  An example of calculation time measurement results of motion estimation processing when the motion estimation processing method described in Patent Document 2 and the motion estimation processing according to the present invention are implemented by software is shown below.

  As for a personal computer environment (PC environment) for executing software, the CPU is 3.2 GHz and the RAM is 2 GB. In addition, in the case of software using 30 images with an image size of 352 × 288 [pixel] and implementing the motion estimation method described in Patent Document 2, the measurement time is 38.041 [s], In the case of software implementing the present invention, the measurement time is 16.812 [s].

  By comparing the measurement times of both, it can be seen that the measurement time according to the present invention is clearly shorter as it is less than half the measurement time according to Patent Document 2.

  First, the focus of the present invention will be described.

  MPEG (Moving Picture Experts Group) 1/2/4 and H.261 / 263/264 have been standardized and are used in output data formats of many existing systems. In these moving image data, motion information between frame images is recorded in order to improve compression efficiency, and decoding is performed based on the information. In other words, since motion information between frame images is calculated at the time of encoding, high-precision alignment is realized by effectively using the motion information and further combining with pixel selection / lightness correction processing. .

  In short, in the present invention, for example, when using a plurality of images (frame images) for alignment, a decoded image of moving image data in which motion information between frame images such as MPEG is recorded is made high-definition. When necessary, based on the motion information recorded in the moving image data, calculate the motion vector value from the reference frame image to be used to the base frame image to be refined using a simple method such as cumulative addition or direction conversion. For each pixel in the registration image, the pixel registration and brightness correction processing is performed on the “registration image” generated by the registration processing. Pixel selection and brightness correction by selecting only pixels that have been determined to be valid pixels. Deployment image was obtained. That is, the present invention realizes high-definition alignment by simple motion vector calculation with low calculation cost.

Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.
(Embodiment 1)
FIG. 1 is a block diagram showing a first embodiment of an “image processing apparatus using moving image motion information” (hereinafter also simply referred to as an image processing apparatus of the present invention) according to the present invention.

  As shown in FIG. 1, an image processing apparatus according to the first embodiment includes a moving image input processing unit 101, a moving image decoding processing unit 102, a motion vector cumulative addition conversion processing unit 103, and a frame selection processing unit 105. A registration processing unit 106, a pixel selection processing unit 107, a high resolution processing unit 108, a memory 109, and an image display unit 110.

  The moving image data with motion information 100 in the first embodiment (that is, moving image data in which motion information between frame images is recorded) already exists, and all the motion information between the frame images is present. For example, moving picture data such as MPEG (Moving Picture Experts Group) 1/2/4 or H.261 / 263/264.

  As shown in FIG. 1, moving image data with motion information 100 is input to the moving image input processing unit 101, then decoded as a continuous frame image by the moving image decoding processing unit 102, and decoded continuous frames. The image is stored in the memory 109.

  Here, for example, when the moving image data with motion information 100 is MPEG moving image data, the moving image decoding processing unit 102 decodes the frame information and also decodes and converts the motion information between the frame images. Then, motion vector information between frame images is extracted. Conversion is motion information recorded in MPEG video data, because the difference value of the motion vector value with the adjacent block of the target block is compression-encoded, and added to the motion amount of the adjacent block after decoding It is to be. The moving image decoding processing unit 102 corresponds to an MPEG4 decoder shown in FIG.

  The decoded data stored in the memory 109 can be displayed as a moving image on the image display unit 110. A user (user) browses a moving image (frame image) displayed on the image display unit 110 to improve the image quality with a frame image (hereinafter also simply referred to as “used frame”) used for image quality improvement. It is possible to designate a desired reference frame image (hereinafter also simply referred to as “reference frame”).

  In the present invention, “use frame” means all frame images used for image quality improvement, and one frame image of the designated “use frames” is set as a “reference frame”, and the reference frame is set as the reference frame. Except for the remaining used frames, they are called “reference frames”. That is, if the use frame and the reference frame are designated, the reference frame (reference frame image) is also determined.

  Designated frame information related to the reference frame and the used frame designated by the user is input to the motion vector cumulative addition conversion processing unit 103 via the frame selection processing unit 105.

  The motion vector cumulative addition conversion processing unit 103 acquires the motion vector information extracted by the moving image decoding processing unit 102 via the memory 109 or the moving image decoding processing unit 102, and uses the acquired motion vector information to specify Based on the frame information, a motion vector value from each reference frame image to the base frame image is calculated.

  The motion vector value calculated by the motion vector cumulative addition conversion processing unit 103 is input to the alignment processing unit 106.

  The alignment processing unit 106 generates a registration image by performing alignment processing so that each reference frame image is aligned with the standard frame image using the input motion vector value.

  After the alignment process is performed, the pixel selection processing unit 107 determines whether each pixel of the registration image is a valid pixel or an invalid pixel, and selects only the pixels determined to be valid pixels. The alignment processing unit 106 and the pixel selection processing unit 107 can freely access the decoded image stored in the memory.

  After the pixel selection processing is performed, only the selected pixel (that is, a registration image with pixel selection / lightness correction) is input to the high resolution processing unit 108. The high resolution processing unit 108 generates a high resolution image by performing high resolution processing on the registration image that has undergone pixel selection and brightness correction, and stores the generated high resolution image in the memory 109.

  The high resolution image stored in the memory 109 can be displayed on the image display unit 101, and the user can check the high resolution image generated by the image display unit 101.

  In the high resolution processing unit 108 in FIG. 1, a high resolution image is generated for the registration image subjected to pixel selection and brightness correction. However, the present invention is not limited to this, and the high resolution processing unit Needless to say, the image processing unit 108 may be replaced with an image processing unit that requires alignment such as noise reduction or electronic blur, that is, an image quality improvement processing unit.

  FIG. 2 is a flowchart showing a flow of image processing performed by the image processing apparatus according to the first embodiment of the present invention shown in FIG. That is, the flow shown in FIG. 2 shows a procedure of “an image processing method using a motion vector of a moving image” according to the present invention.

  As shown in FIG. 2, first, moving image data with motion information is input by moving image data with motion information input processing (A01). Next, the motion vector value and the continuous frame image are decoded from the input moving image data with motion information by a moving image data decoding process (A02).

  Then, from the frame information designated by the user (user) (designated used frame information and designated reference frame information), the frame selection process (A03) is used to increase the resolution of the target frame (reference) in the moving image. Frame) and the frame used for higher resolution (used frame). As described above, if the use frame and the reference frame are selected, the reference frame is also naturally selected.

  Based on the motion vector value decoded by the moving image data decoding process (A02), the motion vector value at each pixel of the base frame image and each reference frame image is calculated by the motion vector cumulative addition conversion process (A04).

  After the motion vector values at the respective pixels of the reference frame image and the reference frame images are calculated, the reference frames are aligned with the reference frame image by the alignment process (A05).

  After the alignment processing, pixel selection is performed by pixel selection processing (A06), and a high-resolution image is generated by high-resolution processing (A07) for the selected pixels.

  FIG. 3 shows a block configuration of MPEG4 encoding / decoding processing. The moving picture decoding processing unit 102 in the first embodiment of the present invention shown in FIG. 1 corresponds to the MPEG4 decoding processing block configuration, that is, corresponds to the decoder of FIG.

  1 corresponds to the encoded signal 312 in FIG. 3 and is decoded by the variable-length decoding block 314, the video data is sent to the inverse quantization block 315, and the motion information data is Each is output to the motion vector decoding block 318.

  Thereafter, the video data is subjected to inverse DCT in an inverse DCT block 316. The motion vector decoded by the motion vector decoding block 318 is motion-compensated for the target block of the previous frame image stored in the memory 320 by the motion compensation block 319.

  Then, the decoded image 321 is generated by adding the motion compensated motion vector to the video data subjected to inverse DCT.

  FIG. 4 is a schematic diagram illustrating an example of a setting method when the user sets a designated frame in the frame selection processing unit 105 according to the first embodiment of the present invention.

  As shown in FIG. 4, on the display screen 201 for designating a frame, the user checks the display of the decoded image 202 while moving the decoded image display frame switching knob 203, and the frame to be increased in resolution. A frame can be specified by setting a number in the reference frame setting tab 205 of the specified frame setting tab 204 and a frame number used for higher resolution in the used frame setting tab 206.

  FIG. 5 is a schematic diagram showing an outline of the “motion vector cumulative addition conversion process” in the first embodiment of the present invention.

  As shown in FIG. 5, the frame to be used and the reference frame are designated by the frame selection process (A03). In FIG. 5, the use frames are all 10 frames, and the reference frames obtained by removing the base frame from the use frames are 9 frames from the reference frame 1 to the reference frame 9.

  As shown in FIG. 5, the motion vector value between the reference frame 1 and the reference frame 2 is MV1, the motion vector value between the reference frame 2 and the reference frame 3 is MV2, and the reference frame 3 and the reference frame The motion vector value between 4 and 4 is MV3, and the motion vector value between the reference frame 4 and the base frame is MV4.

  Therefore, the motion vector value MV between the reference frame 1 and the base frame can be obtained by MV = + MV1 + MV2 + MV3 + MV4.

  In this manner, the motion vector value between the designated base frame image and each reference frame image (reference frame 1 to reference frame 9) can be obtained by the “motion vector cumulative addition conversion process” of the present invention. By aligning each reference frame image into a standard frame image with the obtained motion vector value, alignment can be performed.

  A “motion vector cumulative addition conversion process” for obtaining these motion vector values is performed on each pixel between frame images. Contrary to the above, each reference frame image is aligned with each reference frame image by transforming the base frame image with a value obtained by inverting all the directions of the motion vector values obtained in the “motion vector cumulative addition conversion process”. You can also

  FIG. 6 shows a flowchart of the “motion vector cumulative addition conversion process” in the first embodiment of the present invention. In addition, each processing content of the discrimination | determination processes 1-9 (process 1-9) in the flowchart of FIG. 6 is shown in FIG.

  Hereinafter, the “motion vector cumulative addition conversion process” in the present invention will be described in detail with reference to FIGS. 6 and 7. In the following description, I indicates “I frame (Intra-coded Frame) / I-Picture / I-VOP (Intra-coded Video Object Plane)”, and P indicates “P frame (Predicted Frame) / “P-Picture / P-VOP (Predicted Video Object Plane)”, and “B” indicates “B-Picture (B-directional Predicted Frame) / B-Picture / B-VOP (Bi-directional Predicted Video Object Plane)”. Shall.

  As shown in FIG. 6, in the calculation of the motion vector value in the “motion vector cumulative addition conversion process”, the reference frame specified in the frame selection process (A03) and the number of frames other than the reference frame from the used frame (see Processing is performed in a loop (B01, B25) of the number of frames) and a loop (B02, B24) of all pixels in each reference frame.

  As the in-loop process, first, in the target frame / target pixel setting process (B03), the original target frame and the target frame are set as reference frames, and the original target pixel and the target pixel are set as target pixels in the reference frame.

  After that, the relationship between the target frame and the reference frame is determined (B04), the encoding type of the reference frame is determined in process 1 (B05, B12), and the process 2 (B06, B07, B13, In B14), the encoding type of the target frame is determined.

  Thereafter, in consideration of combinations of encoding types, discrimination selection processing is performed in processing 3 to 9 (B08, B09, B10, B11, B15, B16, B17, and B18).

  If the corresponding frame and the corresponding pixel are not selected (NO) in the processes 3 to 9 (B08, B09, B10, B11, B15, B16, B17, B18), all motions in the reference frame are assumed as no motion vector value (B22) Proceed to the pixel loop end (B24).

  On the other hand, if the corresponding frame and the corresponding pixel are selected (YES) in processes 3 to 9 (B08, B09, B10, B11, B15, B16, B17, B18), the motion vector value update process (B19) The direction vector or cumulative addition is performed to update the motion vector value.

  There are two types of motion vector update methods. As shown in the annotation of FIG. 6, “motion vector update method 1” is a method in which the motion vector values of the target pixel of the target frame and the corresponding pixel in the selected frame are cumulatively added in consideration of the direction. is there.

  The “motion vector update method 2” is a method of accumulatively adding the motion vector value of the target pixel of the target frame to the corresponding pixel in the selected frame in consideration of the direction.

  The selection of the motion vector update method is determined based on the target frame, the encoding type of the selected frame, and the relationship between the target frame and the reference frame (before and after time), as shown in the table in the annotation of FIG.

  Thereafter, the selected frame and the reference frame are compared (B20). If the two match, the motion vector value from the target pixel of the reference frame to the corresponding pixel from the reference frame is obtained. The motion vector value is saved as the original target pixel of the original target frame (B23). The process proceeds to the loop end (B24) for all pixels in the reference frame.

  If they do not match in the comparison of the selected frame and the reference frame (B20), the target frame is updated to the selected frame and the target pixel is updated to the selected target pixel in the target frame / target pixel update process (B21). The process returns to the frame context determination process (B04).

  The motion vector cumulative addition conversion process is completed by processing these in-loop processes in a loop (B02, B24) for all pixels in each reference frame and a loop (B01, B25) for the number of reference frames.

  The details of the above motion vector cumulative addition conversion process will be described by taking several patterns as an example. As a premise for the description, the MPEG4 frame coding type and the macroblock coding type in each coding type will be described.

  In MPEG4, there are three types of I-VOP, P-VOP, and B-VOP. I-VOP is called intra coding. When I-VOP itself is coded, coding is completed within the frame. No prediction with other frames is required. P-VOP and B-VOP are called inter coding, and when P-VOP itself is coded, predictive coding is performed from the front I-VOP or P-VOP. When encoding the B-VOP itself, predictive encoding is performed from a bidirectional I-VOP or P-VOP.

  FIG. 8 shows the prediction direction (1) at the time of motion compensation and the direction of the motion vector (which is encoded and recorded in each frame) (which frame is the motion vector) (2) possessed by each frame by (1). Show.

  For example, the fourth I-VOP from the left in (1) of FIG. 8 is used for other frame predictions, but the prediction from other frames is not required for encoding the I-VOP itself. That is, as shown in (2) of FIG. 8, since there is no corresponding motion vector from the fourth I-VOP from the left, the I-VOP itself has no motion vector.

  Also, the seventh P-VOP from the left in (1) of FIG. 8 is predicted from the fourth I-VOP from the left. That is, as shown in (2) of FIG. 8, since the corresponding motion vector from the seventh P-VOP from the left has a motion vector to the fourth I-VOP from the left, the P-VOP itself Has a motion vector.

  Further, the fifth B-VOP from the left in (1) of FIG. 8 is predicted from the fourth I-VOP from the left and the seventh P-VOP from the left. That is, as shown in (2) of FIG. 8, the corresponding motion vectors from the fifth B-VOP from the left are the motion vectors from the left to the fourth I-VOP and from the left to the seventh P-VOP. Therefore, B-VOP itself has a motion vector.

  However, in encoding such as MPEG4, the entire frame is not encoded at once, but the frame is divided into a plurality of macroblocks. At this time, since several modes are provided for encoding each macroblock, it does not necessarily have a motion vector in the above-described direction.

  Here, FIG. 9 shows each macroblock coding mode of each frame coding type and the motion vector of the macroblock in each mode.

  As shown in FIG. 9, the macroblock coding type of I-VOP is “INTRA (+ Q)” mode only, and 16 × 16 pixel intraframe coding is performed, so there is no motion vector.

  There are four types of P-VOP macroblock coding types: “INTRA (+ Q)”, “INTER (+ Q)”, “INTER4V”, and “NOT CODED”.

  Since “INTRA (+ Q)” performs 16 × 16 pixel intraframe coding, there is no motion vector. Since “INTER (+ Q)” performs forward prediction encoding of 16 × 16 pixels, it has one motion vector for the previous prediction frame. Since “INTER4V” performs forward prediction encoding for every 8 × 8 pixels obtained by dividing 16 × 16 pixels into four, it has four motion vectors for the previous prediction frame. Since “NOT CODED” has a small difference from the previous prediction frame, encoding is not performed and the image data of the macroblock at the same position of the previous prediction frame is used as it is. In the practice of the invention, it is considered to have one motion vector value “0” for the previous prediction frame.

  There are four types of B-VOP macroblock coding types: “INTERPOLATE”, “FORWARD”, “BACKWARD”, and “DIRECT”.

  “INTERPOLATE” performs bi-directional predictive encoding of 16 × 16 pixels, and thus has one motion vector for each of the preceding and following prediction frames. Since “FORWARD” performs forward prediction encoding of 16 × 16 pixels, it has one motion vector for the previous prediction frame. Since “BACKWARD” performs backward prediction encoding of 16 × 16 pixels, it has one motion vector for the backward prediction frame. “DIRECT” performs forward / backward predictive coding for every 8 × 8 pixels obtained by dividing 16 × 16 pixels into four, and thus has four motion vectors for the forward and backward predicted frames.

  Based on the above assumptions, the details of the motion vector cumulative addition conversion process will be described with reference to FIGS.

  In FIG. 10, for example, the first frame is an I-VOP, the second and third frames are P-VOPs, the base frame is the first frame, and the reference frame is the third frame.

  As shown in FIG. 10, when a target pixel in the third frame reference frame is a hatched pixel in the drawing, a motion vector of a macroblock (MB) including the target pixel is searched. In this example, since the macroblock coding type is INTER and the motion vector of this macroblock is MV5, the position of the target pixel is moved using MV5.

  The position of the moved pixel is made to correspond to the position in the frame of the P-VOP of the second frame, and the motion vector of the macroblock including the target pixel is similarly searched for the corresponding target pixel position of the second frame. . In this example, the macroblock coding type is INTER4V, and there are four motion vectors that the macroblock has, but the motion vector that the 8 × 8 pixel block that includes the target pixel has is MV4. The position of the target pixel being followed is further moved using MV4.

  The moved pixel position is made to correspond to the position in the I-VOP frame of the first frame. In this example, since the first frame is the base frame, the target pixel of the reference frame has been able to follow up to the base frame, and by adding the initial values 0, MV5, and MV4 used at the time of tracking, The motion vector value up to the reference frame in the target pixel can be obtained.

  In FIG. 11, for example, the first frame is an I-VOP, the second and third frames are P-VOPs, the base frame is the third frame, and the reference frame is the first frame.

  As shown in FIG. 11, when a target pixel in the first frame reference frame is a hatched pixel in the drawing, all frames of the P-VOP in the second frame having motion information with the first frame are used for one frame. Find the pixel corresponding to the target pixel of the eye. When a corresponding pixel is found, the position of the target pixel is moved by -MV4, in which the direction of the motion vector value (MV4 in INTER4V in this example) of the macroblock in the second frame including the pixel is reversed. The position of the selected pixel is made to correspond to the position in the frame of the P-VOP of the second frame, and from the corresponding pixel position of the second frame, similarly, all the pixels of the P-VOP of the third frame are Find the pixel corresponding to the target pixel of the eye.

  When the corresponding pixel is found, the direction of the motion vector value (MV5 in INTER in this example) of the macroblock in the third frame including the pixel is inverted, and the position of the target pixel is moved by -MV5. The pixel position is made to correspond to the position in the third frame P-VOP. In this example, since the third frame is the base frame, the target pixel of the reference frame has been able to follow up to the base frame, and can be referred to by adding the initial values 0, −MV4, and −MV5 used at the time of tracking. The motion vector value up to the reference frame in the target pixel of the frame can be obtained.

  In the example as shown in FIG. 11, a method of searching for a pixel corresponding to the target pixel of the first frame from all the pixels of the P-VOP of the second frame having motion information with the first frame, or P of the third frame FIG. 12 shows a method of searching for a pixel corresponding to the target pixel in the second frame from all the pixels of −VOP.

  In FIG. 12, the target pixel of the fourth reference frame I-VOP from the left is the pixel of the seventh reference frame P-VOP from the left (the motion vector value of the macroblock including the pixel ( An example of searching for a correspondence with MV1)) is shown.

  As in Example 1 in FIG. 12, the positions of all macroblocks (all pixels) of the reference frame (P) are moved by the motion vector values of the macroblocks (all pixels). The result of movement is the left figure of Example 1 in FIG. In this image area that has been moved, the position where the target pixel of the reference frame corresponds is marked, and the moving pixel of the reference frame at that position corresponds. In the case of Example 1 in FIG. 12, one pixel in the macroblock 2 corresponds to it. In this way, the corresponding pixel is searched.

  Example 2 in FIG. 12 is a case where there are a plurality of reference frame moving pixels at the position where the target pixel is marked, and any of them may be selected. For example, in Example 2 of FIG. 12, the marked position of the target pixel corresponds to a certain pixel in the macroblocks 1 and 6, but since the macroblock 1 is closer to the center, the corresponding pixel of the macroblock 1 For the convenience of processing, when processing is performed in the raster scan order and the flag is overwritten, the macroblock 6 with the later order may be selected.

  13 to 15 show some patterns in which the reference frame is in the forward direction from the reference frame, and FIGS. 16 to 18 show some patterns in which the reference frame is in the backward direction from the reference frame. And FIG. 19 shows several patterns that cannot be followed.

  Below, each pattern is demonstrated.

  In the motion vector cumulative addition conversion process in the case of pattern 1 shown in FIG. 13, the target pixel in the macroblock encoding type INTRA of the fourth reference frame from the left is the macroblock encoding of the seventh P-VOP from the left. MV1 corresponds to the target pixel in the type INTER, and further corresponds to the target pixel in the macroblock encoding type INTER of the tenth P-VOP from the left, so that the target pixel in the reference frame The motion vector value up to the corresponding pixel of the reference frame can be obtained by MV = −MV1−MV2.

  In the motion vector cumulative addition conversion process in the case of the pattern 2 shown in FIG. 13, the target pixel in the macroblock encoding type INTERPOLATE of the fifth reference frame from the left is the macroblock encoding of the seventh P-VOP from the left. MV1 corresponds to the target pixel in type INTER4V, and further corresponds to MV2 in the macroblock coding type INTER of the tenth P-VOP from the left. The motion vector value up to the corresponding pixel of the reference frame can be obtained by MV = MV1-MV2.

  In the motion vector cumulative addition conversion process in the case of pattern 2 shown in FIG. 13, as another method, the target pixel in the macroblock encoding type INTERPOLATE of the fifth reference frame from the left is the fourth pixel from the left. MV3 corresponds to the target pixel in the I-VOP macroblock encoding type INTRA, and MV4 corresponds to the target pixel in the seventh P-VOP macroblock encoding type INTER4V from the left. Further, since the MV5 corresponds to the target pixel in the tenth P-VOP macroblock encoding type INTER from the left, the motion vector value from the target pixel of the reference frame to the corresponding pixel of the base frame is MV = MV3-MV4-MV5.

When there are a plurality of methods for obtaining motion vectors as in pattern 2 shown in FIG. 13, the following conditions are taken into consideration.
Condition 1: Selection to reduce the number of vector cumulative additions Condition 2: Selection to follow the reference frame Condition 1 <Condition 2
In the motion vector cumulative addition conversion process in the case of the pattern 3 shown in FIG. 14, the target pixel in the macroblock encoding type INTRA of the fourth reference frame from the left is the macroblock encoding of the seventh P-VOP from the left. MV1 corresponds to the target pixel in the type INTER, and further corresponds to the target pixel in the macroblock coding type INTER of the tenth P-VOP from the left, MV2, and the eleventh from the left. Since the corresponding pixel in the B-VOP macroblock coding type DIRECT corresponds to MV3, the motion vector value from the target pixel in the reference frame to the corresponding pixel in the base frame is MV = −MV1−MV2−MV3. Can be sought.

  In the motion vector cumulative addition conversion process in the case of the pattern 3 shown in FIG. 14, as another method, the target pixel in the macroblock encoding type INTRA of the fourth reference frame from the left is the seventh pixel from the left. MV1 corresponds to the target pixel in the macroblock encoding type INTER of P-VOP, and further corresponds to the target pixel in MV2 of the macroblock encoding type INTER of the tenth P-VOP from the left. Further, the target pixel in the 13th P-VOP macroblock encoding type INTER from the left corresponds to the target pixel in MV4, and further the target in the 11th B-VOP macroblock encoding type DIRECT from the left. Since MV5 corresponds to the pixel, the motion vector from the target pixel of the reference frame to the corresponding pixel of the base frame It can be obtained by MV = -MV1-MV2-MV4-MV5.

  In the motion vector cumulative addition conversion process in the case of pattern 4 shown in FIG. 15, the target pixel in the macroblock coding type BACKWARD of the third reference frame from the left is the macroblock coding of the fourth I-VOP from the left. It corresponds to the target pixel in the type INTRA by MV1, and further corresponds to the target pixel in the seventh P-VOP macroblock encoding type INTER from the left by MV2, and further, the tenth from the left MV3 corresponds to the target pixel in the macroblock encoding type INTER of P-VOP, and further corresponds to the target pixel in the eleventh B-VOP macroblock encoding type DIRECT from the left in MV4. Therefore, the motion vector value from the target pixel of the reference frame to the corresponding pixel of the base frame is MV = MV1-MV. It can be obtained by -MV3-MV4.

  In the motion vector cumulative addition conversion process in the case of the pattern 4 shown in FIG. 15, as another method, the target pixel in the macroblock encoding type BACKWARD of the third reference frame from the left is the fourth pixel from the left. MV1 corresponds to the target pixel in the I-VOP macroblock encoding type INTRA, and MV2 corresponds to the target pixel in the seventh P-VOP macroblock encoding type INTER from the left. Furthermore, it corresponds to the target pixel in the macroblock coding type INTER of the tenth P-VOP from the left by MV3, and further, the target in the macroblock coding type INTER of the thirteenth P-VOP from the left This corresponds to the pixel and MV5, and the eleventh B-VOP macroblock coding type DIR from the left Since addressable by pixel and MV6 in CT, the motion vector value from the target pixel of the reference frame to the corresponding pixels of the reference frame may be determined in MV = MV1-MV2-MV3-MV5-MV6.

  In the motion vector cumulative addition conversion process in the case of pattern 5 shown in FIG. 16, the target pixel in the macroblock encoding type INTER4V of the tenth reference frame from the left is the macroblock encoding of the seventh P-VOP from the left. MV2 corresponds to the target pixel in the type INTER, and further corresponds to the target pixel in the macroblock encoding type INTRA of the fourth I-VOP from the left in the MV1. The motion vector value up to the corresponding pixel of the reference frame can be obtained by MV = MV2 + MV1.

  In the motion vector cumulative addition conversion process in the case of the pattern 6 shown in FIG. 16, the target pixel in the macroblock encoding type DIRECT of the eleventh reference frame from the left is the macroblock encoding of the tenth P-VOP from the left. MV3 corresponds to the target pixel in the type INTER, and further corresponds to the target pixel in the macroblock coding type INTER of the seventh P-VOP type INTER from the left, and the fourth from the left. Since the corresponding pixel in the macroblock coding type INTRA of I-VOP is associated with MV1, the motion vector value from the target pixel of the reference frame to the corresponding pixel of the base frame can be obtained by MV = MV3 + MV2 + MV1.

  In the motion vector cumulative addition conversion process in the case of the pattern 6 shown in FIG. 16, as another method, the target pixel in the macroblock encoding type DIRECT of the eleventh reference frame from the left is the thirteenth pixel from the left. MV4 corresponds to the target pixel in the macroblock coding type INTER of P-VOP, and MV5 corresponds to the target pixel in the tenth P-VOP macroblock coding type INTER from the left. Furthermore, it corresponds to the target pixel in the seventh macroblock coding type INTER of P-VOP from the left by MV2, and further the target in macroblock coding type INTRA of the fourth I-VOP from the left Since MV1 corresponds to the pixel, the motion vector from the target pixel of the reference frame to the corresponding pixel of the base frame It can be obtained by MV = MV4 + MV5 + MV2 + MV1.

  In the motion vector cumulative addition conversion process in the case of the pattern 7 shown in FIG. 17, the target pixel in the macroblock encoding type INTER of the tenth reference frame from the left is the macroblock encoding of the seventh P-VOP from the left. MV3 corresponds to the target pixel in the type INTER, and further corresponds to the target pixel in the macroblock coding type INTRA of the fourth I-VOP from the left in the MV2, and the third from the left Since MV1 corresponds to the target pixel in the B-VOP macroblock coding type DIRECT, the motion vector value from the target pixel of the reference frame to the corresponding pixel of the base frame can be obtained by MV = MV3 + MV2-MV1. it can.

  In the motion vector cumulative addition conversion process in the case of the pattern 8 shown in FIG. 18, the target pixel in the macroblock coding type DIRECT of the eleventh reference frame from the left is the macroblock coding of the tenth P-VOP from the left. MV3 corresponds to the target pixel in the type INTER, and further corresponds to the target pixel in the macroblock coding type INTER of the seventh P-VOP type INTER from the left, and the fifth from the left. Since the target pixel in the B-VOP macroblock coding type INTERPOLATE corresponds to MV1, the motion vector value from the target pixel of the reference frame to the corresponding pixel of the base frame can be obtained by MV = MV3 + MV2−MV1. it can.

  In the motion vector cumulative addition conversion process in the case of the pattern 8 shown in FIG. 18, as another method, the target pixel in the macroblock encoding type DIRECT of the eleventh reference frame from the left is the thirteenth pixel from the left. MV4 corresponds to the target pixel in the macroblock coding type INTER of P-VOP, and MV5 corresponds to the target pixel in the tenth P-VOP macroblock coding type INTER from the left. Furthermore, it corresponds to the target pixel in the macroblock coding type INTER of the seventh P-VOP from the left by MV2, and further, the target in the macroblock coding type INTERPOLATE of the fifth B-VOP from the left Since it corresponds to the pixel with MV1, from the target pixel of the reference frame to the corresponding pixel of the base frame Can vector value can be obtained by MV = MV4 + MV5 + MV2-MV1.

  In the motion vector cumulative addition conversion process in the case of the pattern 8 shown in FIG. 18, as another method, the target pixel in the macroblock encoding type DIRECT of the eleventh reference frame from the left is the tenth pixel from the left. MV3 corresponds to the target pixel in the macroblock coding type INTER of P-VOP, and MV2 corresponds to the target pixel in the macroblock coding type INTER of the seventh P-VOP from the left. Furthermore, the target pixel in the fourth I-VOP macroblock encoding type INTRA from the left corresponds to the target pixel in MV6, and the target in the fifth B-VOP macroblock encoding type INTERPOLATE from the left Since MV1 corresponds to the pixel, the motion from the target pixel in the reference frame to the corresponding pixel in the base frame Vector value can be obtained by MV = MV3 + MV2 + MV6-MV7.

  In the motion vector cumulative addition conversion process in the case of the pattern 8 shown in FIG. 18, as another method, the target pixel in the macroblock encoding type DIRECT of the eleventh reference frame from the left is the thirteenth pixel from the left. MV4 corresponds to the target pixel in the macroblock coding type INTER of P-VOP, and MV5 corresponds to the target pixel in the tenth P-VOP macroblock coding type INTER from the left. Furthermore, it corresponds to the target pixel in the seventh macroblock coding type INTER of P-VOP from the left by MV2, and further the target in macroblock coding type INTRA of the fourth I-VOP from the left This corresponds to the pixel and MV6, and the fifth B-VOP macroblock encoding type INTER from the left Since addressable by pixel and MV7 in Olate, the motion vector value from the target pixel to the corresponding pixel of the reference frame of the reference frame may be determined in MV = MV4 + MV5 + MV2 + MV6-MV7.

  In the motion vector cumulative addition conversion process in the case of the pattern 9 shown in FIG. 19, the target pixel in the macroblock encoding type INTER of the tenth reference frame from the left is the macroblock encoding of the seventh P-VOP from the left. MV2 corresponds to the target pixel in the type INTER, and further corresponds to the target pixel in the macroblock coding type INTRA of the fourth I-VOP from the left in the MV1, but the first P from the left Since there is no motion information and no correspondence can be obtained up to -VOP, it cannot follow and there is no motion vector value from the corresponding pixel of the reference frame to the corresponding pixel of the base frame.

  In the motion vector cumulative addition conversion process in the case of the pattern 10 shown in FIG. 19, the target pixel in the macroblock encoding type INTER of the tenth reference frame from the left is the macroblock encoding of the seventh P-VOP from the left. MV3 corresponds to the target pixel in the type INTER, and further corresponds to the target pixel in the macroblock coding type INTRA of the fourth I-VOP from the left in the MV2, but the first P from the left -VOP or the second B-VOP from the left cannot be handled because there is no motion information, and therefore cannot follow, and there is no motion vector value from the corresponding pixel in the reference frame to the corresponding pixel in the base frame.

  In the motion vector cumulative addition conversion process in the case of the pattern 11 shown in FIG. 19, the target pixel in the macroblock encoding type INTER of the tenth reference frame from the left is the macroblock encoding of the seventh P-VOP from the left. The target pixel in the type INTER corresponds to MV3. However, although there is no motion information up to the fourth I-VOP from the left, for example, if the position is moved by MV2 of the seventh target pixel from the left, it moves outside the range of the image area Therefore, since the correspondence cannot be obtained, it is not possible to follow, and there is no motion vector value from the corresponding pixel of the reference frame to the corresponding pixel of the base frame.

  Here, the cause of the movement outside the range of the image area is shown in FIG.

  FIG. 20 shows the difference in the reference method to the reference image for prediction at the time of MPEG1 / 2 and MPEG4 predictive encoding. As shown in FIG. 20, in the case of MPEG1 / 2, the macroblock of the target image has to be within the image area of the reference image for prediction, but in the case of MPEG4, the macroblock is referenced in the image area. Since the “unrestricted motion vector” method is adopted in which all the macroblocks need not be stored, when the present invention is implemented, the macroblock may move out of the image area.

  The flow of super-resolution processing performed by the resolution enhancement processing unit 108 will be described with reference to FIGS.

  FIG. 21 shows a super-resolution processing algorithm. Hereinafter, description will be given along the flow of the algorithm processing.

S1:
A plurality of low-resolution images are read for use in high-resolution image estimation.

S2:
The reference frame image selected by the frame selection processing unit 105 is set as a high resolution target image. An initial high-resolution image is created by performing complement processing on the target image. This interpolation process can be omitted in some cases.

S3:
Based on the image displacement information converted by the motion vector cumulative addition conversion processing unit 103, the pixel corresponding position between the target image and the other image is clarified, and the enlarged coordinates of the target image are obtained based on the image corresponding position information. A registration image y is generated by performing an overlapping process in a reference coordinate space. The registration image y is generated by the alignment processing unit 106. The details of the method for generating the registration image y are disclosed in Non-Patent Document 1. For example, when the position is associated between each pixel value of a plurality of images and the enlarged coordinates of the target image, the superimposing process is performed by setting each pixel value on a lattice point closest to the enlarged coordinates of the target image. The process that keeps it. At this time, it is conceivable to set a plurality of pixel values on the same grid point. In this case, averaging processing is performed on the plurality of pixel values. The image displacement between the target image and the other images is integrated with the inter-image displacement between the reference image and the target image and the inter-image displacement between the reference image and an image other than the target image estimated by the image displacement estimation unit 201. To generate.

S4:
A point spread function (PSF) taking into consideration imaging characteristics such as an optical transfer function (OTF) and a CCD aperture is obtained. The PSF simply uses a Gauss function, for example.

S5:
Based on the information of S3 and S4, the evaluation function f (z) is minimized.

  However, f (z) takes the form of the following formula 1.

Here, y is a registration image generated in S3, z is a result image obtained by increasing the resolution of a target image, A is a point spread function of an optical system, blur due to a sampling aperture, and each color by a color mosaic filter (CFA) It is a matrix showing an imaging system including components and the like. g (z) is a constraint term in consideration of the smoothness of the image and the correlation between the colors of the image. λ is a weighting factor. For example, the steepest descent method is used to minimize the evaluation function. When the steepest descent method is used, the differential value at z of f (z) is calculated, and the differential value is added to z to update the image to obtain the minimum value of f (z). The differential value of f (z) is as shown in Equation 2 below.

This differential value is added to z to update the image to obtain the minimum value of f (z).

Here, z _n represents a result image that has been repeatedly increased in resolution n times, and α represents a step of the update amount.

S6:
When f (z) obtained in S5 is minimized, the process is terminated and a high-resolution image z of the target image is obtained.

  The above processing is performed for all the low resolution images of the used frames selected by the frame selection processing unit 105.

  FIG. 22 shows the configuration of super-resolution processing. The outline of the super-resolution processing will be described below with reference to FIG. 22 as an example of processing for generating a high-resolution image using a plurality of images.

  In FIG. 22, the super-resolution processing unit includes an interpolation enlargement unit 01, an image storage unit 02, a PSF data holding unit 3, a convolution integration unit 04, a registration image generation unit 05, an image comparison unit 06, a convolution integration unit 07, The regularization term calculation unit 08, the update image generation unit 09, and the convergence determination unit 10 are configured.

  First, a high-resolution target image selected by the frame selection processing unit 105 among a plurality of decoded images stored in the memory 109 is given to the interpolation enlargement unit 01, and this image is subjected to interpolation enlargement (S1). Corresponding). Examples of the interpolation enlargement method used here include bilinear interpolation and bicubic interpolation.

  The interpolated and enlarged image is sent to the image storage unit 02 where it is stored.

The interpolated and enlarged image is given to the convolution integration unit 04, and convolution integration is performed with the PSF data given by the PSF data holding unit 03 (A of the calculation of A ^T (y−Az) in Expression 2). Xz).

  Further, a plurality of images other than the target image are given to the registration image generation unit 05 of the alignment processing unit 106, and based on the image displacement information between the frames converted by the motion vector cumulative addition conversion processing unit 103, the target image Overlay processing is performed in a coordinate space based on the enlarged coordinates of the image to generate a registration image (corresponding to S3). For example, when the position is associated between each pixel value of a plurality of images and the enlarged coordinates of the target image, the superimposing process is performed by setting each pixel value on a lattice point closest to the enlarged coordinates of the target image. The process that keeps it. At this time, it is conceivable to place a plurality of pixel values on the same grid point. In this case, averaging processing is performed on the plurality of pixel values.

The image data subjected to the convolution operation is sent to the image comparison unit 06, and the difference value of the pixel value is calculated at the same pixel position with the registration image generated in the registration image generation unit 05, and the difference image Data is generated (corresponding to y-Az in the calculation of ^AT (y-Az) in Equation 2).

The difference image generated in the image comparison unit 06 is given to the convolution integration unit 07, and the convolution integration is performed with the PSF data given by the PSF data holding unit 03 (A ^T (y-Az in Equation 2). ) Corresponding to the calculation of ^AT ) (y-Az)).

  In addition, a regularized image is generated from the image stored in the image storage unit 02 by the regularization term calculation unit.

  As the regularization term calculation unit, for example, RGB → YCr, Cb color conversion processing is performed on the image stored in the image storage unit 02, and the frequency high-pass filter (color difference component) in the Cr, Cb component (color difference component) There is a method of generating a regularized image by performing a convolution operation processing of a Laplacian filter). By using the regularized image generated by this method in the updated image generation unit 09, the a priori information of the image that “the color difference component of the image is generally a smooth change” is used. A resolution image can be obtained stably (corresponding to the partial differentiation of g (z) in Equation 2).

  The image generated by the convolution integration unit 07, the image stored in the image storage unit 02, and the image generated by the regularization term calculation unit 08 are given to the update image generation unit 09, and the weighted addition of the three images is performed. An updated image is generated by generating the processed image (corresponding to Equation 3).

  The image generated in the updated image generation unit 09 is given to the convergence determination unit 10 and the convergence determination is performed.

  In the convergence determination, it may be determined that the update operation of the image has converged when the number of repeated computations required for the convergence is greater than a certain number of times, or past update images are retained and the current The update operation of the image may be determined to have converged when it is determined that the update amount is less than a certain value.

  When the convergence determination unit 10 determines that the update operation has converged, the updated image is output to the outside as a high-resolution generated image. If it is determined that the update operation has not converged, the updated image is transferred to the image storage unit 02 and used for the next updated image generation. In addition, it is given to the convolution integration unit 04 and the regularization term calculation unit 08 for the next updated image generation.

  By repeating the above processing, the update image generated by the update image generation unit 09 is updated to obtain a good high-resolution image.

  The resolution is increased by the super-resolution processing in the resolution enhancement processing unit 108 described above.

Instead of the high resolution processing unit 108 in FIG. 1, a configuration may be adopted in which an image is improved in image quality by a smoothing process using addition averaging that reduces random noise.

(Embodiment 2)
FIG. 23 shows a second embodiment of the present invention.

  As shown in FIG. 23, a plurality of input images 400 that are moving images is a block-based matching processing block, and between the previous frame image data and the current frame image data stored in the memory 401. By performing motion detection 403 and motion compensation 402 by block-based matching processing, motion information (motion vector) data 404 between images is recorded so as to be attached to the moving image 400, and moving image data with motion information is recorded. 405 is obtained.

  Of the moving image data with motion information 405, the image data 406 is input to the memory 409 of the alignment processing block, and the motion information (motion vector) data 407 is input to the motion vector cumulative addition conversion processing means 408, and motion vector cumulative addition conversion processing is performed. Using the motion vector value calculated by the means 408 and the image data 406 stored in the memory 409, an alignment image 410 can be generated, and alignment between images can be performed.

Here, the details of the “motion vector cumulative addition conversion processing” performed by the motion vector cumulative addition conversion processing means 408 are performed in the same manner as in the first embodiment.

The “pixel selection process (A06)” performed by the pixel selection processing unit 107 in the first embodiment of the present invention shown in FIG. 1 will be described below.

  As described above, the registration processing unit 106 generates a registration image based on the motion vector value calculated by the motion vector cumulative addition conversion processing unit 103.

  FIG. 24 is a schematic diagram for explaining the “pixel selection process” performed by the pixel selection processing unit 107.

Here, the “pixel selection process” performed after the alignment process (deformation) is performed on a certain input image I _k (x, y) and the reference image T (x, y) will be described in detail.

As shown in FIG. 24, an image obtained by deforming the reference image T (x, y) so as to match the input image I _k (x, y)
And Hereafter, this
Is simply referred to as a “deformation reference image”.

  That is, a deformed reference image can be obtained by performing alignment processing between the reference image and the input image. Incidentally, an existing method is used for this alignment process (deformation process).

  However, the deformation of the reference image with respect to the input image is not necessarily complete, that is, the alignment process between the reference image and the input image is not necessarily performed precisely, and may include an alignment error, There may be a (shielded area).

  Here, a case is considered where pixel selection processing is performed on a pixel at a position (x, y) in the input image (hereinafter also simply referred to as “target pixel”).

When the following two conditions are satisfied for a small region around the pixel of interest, the pixel of interest is selected as a “pixel to be used for high resolution processing”.
Condition 1:
“Registered reference image” obtained by aligning the input image and the reference image with a small area of the input image including the target pixel (in the case of FIG. 24, the reference image is deformed to match the input image) Deformation reference image obtained by
) Is equal to or greater than a predetermined threshold (also simply referred to as a first threshold).
Condition 2:
The estimated amount of positional deviation of the target pixel using either the small area of the input image or the corresponding small area of the alignment-processed reference image (deformation reference image) is a predetermined threshold value (simply the second value). Also referred to as a threshold.)

  A pixel of interest that satisfies the above conditions 1 and 2 is set to a pixel value 1 and a pixel of interest that is not satisfied is set to a pixel value 0, so that a binary image (pixel) as shown in FIG. (Selection result) can be generated, that is, a pixel to be used for high resolution processing can be selected.

  Specifically, in the binary image (pixel selection result) of FIG. 24, the pixel value of the white portion pixel is 1, and the pixel is selected as a pixel used for the high resolution processing, and the black portion The pixel value of the pixel is 0, and it is not selected as a pixel used for the high resolution processing. That is, the black portion of the pixel is a pixel that is removed by the “pixel selection process” of the present invention. The binary image in FIG. 24 can be considered as a mask image.

  Furthermore, in the “pixel selection process”, if necessary, a low-pass filter is applied to the binary image in FIG. 24 and a binarization process using an appropriate threshold value is performed, so that the pixels used for the high resolution process are processed. It is also possible to make a selection.

  In short, in the “pixel selection process”, a target pixel that satisfies the above condition 1 and condition 2 is selected as a pixel to be used for the resolution enhancement process, and a target pixel that does not satisfy the above condition 1 and condition 2 is removed. , A pixel selection is performed, and a binary image as a result of pixel selection (a pixel having a pixel value of 1 means a pixel to be used for the high resolution processing, and a pixel having a pixel value of 0 is a pixel not to be used for the high resolution processing, This means a pixel to be removed.) And an image composition (mask composition) based on the input image, thereby generating a “pixel-selected input image” as shown in FIG.

  In the above-described embodiment of “pixel selection processing”, the above conditions 1 and 2 are satisfied as the pixel selection conditions. However, the “pixel selection processing” is not limited to this and satisfies only the above conditions 1. It is also possible to select a pixel of interest to be used or a pixel of interest that satisfies only the above condition 2 as a pixel to be used for the high resolution processing.

  The pixel selection processing unit 107 can perform “lightness correction processing” after performing “pixel selection processing”.

  The “lightness correction process” here means that the lightness of the target pixel of the input image in the lightness correction process is equal to the lightness of the pixel of the corresponding “aligned reference image (ie, deformation reference image)”. In addition, this is a process of correcting the pixel value of the target pixel of the input image in the brightness correction process.

  Incidentally, the input image in the brightness correction process is the “pixel-selected input image” generated by the “pixel selection process”.

  FIG. 25 is a schematic diagram for explaining the “lightness correction process”.

  The “brightness correction process” shown in FIG. 25 is a “brightness correction process” performed after the “pixel selection process” is performed. The input image in the “brightness correction process” in FIG. Means the “pixel-selected input image”.

As shown in FIG. 25, in the brightness correction process, the target pixel of the “pixel-selected input image” and the corresponding modified reference image
The brightness of each of the pixels is estimated by “lightness estimation processing”, and based on the estimated brightness, the brightness image (first brightness image) for the deformation reference image and the brightness image (second brightness) for the pixel-selected input image Image).

  Based on the generated first brightness image and second brightness image, the brightness ratio between them is calculated, and the brightness of the pixel of the deformation reference image corresponding to the brightness of the target pixel of the “pixel-selected input image” By performing “brightness correction processing” for correcting the pixel value of the target pixel of the “pixel-selected input image” so as to be equal to the “pixel-selected and lightness-corrected input image”.

  Here, as a specific example of the method for estimating the lightness of the target pixel, the lightness is obtained for each pixel in the small area including the target pixel, and the obtained lightness is averaged or weighted. A method may be considered in which the averaged brightness is estimated as the brightness of the target pixel. The brightness of the pixels of the corresponding deformation reference image can also be estimated by the same method.

  Further, as a method for obtaining the brightness for each pixel, for example, there is a method of using I in the HSI color space of a hexagonal pyramid model. I in the HSI color space of the hexagonal spindle model is the maximum value of the R channel, G channel, or B channel pixel value of each pixel (see Non-Patent Document 2).

  As described above, since the brightness correction process is performed after the pixel selection process, the pixel selection method uses a pixel selection method that is robust against a change in brightness. In short, the similarity referred to in the above condition 1 uses the similarity that does not depend on the change in brightness, and the estimated amount of misregistration referred to in the above condition 2 is estimated using the estimation method of the misalignment amount that does not depend on the change in brightness. It is preferable to do.

A specific preferred example of the similarity that does not depend on the change in brightness is, for example, normalized cross-correlation of a corresponding small region. Further, as a specific preferred example of the positional deviation estimation method that does not depend on the brightness change, for example, assuming parallel translation, the positional deviation is calculated by using parabolic fitting of normalized cross-correlation in the positional deviation in pixel units. There is a method for estimating the quantity.

Here, using the standard moving image called “standard video sequence (mobile & calendar)”, super-resolution processing was performed after applying the present invention. Of the data obtained by MPEG4-encoding the standard moving image (MPEG4 moving image), 1 to 50 frames were used as used frames. The 25th frame was used as a reference frame.

  FIGS. 26 to 28 are images showing five frames (1, 13, 25, 38, and 50 frames) out of the 50 used frames, and the subject has a motion.

  29 and 30 are images aligned by the motion vector cumulative addition conversion process according to the first embodiment of the present invention.

  FIG. 31 is a registration image generated by the alignment processing unit 106.

  FIG. 32 shows a registration image that has been generated by the pixel selection processing unit 107 and has undergone pixel selection and brightness correction.

  FIG. 33 shows a super-resolution image to which the present invention is applied, and FIG. 34 shows an image by the nearest neighbor method. By applying the present invention, a high-definition image can be obtained with a small amount of calculation.

1 is a block configuration diagram showing a first embodiment of an “image processing apparatus using moving image motion information” according to the present invention. FIG. It is a flowchart which shows the flow of the image processing performed with the image processing apparatus which concerns on the 1st Embodiment of this invention. It is a block block diagram of the encoding / decoding process of MPEG4. It is a schematic diagram which shows the example of the designation | designated frame setting method in the "frame selection process" concerning this invention. It is a schematic diagram which shows the outline | summary of the "motion vector cumulative addition conversion process" concerning this invention. It is a flowchart which shows the flow of the "motion vector cumulative addition conversion process" concerning this invention. It is explanatory drawing of the processing content of the discrimination processes 1-9 (process 1-9) in the flowchart of FIG. It is a schematic diagram for demonstrating the prediction direction at the time of motion compensation, and the direction of the motion vector which each frame has. It is explanatory drawing about each macroblock encoding mode of MPEG4, and the motion vector which a macroblock has. It is a schematic diagram for demonstrating the example of the motion vector update method when the 1st frame is a reference | standard frame and the 3rd frame is a reference frame. It is a schematic diagram for demonstrating the example of the motion vector update method when the 1st frame is a reference frame and the 3rd frame is a reference | standard frame. It is explanatory drawing of the method of searching for the pixel corresponding to an object pixel. FIG. 6 is a schematic diagram for explaining pattern examples 1 and 2 in which an MPEG4 reference frame is in a forward direction from a base frame. It is a schematic diagram for demonstrating the example 3 of a pattern in which the reference frame of MPEG4 is ahead of the standard frame. It is a schematic diagram for demonstrating the example 4 of a pattern in which the reference frame of MPEG4 exists ahead of a reference | standard frame. It is a schematic diagram for demonstrating pattern examples 5 and 6 in which the MPEG4 reference frame is in the backward direction from the base frame. It is a schematic diagram for demonstrating the example 7 of a pattern in which an MPEG4 reference frame is back from a base frame. It is a schematic diagram for demonstrating the example 8 of a pattern with an MPEG4 reference frame located behind a reference frame. It is a schematic diagram for demonstrating the pattern examples 9, 10, and 11 which cannot follow MPEG4. It is explanatory drawing about the reference method to the reference image for a prediction at the time of the MPEG1 / 2 and MPEG4 prediction encoding. It is a flowchart of high resolution image estimation. It is a block diagram of a high-resolution image estimation calculating part. FIG. 6 is a block diagram of a second embodiment of the present invention with block-based matching. FIG. 10 is a schematic diagram for explaining “pixel selection processing” performed by the pixel selection processing unit 107. It is a schematic diagram for demonstrating "brightness correction processing." It is a figure which shows the image of the 1st frame and the 13th frame. It is a figure which shows the 25th frame (reference | standard frame image). It is a figure which shows the image of the 38th frame and the 50th frame. It is a figure which shows the 1st frame image and the 13th frame image aligned with the reference | standard frame image of 25th frame. It is a figure which shows the 38th frame and the 50th frame image aligned with the reference | standard frame image of 25th frame. It is a figure which shows the registration image which used 1-50 frames. It is a figure which shows the registration image which carried out pixel selection and brightness correction. It is a figure which shows the super-resolution image from the registration image which carried out pixel selection and brightness correction. It is a figure which shows the high resolution image by the conventional method of the reference | standard frame image of the 25th frame.

Claims (16)

An image processing apparatus using motion information between frame images recorded in moving image data,
A frame selection processing unit for designating a base frame and a reference frame from two or more consecutive frame images obtained by decoding the moving image data;
A motion vector cumulative addition conversion processing unit for calculating a motion vector value from the reference frame to the base frame;
With
The motion vector cumulative addition conversion processing unit calculates a motion vector value from the reference frame to the base frame by calculation based on cumulative addition and direction conversion based on the motion information recorded in the moving image data. An image processing apparatus using motion information of a moving image characterized by An image processing apparatus using motion information between frame images recorded in moving image data,
A frame selection processing unit for designating a base frame and a reference frame from two or more consecutive frame images obtained by decoding the moving image data;
A motion vector cumulative addition conversion processing unit for calculating a motion vector value from the reference frame to the base frame;
With
The motion vector cumulative addition conversion processing unit converts the motion information recorded in the moving image data into a motion vector value, and uses the motion vector value from the reference frame toward the base frame for a target pixel. An image processing apparatus using motion information of a moving image, wherein the motion vector value used for tracking is cumulatively added and direction-converted.   The image processing apparatus using moving image motion information according to claim 1, wherein the moving image data is MPEG 1/2/4 or H.261 / 263/264 moving image data.   The motion information recorded in the moving image data is a motion vector value obtained by block-based matching processing between frame images, or a value converted based on the obtained motion vector value. An image processing apparatus using the motion information of the moving image described. An alignment processing unit that performs alignment based on correspondence information of each pixel between the reference frame and the base frame obtained by the motion vector cumulative addition conversion processing unit;
Pixel selection processing for selecting each pixel of the registration image output from the alignment processing unit based on the similarity, the estimated amount of misalignment, or the estimated amount of similarity and misalignment And
An image processing apparatus using motion information of a moving image according to any one of claims 1 to 4.   The image processing using the motion information of the moving image according to claim 5, further comprising an image quality improvement processing unit that performs image quality improvement processing on the reference frame using only the pixels selected by the pixel selection processing unit. apparatus. An alignment processing unit that performs alignment based on correspondence information of each pixel between the reference frame and the base frame obtained by the motion vector cumulative addition conversion processing unit;
For each pixel of the registration image output from the registration processing unit, based on the similarity that is not affected by the change in brightness, the estimated amount of misalignment that is not affected by the change in brightness, or the estimated amount of similarity and the misalignment A pixel selection processing unit that selects the pixel and corrects the pixel value of the selected pixel so that the brightness of the selected pixel matches the brightness of the corresponding pixel of the reference image;
An image processing apparatus using motion information of a moving image according to any one of claims 1 to 4.   The moving image motion information according to claim 7, further comprising an image quality improvement processing unit that performs image quality improvement processing on the reference frame using pixels selected and lightness corrected by the pixel selection processing unit. Image processing device. An image processing method using motion information between frame images recorded in moving image data,
Designating a base frame and a reference frame from two or more consecutive frame images obtained by decoding the moving image data;
A step of calculating a motion vector value from the reference frame to the base frame by motion vector cumulative addition conversion processing means when aligning the reference frame with the base frame;
Have
The motion vector cumulative addition conversion processing means calculates a motion vector value from the reference frame to the reference frame by calculation based on the cumulative addition and direction conversion based on the motion information recorded in the moving image data. An image processing method using motion information of a moving image characterized by the above. An image processing method using motion information between frame images recorded in moving image data,
Designating a base frame and a reference frame from two or more consecutive frame images obtained by decoding the moving image data;
A step of calculating a motion vector value from the reference frame to the base frame by motion vector cumulative addition conversion processing means when aligning the reference frame with the base frame;
Have
The motion vector cumulative addition conversion processing means converts the motion information recorded in the moving image data into a motion vector value, and uses the motion vector value from the reference frame toward the reference frame for the target pixel. An image processing method using motion information of a moving image, wherein the motion vector value used for tracking is cumulatively added and the direction is changed.   11. The image processing method using moving image motion information according to claim 9 or 10, wherein the moving image data is MPEG 1/2/4 or H.261 / 263/264 moving image data.   The motion information recorded in the moving image data is a motion vector value obtained by block-based matching processing between frame images, or a value converted based on the obtained motion vector value. An image processing method using the motion information of the described moving image. Based on the correspondence information of each pixel of the reference frame and the base frame obtained by the motion vector cumulative addition conversion processing means,
The pixel according to any one of claims 9 to 12, wherein for each pixel of the registration image obtained by registration, the pixel is selected based on the similarity, the estimated amount of positional deviation, or the estimated amount of the similarity and the positional deviation. An image processing method using the motion information of the moving image described in the above.   The image processing method using motion information of a moving image according to claim 13, wherein image quality improvement processing is performed on the reference frame using only selected pixels. Based on the correspondence information of each pixel of the reference frame and the base frame obtained by the motion vector cumulative addition conversion processing means,
For each pixel of the registration image obtained by the alignment, the pixel based on the similarity that is not affected by the change in brightness, the estimated amount of misalignment that is not affected by the change in brightness, or the estimated amount of similarity and the misalignment Select
The moving image motion information according to claim 9, wherein the pixel value of the selected pixel is corrected so that the brightness of the selected pixel matches the brightness of the corresponding pixel of the reference image. The image processing method used.   The image processing method using motion information of a moving image according to claim 15, wherein image quality improvement processing is performed on the reference frame using a pixel that has been subjected to selection / lightness correction.