JP2007279800A - Motion vector detection apparatus, motion vector detection method, camera shake correction apparatus, camera shake correction method, camera shake correction program, and moving image display apparatus including camera shake correction apparatus - Google Patents

Abstract

The influence of motion vector errors on moving image processing is reduced.
A frame image acquisition unit that acquires a reference frame image and a search frame image, which are two frame images moving back and forth in time series from a moving image, and a correlation value between the reference frame image and the search frame image are calculated. A correlation value calculating means; and a motion vector calculating means for calculating a motion vector indicating a motion of the entire search frame image based on the reference frame image based on the correlation value, the motion vector calculating means calculating the correlation value If the absolute value of the difference between the correlation value with the strongest correlation value and the correlation value with the second strongest correlation value among the correlation values calculated by the means is smaller than a predetermined threshold A, the previous frame image in time series order If the correlation vector having the strongest correlation among the correlation values is weaker than a predetermined threshold B indicating the strength of the correlation, Not calculated a motion vector of a search frame image.
[Selection] Figure 1

Description

The present invention acquires a frame image from a moving image, detects a motion vector of the acquired frame image, a motion vector detection device, a motion vector detection method, a camera shake correction device, a camera shake correction method using the detected motion vector, The present invention relates to a camera shake correction program and a moving image display device including a camera shake correction device.

Conventionally, a method using a correlation value of pixels such as a representative point matching method and a block matching method has been used for detection of a motion vector, which is an important technique in camera shake correction and moving image compression (for example, Patent Document 1). . The motion vector detection device of Patent Document 1 uses a motion vector based on a reference frame image preceding or following the search frame image of a search frame image that is one of the frame images constituting the moving image. It is obtained from the correlation of pixels between frames.

JP 2004-206638 A

Since both the representative point matching method and the block matching method detect the motion vector using the correlation of the pixel values, the error of the detected motion vector may increase depending on the content and motion of the image. There is. When the error of the motion vector becomes large, there is a problem that the effect of the error is exerted on the moving image processing using the motion vector such as camera shake correction and compression of the moving image, and the effect cannot be sufficiently obtained.

The present invention has been made paying attention to such a conventional problem, and its purpose is to provide a motion vector detection apparatus and a motion that can reduce the influence of motion vector errors on moving image processing using motion vectors. An object of the present invention is to provide a vector detection method, a camera shake correction device, a camera shake correction method, a camera shake correction program, and a moving image display device including the camera shake correction device.

In order to solve the above problems, a motion vector detection device according to the present invention basically includes a frame image acquisition means for acquiring a reference frame image and a search frame image, which are two frame images moving back and forth in chronological order from a moving image. Correlation value calculating means for calculating a correlation value between the reference frame image and the search frame image, and based on the correlation value, movement of the entire search frame image based on the reference frame image. Motion vector calculation means for calculating a motion vector to be shown, wherein the motion vector calculation means has the second highest correlation with the correlation value having the strongest correlation among the correlation values calculated by the correlation value calculation means. When the absolute value of the difference from the correlation value is smaller than a predetermined threshold A, the motion vector of the previous frame image in time series order is determined as the search frame. The motion vector of the search frame image is not calculated when the correlation value is weaker than a predetermined threshold value B indicating the strength of the correlation. The gist.

According to the motion vector detection device of the present invention, the motion vector calculation means includes the search frame based on the correlation value between images calculated by the correlation value calculation means from the reference frame image acquired by the frame image acquisition means and the search frame image. The motion vector of the image is calculated. If the absolute value of the difference between the correlation value having the strongest correlation and the correlation value having the second highest correlation among the correlation values calculated by the correlation value calculation means is smaller than a predetermined threshold A, the motion of the previous frame image in time series order Let the vector be the motion vector of the search frame image. If the correlation value having the strongest correlation among the correlation values calculated by the correlation value calculation means is weaker than a predetermined threshold B indicating the strength of the correlation, the correlation value of the search frame image is not calculated.

When the absolute value of the difference between the correlation value having the strongest correlation and the correlation value having the second strongest correlation is small, the change in the content of the moving image is small, but the error of the motion vector calculated based on the correlation value becomes large. Probability is high. However, since there are two strong correlation values, it is considered that there is little change in the content of the moving image, and it is more error to use the motion vector of the previous frame in chronological order than the motion vector calculated based on the correlation value. Few. Therefore, a motion vector with less error can be used by using the motion vector of the previous frame in time series order.

If the correlation value having the strongest correlation is weak, there is a high possibility that an error is included in the motion vector calculated based on the correlation value. Therefore, when the correlation of the correlation value having the strongest correlation that increases the error of the motion vector is weak, the motion vector is not calculated, and the motion vector containing a large amount of error can be prevented from being used. Also, when the absolute value of the difference between the correlation value with the next largest error and the correlation value with the second strongest correlation is small, the motion vector of the previous frame is used in time series order to reduce the error. Fewer motion vectors can be used instead.

In the above-described motion vector detection device of the present invention, the correlation value calculating unit changes a relative position between the reference frame image and the search frame image within a predetermined range, and a pixel of a predetermined pixel block at each position. A difference average value, which is an average value of differences in luminance values of data, is set as the correlation value.

In this way, the correlation value and the motion vector between the reference frame image and the search frame image can be calculated by the block matching method using the pixel block. Therefore, the conventional motion vector detection device using the block matching method can be easily replaced with the motion vector detection device of the present invention.

In addition, the camera shake correction method corresponding to the above-described camera shake correction device of the present invention includes a frame image acquisition step of acquiring a reference frame image and a search frame image, which are two frame images moving back and forth in time series from a moving image, and the reference A correlation value calculating step for calculating a correlation value between a frame image and the search frame image; and a motion vector indicating a motion of the entire search frame image based on the reference frame image based on the correlation value A motion vector calculation step for calculating the correlation value, wherein the motion vector calculation step includes the correlation value having the strongest correlation among the correlation values calculated by the correlation value calculation step and the correlation value having the second highest correlation. When the absolute value of the difference between the two is smaller than a predetermined threshold A, the motion vector of the previous frame image in time-series order is used as the search frame image. The gist is that the motion vector of the search frame image is not calculated when the correlation is weaker than a predetermined threshold B indicating the strength of the correlation. To do.

  In this way, an effect equivalent to that of the motion vector detection device of the present invention can be obtained.

Further, in order to solve the above problems, a camera shake correction device according to the present invention basically includes the above-described motion vector detection device, and a camera shake correction unit that corrects camera shake in the search frame image based on the motion vector. The camera shake correction means is configured not to correct the camera shake when the motion vector calculation means does not calculate the motion vector.

In this way, when the correlation value of the correlation having the strongest correlation among the correlation values calculated by the correlation value calculation means is weaker than the predetermined threshold value B indicating the correlation, that is, when the motion vector error is the largest, In order to reduce the influence of errors, it is possible to avoid correcting camera shake. If the absolute value of the difference between the correlation value with the strongest correlation value and the correlation value with the second strongest correlation value among the correlation values calculated by the correlation value calculation step is smaller than the predetermined threshold A, the previous error is small. Camera shake can be corrected using the motion vector of the frame.

Further, a camera shake correction method and a camera shake correction program corresponding to the above-described camera shake correction apparatus include the above-described motion vector detection method and a camera shake correction process for correcting camera shake in the search frame image based on the motion vector. The camera shake correction step is summarized in that the camera shake correction is not performed when the motion vector calculation step does not calculate the motion vector.

  In this way, the same effect as the camera shake correction device of the present invention can be obtained.

A moving image display apparatus according to the present invention includes the above-described camera shake correction apparatus. By doing so, it is possible to display a moving image in which the influence of motion vector errors is reduced and camera shake is corrected.

  Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is an explanatory diagram for explaining a schematic configuration of the camera shake correction apparatus 100. The camera shake correction apparatus 100 according to the first embodiment includes a motion vector detection device 90 and an image correction unit 14 that corrects camera shake.
Consists of zero. The motion vector detection device 90 includes a frame image acquisition unit 110, a correlation value calculation unit 120, and a motion vector calculation unit 130, acquires a moving image from the input device 200, and calculates a motion vector of the frame image. The image correction unit 140 also includes a frame image acquisition unit 110 based on the motion vector calculated by the motion vector calculation unit 130.
Corrects camera shake in the acquired frame image and outputs it to the output device 210.

As the input device 200, a device capable of outputting a moving image such as a video, a digital video camera (DVC), a broadcast, and a moving image database (DB) for storing a moving image can be used. Further, a storage medium such as a hard disk or a memory card in which moving images are stored may be used as the input device 200.

The frame image acquisition unit 110 sequentially acquires frame images F (n) (n is an integer indicating a time-series order) from the moving image output from the input device 200.

The correlation value calculation unit 120 is configured to display the frame image F (n
-1) and a correlation value between images representing the movement of the entire image of F (n) is calculated.

The motion vector calculation means 130 is based on the correlation value calculated by the correlation value calculation means 120.
A motion vector V (n) indicating the motion of the entire frame image F (n) is calculated.

The image correction unit 140 generates a corrected frame image HFG (n) in which the camera shake of the frame image F (n) is corrected using the motion vector V (n), and outputs it to the output device 210.

As the output device 210, a CRT, a projector, a television, or the like can be used.
Further, the corrected frame image HFG (n) may be distributed via a network. Further, the output frame 210 may be a storage medium such as a hard disk or a DVD-ROM, and the corrected frame image HFG (n) may be recorded.

The camera shake correction apparatus 100 having the above configuration is configured by a computer that implements the processing in each of the above-described units by a software program and installs the camera shake correction program as the processing program. This computer (not shown) includes a CPU, ROM, RAM
, A general computer equipped with a hard disk, an interface, etc., connected to a keyboard, a display, etc. The computer's ROM stores a camera shake correction program, and the CPU functions as the camera shake correction device 100 when the camera shake correction program is executed. In the following, the processing of each unit executed by the CPU in this computer will be described with reference to the flowchart of FIG.

When the process is started, the frame image acquisition unit 110 acquires the frame image F (n) from the input device 200 (step S100).

Next, the correlation value calculation unit 120 calculates a correlation value between the images of the frame images F (n−1) (reference frame image) and F (n) (search frame image) acquired by the frame image acquisition unit 110. (Step S110). Here, the correlation value calculation method will be described with reference to the explanatory diagram for explaining the correlation value calculation method of FIG.

First, the frame images F (n−1) and F (n) are overlaid at positions shifted by the number of pixels SX and SY. Then, the pixel block B (SX, SY) is placed in F (n−1) of the frame image. The pixel block B (SX, SY) indicates a rectangular area having a predetermined size of vertical N pixels and horizontal M pixels. At this time, the lower left coordinates of the pixel block B (SX, SY) are (x0, y0) in the coordinates of the frame image F (n−1), and (x) in the coordinates of the frame image F (n).
1, y1). Since the frame images F (n−1) and F (n) are shifted by SX and SY, (x1, y1) = (x0−SX, y0 + SY).

The correlation value D (SX, SY) between the images of the frame images F (n−1) and F (n) at this time is the luminance in the pixel block B (SX, SY) as shown in Equation (1). It can be calculated by the average value difference (difference average value).

Ｐ０（ｘ，ｙ）：フレーム画像Ｆ（ｎ−１）の座標（ｘ，ｙ）の輝度値
Ｐ１（ｘ，ｙ）：フレーム画像Ｆ（ｎ）の座標（ｘ，ｙ）の輝度値
Ｎ ：画素ブロックの縦の画素数
Ｍ ：画素ブロックの横の画素数
（ｘ０，ｙ０）：フレーム画像Ｆ（ｎ−１）の中の画素ブロックの左下の座標
（ｘ１，ｙ１）：フレーム画像Ｆ（ｎ）の中の画素ブロックの左下の座標

P0 (x, y): luminance value of coordinates (x, y) of the frame image F (n−1) P1 (x, y): luminance value of coordinates (x, y) of the frame image F (n) N: Number of vertical pixels of the pixel block M: Number of horizontal pixels of the pixel block (x0, y0): Lower left coordinates of the pixel block in the frame image F (n-1) (x1, y1): Frame image F (n The lower left coordinates of the pixel block in

Here, assuming that the motion vector detection range is ± TX in the horizontal direction and ± TY in the vertical direction, the correlation value calculation unit 120 changes SX and SY pixel by pixel within the motion vector detection range.
2 × TX + 1) × (2 × TY + 1) correlation values D (SX, SY) (−TX ≦ SX ≦ TX,
-TY≤SY≤TY) is calculated. The correlation values D (SX, SY) calculated in this way are similar to the images of the frame images F (n−1) and F (n) in the pixel block B (SX, SY) (strongly correlated). The smaller the frame images F (n−1) and F (n), the larger the images.

Next, the motion vector calculation means 130 calculates a motion vector V (n) indicating the motion of the entire frame image F (n) using the correlation value D (SX, SY) (step S120).
). Here, the motion vector V (n) will be described with reference to an explanatory diagram for explaining the frame image F (n) and the motion vector V (n) in FIG.

FIG. 4 (upper side) shows an example of the first frame image F (1) in time order and the second and third frame images F (2) and F (3) acquired by the frame image acquisition unit 110. Show. FIG. 4 is an example in which the house shown in the image is moving in the lower left direction.

The camera shake correction apparatus 100 cuts out an image of a predetermined size from the frame image F (n) at a position that reduces image shake due to camera shake, thereby correcting the camera shake correction frame image HF.
G (n) is generated. The default cut-out positions of the corrected frame image HFG (n) are frame images FK (1) to FK (3) indicated by broken lines in FIG. Here, as shown in FIG. 4, the frame image FK (n) is smaller in screen size than the frame image F (n) by a predetermined correction limit value Smax (Sxmax in the X direction and Symax in the Y direction). Become an image. In the first embodiment, a predetermined correction limit value Smax (X component is Sxmax, Y component is Symax) is set to a predetermined value in advance in the camera shake correction program.

Similarly, in FIG. 4 (lower side), frame images F (1) to F (3) and FK (1) are arranged so that the objects shown in the frame images F (1) to F (3) are in the same position. -FK (3) was shown. A difference in relative position of the frame image FK (n) when arranged in this way is a motion vector. That is, the motion vector V (2) is a difference between the relative positions of the frame images FK (2) and FK (1), and the motion vector V (3) is the frame image FK (3) and the frame image FK (2).
) Relative position difference. In the example of FIG. 4, there is no frame preceding the frame image F (1), so the motion vector V (1) is zero.

This relative position indicates a position where the frame images F (n−1) and F (n) in FIG.
X, SY). Assuming that the relative positions where the objects shown in FIG. 4 (lower side) are arranged at the same position are (X1, Y1), the correlation value D (X1, Y1) at this time is the pixel block B ( The images in X1, Y1) are the most similar, and the correlation is the strongest. Accordingly, among the correlation values D (SX, SY) calculated by the correlation value calculation unit 120, the correlation (X1, Y1) having the strongest correlation is the motion vector V (n).

Next, the motion vector calculation means 130 calculates the difference between the correlation value D (X1, Y1) having the strongest correlation and the correlation value D (X2, Y2) having the second highest correlation among the correlation values D (SX, SY). Absolute value of (
Whether or not | D (X1, Y1) -D (X2, Y2) |) is smaller than a predetermined threshold A is checked (step S130). When (| D (X1, Y1) −D (X2, Y2) |) is smaller than the predetermined threshold A (step S130: YES), the motion vector V calculated in the previous frame is used.
Let (n-1) be a motion vector V (n) (step S140).

When (| D (X1, Y1) −D (X2, Y2) |) is small, there are two or more similar parts in the image, and therefore the motion vector V calculated from the correlation value D (SX, SY). The error of (n) becomes large. However, since there is a highly correlated part, the frame images F (n−1) and F (n
In n), the difference in image contents is not large. Further, since the difference in image contents is small, there is little change in the movement of the entire image from the previous frame image F (n−1). Therefore, by using the motion vector V (n−1) of the previous frame as the motion vector V (n) of the frame image F (n), the motion vector directly calculated from the correlation value D (SX, SY) is used. The motion vector V (n) with a small error can be calculated. The predetermined threshold A is set by the user of the camera shake correction apparatus 100 according to the content of the moving image for correcting camera shake. For example, in Example 1, it is set to 5% of the maximum correlation value.

Next, of the correlation values D (SX, SY), the absolute value (| D () of the difference between the correlation value D (X1, Y1) having the strongest correlation and the correlation value D (X2, Y2) having the second highest correlation. X1, Y1) -D (X2-
Y2) |) is greater than or equal to the predetermined threshold A (step S130: NO), it is checked whether or not the correlation value D (X1, Y1) having the strongest correlation is smaller than the predetermined threshold B (step S150).
). When the correlation value D (X1, Y1) having the strongest correlation is smaller than the predetermined threshold B (step S150: YES), the motion vector calculation unit 130 invalidates the motion vector V (n) of the frame image F (n). (Step S160). On the other hand, the correlation value D (
If X1, Y1) is not smaller than the predetermined threshold B (step S150: NO), the process proceeds to step S170.

When the correlation value D (SX, SY) is strongly influenced by the change in the image content rather than the movement of the entire image, the correlation value D (X1, Y1) having the strongest correlation becomes small. Therefore, the correlation value D
(X1, Y1) is compared with a predetermined threshold B, and if it is smaller than the predetermined threshold B, the correlation value D (
The motion vector calculation by SX, SY) is determined to be impossible due to an excessive error, and the motion vector calculation means 130 stops calculating the motion vector, that is, the motion vector V (n
) Is disabled. By doing so, it is possible to prevent the motion vector V (n) having a large error from being used when the motion vector error becomes large. Further, the predetermined threshold B is set by the user of the camera shake correction apparatus 100 according to the content of the moving image for correcting camera shake. For example, in Example 1, it is set to 5% of the maximum correlation value.

Next, the image correction unit 140 calculates a correction position vector S (n) indicating a correction amount for correcting the camera shake of the frame image F (n), and uses the correction position vector S (n) to reduce the camera shake. A corrected frame image HFG (n) is generated (step S170).

FIG. 5 is an explanatory diagram for explaining correction of camera shake by the image correction unit 140. FIG.
Will be used to describe a method of correcting camera shake in the frame images F (2) and F (3) described with reference to FIG. The correction position vectors S (2) and S (3) are vectors indicating positions where the correction frame images HFG (2) and HFG (3) are cut out from the frame images F (2) and F (3). It is represented by a relative position from the frame images FK (2) and FK (3) at the same position as the frame image.

Here, if the correction position vectors S (2) and S (3) are vectors obtained by changing the directions of the motion vectors V (2) and V (3) in the opposite directions, the corrected frame images HFG (2),
The position of the house shown in HFG (3) is not changed, and camera shake can be corrected.

Then, the image correcting unit 140 cuts out the frame image F (n) at a position indicated by the correction position vector S (n) and generates a corrected frame image HFG (n).

In addition, when the motion vector V (n) is invalidated by the motion vector calculation means 130, the correction position vector S (n) is set to 0 so that the camera shake is not corrected.

Next, the corrected frame image HFG (n) is output to the output device 210 (step S180).
). When the camera shake correction of the frame image F (n) is completed in this way, it is next checked whether or not the frame image F (n) is the end of the moving image (step S190). Frame image F (
If n) is the end of the moving image (step S190: YES), the camera shake correction program is terminated.

If the frame image F (n) is not the end of the moving image (step S190: NO), 1 is added to n to correct the camera shake of the frame image F (n + 1), and step S100 is performed.
Return to. In this way, correction of camera shake of the frame image F (n) proceeds in time-series order, one frame at a time.

As described above, in the camera shake correction apparatus 100 according to the first embodiment, when the difference between the strongest correlation value D (X1, Y1) and the second strongest correlation value D (X2, Y2) is small, the previous error is smaller. The camera shake can be corrected with high accuracy by correcting the camera shake using the motion vector V (n−1) of the frame. In addition, the strongest correlation value D (X1, Y1) is small and the motion vector V (
When there is a high possibility that the error in n) is likely to increase, the camera shake correction using the motion vector V (n) is suspended, and a camera shake correction error due to the error in the motion vector V (n) can be prevented. .

Hereinafter, modified examples related to the camera shake correction apparatus according to the first embodiment will be described.
(Modification 1)
A camera shake correction device may be provided in a moving image display device such as a projector equipped with a CPU, ROM, and RAM. The camera shake correction program according to the first embodiment is stored in the ROM of the moving image display apparatus, and the CPU of the moving image display apparatus executes the camera shake correction program, thereby causing the moving image display apparatus to function as a camera shake correction apparatus. In this way, it is possible to display a moving image with camera shake correction with high accuracy.

Explanatory drawing explaining the structure outline of a camera-shake correction apparatus. The flowchart figure explaining the process sequence of a camera shake correction apparatus. Explanatory drawing for demonstrating the calculation method of a correlation value. Explanatory drawing for demonstrating frame image F (n) and motion vector V (n). Explanatory drawing for demonstrating the correction of camera shake by an image correction means.

Explanation of symbols

DESCRIPTION OF SYMBOLS 90 ... Motion vector detection apparatus, 100 ... Camera shake correction apparatus, 110 ... Frame image acquisition means, 120 ... Correlation value calculation means, 130 ... Motion vector calculation means, 140 ... Image correction means,
200 ... input device 210 ... output device F (n) frame image FK (n) frame image V (n) motion vector S (n) correction position vector HFG (n) correction frame image.

Claims (7)

Frame image acquisition means for acquiring a reference frame image and a search frame image, which are two frame images moving back and forth in chronological order from a moving image;
Correlation value calculating means for calculating a correlation value between the reference frame image and the search frame image;
A motion vector calculating means for calculating a motion vector indicating a motion of the entire image of the search frame image based on the reference frame image based on the correlation value;
The motion vector calculating means has an absolute value of a difference between the correlation value having the strongest correlation and the correlation value having the second highest correlation among the correlation values calculated by the correlation value calculating means being greater than a predetermined threshold A. If the correlation value is small, the motion vector of the previous frame image in time series order is the motion vector of the search frame image, and the correlation value having the strongest correlation among the correlation values indicates a correlation strength A motion vector detection device that does not calculate a motion vector of the search frame image when the correlation is weaker than a threshold value B of the search frame image. The motion vector detection device according to claim 1,
The correlation value calculating unit changes a relative position between the reference frame image and the search frame image within a predetermined range, and a difference average that is an average value of differences in luminance values of pixel data of a predetermined pixel block at each position A motion vector detection apparatus characterized in that a value is the correlation value. A frame image acquisition step of acquiring a reference frame image and a search frame image, which are two frame images moving back and forth in chronological order from a moving image;
A correlation value calculating step of calculating a correlation value between the reference frame image and the search frame image;
A motion vector calculating step of calculating a motion vector indicating a motion of the entire image of the search frame image based on the reference frame image based on the correlation value;
In the motion vector calculation step, an absolute value of a difference between the correlation value having the strongest correlation and the correlation value having the second highest correlation among the correlation values calculated by the correlation value calculation step is smaller than a predetermined threshold A. In this case, the motion vector of the previous frame image in time series order is the motion vector of the search frame image, and the correlation value having the strongest correlation among the correlation values indicates a predetermined correlation strength. A motion vector detection method characterized by not calculating a motion vector of the search frame image when the correlation is weaker than a threshold value B. The motion vector detection device according to claim 1;
A camera shake correction unit that corrects camera shake of the search frame image based on the motion vector;
The camera shake correction device according to claim 1, wherein the camera shake correction unit does not correct the camera shake when the motion vector calculation unit does not calculate the motion vector. The motion vector detection method according to claim 3,
A camera shake correction step of correcting camera shake of the search frame image based on the motion vector,
The camera shake correction method is characterized in that the camera shake correction is not performed when the motion vector calculation step does not calculate the motion vector. 6. A camera shake correction program that causes a computer to execute the camera shake correction method according to claim 5. A moving image display apparatus comprising the camera shake correction apparatus according to claim 4.

Images