JP2010109461A - Device and method for detecting motion vector, image capturing apparatus, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for detecting a motion vector capable of reliably correcting camera shake while removing a panning component. <P>SOLUTION: The device for detecting a motion vector includes: a detection unit 106 for dividing an image into a plurality of block areas to detect a motion vector for every divided block area; a panning decision unit 30 for executing panning decision for every block area; a motion vector classification unit 31 for distributing the motion vectors other than those in the block area determined to be in a panning state on a frequency distribution using the values thereof as classes; and a motion vector calculation unit 32 for defining a representative motion vector based on the value of the frequency. The motion vector classification unit 31 can remove the block area determined to be in the panning state by the panning decision unit 30 from the distribution objects onto the frequency distribution, and thereby prevent hindering of the panning operation of a user in panning imaging. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for detecting a motion vector of an image and correcting blurring of a moving image photographed using the detected motion vector.

  In an imaging apparatus such as a video camera that captures a moving image, there is a problem that the image is blurred due to camera shake, particularly when the lens is zoomed to the telephoto side. In order to prevent image blur due to such camera shake, there has conventionally been proposed a technique for detecting a motion vector of an image from a captured image signal and correcting the blur of the image based on the motion vector.

  As a method for detecting a motion vector of an image, a correlation method based on a correlation calculation, a block matching method, or the like is known.

  In the block matching method, an input image signal is divided into a plurality of block areas (for example, 8 pixels × 8 lines) having an appropriate size. Then, the difference between the previous field (or frame) and a certain range of pixels is calculated in units of blocks, and the block of the previous field (or frame) that minimizes the sum of the absolute values of the differences is searched. And the relative shift | offset | difference between screens has shown the motion vector of the block.

  For matching operations, Morio Onoe et al., Information Processing Vol. 17, no. 7, p. 634-640 July 1976.

  Next, an example of a conventional motion vector detection method using the block matching method will be described with reference to the drawings. FIG. 11 is a schematic block diagram of an apparatus for preventing shake by a conventional motion vector detection method.

  First, an image signal (field or frame) to be detected as a motion vector is applied to the image memory 101 and a filter 102 for extracting a spatial frequency component. The image memory 101 temporarily stores image signals. The filter 102 extracts a spatial frequency component useful for motion vector detection from the image signal. That is, the low spatial frequency component and the high spatial frequency component of the image signal are removed.

  The image signal that has passed through the filter 102 is added to the binarization circuit 103. The binarization circuit 103 binarizes the image signal with reference to the zero level. Specifically, the sign bit of the output signal is output.

  The binarized image signal is added to the correlation calculation circuit 104 and the memory 105 as one field period delay means. The image signal of the previous field is further added from the memory 105 to the correlation calculation circuit 104.

  In accordance with the block matching method, the correlation calculation circuit 104 divides the image area into block areas of an appropriate size as described above, performs a correlation calculation between the current field and the previous field in units of blocks, and moves the correlation value of the result. Add to the vector detection circuit 106. The motion vector detection circuit 106 detects a motion vector in block units from the calculated correlation value. Specifically, the block of the previous field having the smallest correlation value is searched, and the relative shift is used as the motion vector.

  This block-wise motion vector is added to the motion vector determination circuit 107. The motion vector determination circuit 107 determines the entire motion vector from the block-based motion vectors. Specifically, the median value or average value of the motion vectors in units of blocks is used as the motion vector of the entire image.

  The motion vector determination circuit 107 adds the entire motion vector to the memory read control circuit 108. The memory read control circuit 108 controls the read position of the image memory 101 so that the motion of the image is canceled in accordance with the overall motion vector, and an image signal in which the blur is corrected is output from the image memory 101.

  However, in the above method, when panning is performed, for example, if the average value of the detected motion vectors is set to the entire motion vector, the reading position of the image memory is changed so that the screen remains at the original position. Become. As a result, there is a problem that the user's panning operation is hindered.

As a method for solving this problem, Patent Document 1 discloses the following method. In other words, panning determination is performed on peripheral image data including the positions of the four corners of the screen, and if it is determined that the image is in the panning state, the shake correction operation is not performed, thereby preventing the user's panning operation. A method for avoiding this is disclosed.
JP-A-6-350895 Morio Onoe at el. : Image superposition by residual sequential test method, Information processing Vol. 17, no. 7, p. 634-640 July 1976

  In the method of Patent Document 1 described above, when shooting is performed to chase a subject moving in the middle of the screen, it is determined that the area of the image data in the peripheral portion of the screen is in a panning state. When it is determined that the camera is in the panning state, the shake correction operation is not performed, and the user's panning operation can be prevented from being hindered. However, at this time, especially in telephoto shooting, since the shake correction is not performed and the screen shake due to camera shake becomes intense, the position of the subject being shot shakes violently, resulting in a very difficult to see image. there were.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a motion vector detection device capable of reliably performing shake correction while preventing a user's panning operation from being disturbed during panning shooting. It is to be.

  In order to solve the above-described problems and achieve the object, the motion vector detection device according to the present invention divides an image into a plurality of block areas and detects a motion vector for each divided block area. Means, panning determination means for determining whether each block area is panned, and motion vector classification means for allocating the motion vector for each block area on a frequency distribution whose magnitude is a class. Motion vector calculating means for determining a representative motion vector based on the magnitude of the frequency, and the motion vector classification means assigns the block region determined to be panned by the panning determination means to the frequency distribution. It is excluded from the target of upward distribution.

  In addition, an imaging apparatus according to the present invention includes the motion vector detection device described above.

  The motion vector detection method according to the present invention includes a detection step of dividing an image into a plurality of block areas and detecting a motion vector for each of the divided block areas, and panning for each of the block areas. A panning determination step for determining whether or not the motion vector is classified, a motion vector classification step for allocating the motion vector for each block area onto a frequency distribution whose magnitude is a class, and a representative motion based on the magnitude of the frequency A motion vector calculation step for determining a vector, wherein the motion vector classification step excludes a block region determined to be in a panning state by the panning determination step from an object to be distributed on the frequency distribution. And

  According to the present invention, it is possible to reliably perform shake correction while preventing a user's panning operation from being hindered during panning shooting.

  Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a video camera as an imaging device incorporating a motion vector detection device according to an embodiment of the present invention. FIG. 1 is a block diagram illustrating an electrical configuration of a video camera according to an embodiment.

  In FIG. 1, 11 is an imaging optical system comprising a lens and a diaphragm, and 12 is a solid-state imaging device comprising a CCD, a CMOS sensor, or the like. Reference numeral 21 denotes a drive circuit for driving a zoom lens, a focus lens, a diaphragm, etc. (not shown) inside the imaging optical system 11. Reference numeral 23 denotes a drive circuit for driving the image sensor 12. Reference numeral 13 denotes a signal processing circuit that performs signal processing necessary for captured image data. Reference numeral 14 denotes a motion vector detection device that detects a motion vector from the image data signal-processed by the signal processing circuit 13. Reference numeral 15 denotes a memory read control circuit that performs control for determining the range of image data to be recorded or displayed from the motion vector detected by the motion vector detection device 14. Reference numeral 16 denotes an image memory for storing image-processed image data.

  Reference numeral 18 denotes a recording medium composed of a memory card, hard disk, etc., 17 denotes a recording circuit for recording the signal processed image data on the recording medium 18, 20 denotes a display device for displaying the signal processed image data, and 19 denotes a display device. The display circuit 20 displays an image. A system control unit 22 controls the entire video camera.

  Hereinafter, a photographing operation in the video camera configured as described above will be described.

  First, the drive circuit 21 drives a zoom lens, a focus lens, and a diaphragm in the imaging optical system 11 based on a control signal from the system control unit 22 to form a subject image on the imaging element 12.

  The image sensor 12 is driven by a drive pulse generated by a drive circuit 21 controlled by the system control unit 22, photoelectrically converts a subject image into an electrical signal, and outputs an analog image signal. The analog image signal output from the image sensor 12 is converted into a digital image signal by an A / D converter (not shown) inside the signal processing circuit 13.

  Next, the signal processing circuit 13 controlled by the system control unit 22 performs image processing such as color conversion, white balance correction, and gamma correction (not shown), resolution conversion processing, image compression processing, and the like on the digital image signal. Done. The image memory 16 is used for temporarily storing a digital image signal during signal processing or for storing image data which is a digital image signal subjected to signal processing.

  The motion vector detection device 14 detects a motion vector from the image data signal-processed by the signal processing circuit 13 (details will be described later). The memory read control circuit 15 performs control to determine the range of image data to be recorded or displayed based on the motion vector information detected by the motion vector detection device 14 so that the image blur is corrected.

  The image data signal-processed by the signal processing circuit 13 and the image data stored in the image memory 16 are converted into data suitable for recording on the recording medium 18 (for example, file system data having a hierarchical structure) in the recording circuit 17. And recorded on the recording medium 18. Further, after the resolution conversion process is performed by the signal processing circuit 13, the display circuit 19 converts the signal into a signal suitable for the display device 20 (for example, an NTSC analog signal) and displays the signal on the display device 20.

  FIG. 2 is a block diagram illustrating a schematic configuration of the motion vector detection device 14. The same reference numerals as those in FIG. 11 denote the same components, and redundant description is omitted.

  In accordance with the block matching method, the correlation calculation circuit 104 divides the image area into block areas of an appropriate size to divide the image data into a plurality of blocks, and performs a correlation calculation between the current field and the previous field in units of blocks. The resulting correlation value is added to the motion vector detection circuit 106, and the motion vector detection circuit 106 detects a motion vector for each block from the calculated correlation value. The panning determination circuit 30, the motion vector classification circuit 31, and the motion vector calculation circuit 32 are provided in the subsequent stage of the motion vector detection circuit 106, which differs from FIG. 2 and FIG.

  The operation of the panning determination circuit 30 will be described using the flowchart of FIG. The panning determination circuit 30 performs the following processing on all blocks using the block-unit motion vector detected by the motion vector detection circuit 106.

  First, in step S10 of FIG. 3, high frequency component removal processing is performed on the motion vector detected by the motion vector detection circuit 106. A known low-pass filter circuit or the like is used for the high frequency component removal processing.

  In step S11, it is determined whether the output after the high frequency component removal processing (hereinafter referred to as high frequency component removal output) exceeds a predetermined value that is a threshold for performing panning determination. If the high frequency component removal output exceeds a predetermined value in step S11, the process proceeds to step S12. If the high frequency component removal output does not exceed the predetermined value in step S11, the process proceeds to step S13.

  In step S12, it is determined that the block currently undergoing panning determination is in a panning state. In step S13, it is determined that the block currently being panned is not in the panning state.

  Here, the operation of the panning determination circuit 30 will be described in more detail with reference to FIGS. 4 and 5.

  FIG. 4A is a diagram illustrating an example of a block area for detecting a motion vector on a screen of a captured current field. FIG. 4B is a diagram showing a motion vector in each block area obtained by a difference from the immediately preceding field by an arrow.

  In FIG. 4A, it is assumed that the subject 401 existing in the screen is moving leftward, and the photographer is performing panning photographing so that the position of the subject in the screen does not change. In FIG. 4B, since the photographer performs panning photographing so that the position of the subject 401 on the screen does not change, the motion vector output differs between the subject region and the background region, and the subject region 402 is extracted. can do.

  5A and 5B are diagrams showing temporal transition of motion vectors in the block area of the screen. The temporal transition of the motion vector of the block area where the subject exists in FIG. 4A is as shown in the motion vector signal 501 (dotted line in FIG. 5A) of the graph shown in FIG. 5A. In other words, since the photographer performs photographing so that the position of the subject on the screen does not change, the motion vector detection result changes so as to swing positively or negatively across zero. At this time, the result of performing the high-frequency component removal process on the motion vector detection result changes in the vicinity of zero as shown in the high-frequency component removal output 502 (thick line in FIG. 5A) of the motion vector signal in FIG. 5A. .

  On the other hand, the temporal transition of the motion vector of the block area where no subject exists is taken so that the photographer does not change the position of the subject on the screen, so the motion vector detection result of each block without the subject is positive. Or it shifts in the negative direction. At this time, the result of performing the high frequency component removal process on the motion vector detection result also shifts in a positive or negative direction. For example, when the temporal transition of the motion vector in the block area where no subject exists is biased in the negative direction as in the motion vector signal 503 in the graph shown in FIG. 5B, the high-frequency component removal output 504 (see FIG. The 5B thick line) is also biased in the negative direction.

  Further, it is assumed that the panning determination threshold value 500 of the high frequency component removal output is set as shown in FIGS. 5A and 5B. Then, it is determined that the panning state is reached from the point 510 when the high frequency component removal output 504 of the motion vector signal in FIG. 5B exceeds the panning determination threshold value 500. In the example of FIG. 5A, since the high frequency component removal output 502 of the motion vector signal does not exceed the panning determination threshold value 500, panning determination is not performed.

  Next, the operation of the motion vector classification circuit 31 will be described using the flowchart of FIG. The operation of the motion vector calculation circuit 32 will be described with reference to the flowchart of FIG.

  The motion vector classification circuit 31 performs the following processing using the motion vector in units of blocks detected by the motion vector detection circuit 106 and the panning determination result obtained by the panning determination circuit 30.

  First, in step S20 in FIG. 6, it is determined whether or not the number of blocks determined to be in the panning state by the panning determination circuit 30 exceeds a predetermined number, for example, the number of blocks is 90% of the total number of blocks. .

  In step S20, if the number of blocks determined to be in the panning state by the panning determination circuit 30 does not exceed the predetermined number, the process proceeds to step S21.

  In step S21, a histogram is created using the motion vector of the block determined not to be in the panning state in step S20. The creation of a histogram using the motion vector of a block determined not to be in the panning state will be specifically described with reference to FIG.

  FIG. 7A is a histogram (frequency distribution) showing the frequency of the magnitude in the X direction (horizontal direction of the screen) of the motion vectors of all the block areas shown in FIG. 4B. In the histogram, the horizontal axis of the frequency distribution is set as the magnitude of the motion vector, and the class is set by being divided into predetermined magnitudes (ranges). Then, the detected motion vector of each block is determined on the frequency distribution according to which class it belongs to, and plotted as the respective frequency.

  In the example of FIG. 4B, a total of 40 motion vectors are detected as the motion vectors of each block, but the frequency (frequency) of each class is distributed and plotted to the class to which the 40 frequencies belong on the frequency distribution. Can be obtained. As the class value of each class having a range, an intermediate value between the upper limit value and the lower limit value of the class may be used. In addition, since processing in the Y direction (vertical screen direction) and processing in the X direction (horizontal direction in the screen) perform the same processing, only the processing in the X direction will be described below as an example.

  FIG. 7A is a histogram created by using the motion vectors of all block areas as a histogram creation target. Therefore, there are two groups that show different movements: a group 11 corresponding to the subject area and a group 10 corresponding to the background area where no subject exists. If the average value of all the motion vectors including all of the group 10 and the group 11 is the entire motion vector, a panning component is superimposed on the motion vector, which hinders the photographer's panning operation. Become.

  Also, consider the case where only the peripheral part of the screen is used as disclosed in Patent Document 1. In this case, in the histogram of FIG. 7A, the group 11 corresponding to the subject area is not subject to motion vector detection because it is located in the center of the screen, and only blocks belonging to the group 10 that is the background area are subjected to motion vector detection. It becomes a target. At this time, since the background area of the group 10 is determined to be in the panning state, the image is not subjected to the shake correction and the camera shake remains as it is.

  On the other hand, in the present invention, the motion vector of the block determined to be in the panning state is excluded in step S21 in the flowchart of FIG. Therefore, a histogram is created using the motion vector of the block determined not to be in the panning state. As a result, as shown in FIG. 7B, only the group 11 corresponding to the subject region is the target of histogram creation. In addition, the motion vectors of the blocks determined to be in the panning state, that is, the blocks belonging to the group 10 in FIG. 7A are not subjected to histogram creation. This makes it possible to extract only the blur of the subject area that does not include the panning component.

  In step S20, when the number of blocks determined to be in the panning state exceeds a predetermined number, the process proceeds to step S22, and a histogram is generated using the motion vector of the block determined to be in the panning state. .

  For example, as shown in FIG. 8A, when panning shooting is performed in a state where there is no moving object in the screen, the motion vector detection result in each block region is as shown in FIG. 8B. At this time, in the process of step S22, since the panning determination circuit 30 determines that all the block areas are in the panning state, the motion vector detection result of the block area determined to be in the panning state as shown in FIG. 8C. Create a histogram using.

  Next, the motion vector calculation circuit 32 performs the processing shown in the flowchart of FIG.

  First, in step S30, it is determined whether or not the number of blocks determined to be in the panning state by the panning determination circuit 30 exceeds a predetermined number. The predetermined number here is the same numerical value as the predetermined number in step S20 of FIG. In step S30, if the number of blocks determined to be in the panning state by the panning determination circuit 30 does not exceed the predetermined number, the process proceeds to step S31.

  Note that in both step S20 and step S30, it is determined whether or not the number of blocks determined to be in the panning state by the panning determination circuit 30 exceeds a predetermined number. Therefore, among the processes of FIGS. 7 and 9, the process of step S31 may be performed after step S21, and the process of step S32 may be performed after step S22. Also in this case, as described above, the panning determination circuit 30 performs determination in step S20 (step S30). In addition, the motion vector classification circuit 31 performs processing in steps S21 and S22, and the motion vector calculation circuit 32 performs processing in steps S31 and S32 as described later.

  In step S31, the extreme value of the histogram using the motion vector of the block determined not to be in the panning state created in step S21 is used as a motion vector that represents the entire screen (hereinafter referred to as a representative motion vector). . For example, an extreme value (peak) is searched for a histogram created using a motion vector of a block determined not to be in the panning state as shown in FIG. 7B, and this extreme value is set as a representative motion vector.

  In step S30, if the number of blocks determined to be in the panning state by the panning determination circuit 30 exceeds a predetermined number, the process proceeds to step S32.

  In step S32, the value obtained by removing the low frequency component from the extreme value of the histogram created in step S22 is set as the representative motion vector.

  As shown in FIG. 10A, the motion vector 1000 indicated by the extreme value of the histogram created in step S22 is a value on which a panning component (low frequency component) is superimposed. If the number of blocks determined to be in the panning state by the panning determination circuit 30 exceeds a predetermined number, the panning component is removed by performing a process of removing the low frequency component by the motion vector calculation circuit 32. be able to. The motion vector signal from which the panning component has been removed by this low frequency component removal becomes a graph with no bias in either positive or negative as shown by the motion vector signal low frequency component removal component 1001 in FIG. 10B. A known high-pass filter circuit or the like is used for the process of removing the low frequency component in step S32.

  Then, according to the representative motion vector, the memory read control circuit 15 performs control to determine the range of image data to be recorded or displayed so that image blur is corrected. That is, the reading position of the image memory 101 is controlled so that image blur is canceled out. As a result, it is possible to reliably perform the blur correction of the subject area without hindering the user's panning operation.

  As described above, the process of step S22 in FIG. 6 and step S32 in FIG. 9 does not hinder the user's panning operation even in the case of panning shooting that is not in a shooting state in which the moving subject is chased. In addition, high-frequency blur correction that is not a panning component can be reliably performed.

  As described above, by using the motion vector detection device of the present invention, a histogram is not created in a motion vector detection block region determined to be in a panning state. Therefore, since the histogram is created only in the block area for motion vector detection that has not been determined to be in the panning state, blurring of the subject area can be extracted more reliably. Therefore, it is possible to provide an imaging apparatus capable of reliably performing blur correction on a subject area without hindering a user's panning operation.

(Other examples)
In addition, this invention can be achieved also with the following structures.

  It is assumed that a storage medium storing software program codes for realizing the functions of the above-described embodiments is supplied to a computer or an apparatus. Thereby, the control unit of the computer or apparatus can also be achieved by reading and executing the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the storage medium storing the program code of the software realizing the function of the above-described embodiment constitutes the present invention. Will do.

  As a storage medium for supplying the program code, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, or the like is used. be able to.

It is a block diagram which shows the structure of the video camera as an imaging device incorporating the motion vector detection apparatus concerning one Embodiment of this invention. It is a block diagram which shows schematic structure of a motion vector detection apparatus. It is a flowchart of panning determination. A is a figure which shows the state of the screen in panning operation | movement when a moving object exists in a screen. FIG. 4B is a diagram showing a motion vector distribution in the case of FIG. 4A. FIG. 4A is a diagram showing a temporal transition of a motion vector in a block area where a subject exists in FIG. 4A. B is a diagram showing a temporal transition of a motion vector of a block area where no subject exists by the panning determination circuit 30. FIG. 6 is a flowchart relating to a histogram creation process based on the number of blocks that are assumed to be in a panning state by a motion vector classification circuit 31; FIG. 4A is a diagram showing a histogram of motion vectors before panning determination in the case of FIG. 4A. FIG. 4B is a diagram showing a motion vector histogram after panning determination in the case of FIG. 4A. A is a figure which shows the state of the screen in panning operation | movement when a moving object does not exist in a screen. B is a diagram showing a motion vector distribution in the case of FIG. 8A. C is a diagram showing a histogram of motion vectors in the case of FIG. 8A. 5 is a flowchart illustrating a method for determining a representative vector by a motion vector classification circuit 31. FIG. 8A is a diagram showing the passage of time of the magnitude of the motion vector indicated by the extreme value of the histogram in the case of FIG. 8A. FIG. 8B is a diagram illustrating a time lapse of the magnitude of the motion vector indicated by the extreme value of the histogram when the low frequency component is removed from the motion vector signal in the case of FIG. 8A. It is a schematic block diagram of the apparatus which prevents blurring by the conventional motion vector detection method.

Explanation of symbols

401 Subject 402 Subject region 1000 Motion vector signal 1001 Motion vector signal low frequency component removal output

Claims (8)

Detecting means for dividing an image into a plurality of block areas and detecting a motion vector for each of the divided block areas;
Panning determination means for determining whether panning is performed for each of the block regions;
A motion vector classifying means for distributing the motion vector for each block area onto a frequency distribution having a magnitude as a class;
Motion vector calculation means for determining a representative motion vector based on the magnitude of the frequency,
The motion vector classifying unit excludes a block area determined to be in a panning state by the panning determination unit from an object to be distributed on the frequency distribution. The panning determination means includes high frequency component removal means for removing a high frequency component of a motion vector detected by the motion vector detection means for each of the block regions.
2. The motion vector detection device according to claim 1, wherein a block region in which an output of the high frequency component removing unit exceeds a predetermined value is determined to be in a panning state. As a result of the determination by the panning determination means, when the number of block areas determined to be in the panning state exceeds a predetermined number, the motion vector classification means determines the motion vector for each block area determined to be in the panning state. Are distributed on the frequency distribution with the size as the class,
The said motion vector calculating means determines a representative motion vector based on the magnitude | size of the said frequency from the motion vector distributed on the said frequency distribution by the said motion vector classification | category means. Motion vector detection device. The motion vector calculation means includes low frequency component removal means for removing a low frequency component of the motion vector,
The motion vector calculating means determines a motion vector obtained by removing a low frequency component by the low frequency component removing means as a representative motion vector from one motion vector determined based on the magnitude of the frequency. Item 4. The motion vector detection device according to Item 3. The motion vector detection device according to any one of claims 1 to 4,
An image pickup apparatus, comprising: correction means for correcting image blur based on a motion vector detected by the motion vector detection apparatus. A detection step of dividing an image into a plurality of block areas and detecting a motion vector for each of the divided block areas;
For each of the block regions, a panning determination step for determining whether panning is performed,
A motion vector classification step of allocating the motion vector for each block region onto a frequency distribution having a magnitude as a class;
A motion vector calculation step for determining a representative motion vector based on the magnitude of the frequency,
The motion vector classification method, wherein the motion vector classification step excludes a block region determined to be in a panning state by the panning determination step from an object to be distributed on the frequency distribution.   A program for causing a computer to execute the motion vector detection method according to claim 6.   A computer-readable storage medium storing the program according to claim 7.

Images