KR20120045812A - Method for detecting flicker using motion compensation and apparatus for performing the same - Google Patents

Abstract

PURPOSE: A flicker detecting method using motion compensation and a device thereof are provided to completely remove the flicker by exposure compensation based on an accurately detected flicker cycle. CONSTITUTION: A motion vector extracting unit obtains a motion vector based on an image of consecutive two frames offered from an image sensor(100). A flicker cycle detecting unit calculates the SAD(Sum of Absolute Difference) of vertical line average values corresponding to the two frames by using an obtained motion vector(200). The flicker cycle detecting unit detects a flicker cycle(300).

Description

Flicker detection method using motion compensation and apparatus for performing the same {METHOD FOR DETECTING FLICKER USING MOTION COMPENSATION AND APPARATUS FOR PERFORMING THE SAME}

The present invention relates to an image sensor, and more particularly, to a flicker detection method using a motion compensation that can accurately detect flicker using motion compensation, and an apparatus for performing the same.

CMOS image sensor (image sensor) is a device for imaging the image using the nature of the semiconductor in response to light, and converts the electrical signal corresponding to the brightness and wavelength of light to a level capable of signal processing and outputs.

The CMOS image sensor includes a plurality of pixel lines including a plurality of pixels, and operates by an electronic rolling shutter method of sequentially generating and outputting image signals of the pixel lines by sequentially exposing the pixel lines to light. do.

However, in a case where the CMOS image sensor operates in a room using a fluorescent lamp or an incandescent lamp as a light source, the fluorescent lamp or the incandescent lamp periodically changes its intensity according to the frequency of the supplied AC power. When the intensity of the light source does not match the exposure time of each line of the CMOS image sensor, each pixel line of the CMOS image sensor varies in the amount of light condensed by the difference in the intensity of the light source, resulting in a difference in luminance. As described above, the luminance difference of each pixel line generates streaks in the entire image generated by the CMOS image sensor. Such streaks are referred to as flicker noise.

In order to remove the above-mentioned flicker noise, the flicker is removed by determining the flicker period and adjusting the exposure time using the determined flicker period. However, the above-described conventional flicker removal method cannot accurately detect the flicker period because only the horizontal line change is taken in the continuous image provided by the image sensor in the process of detecting the flicker period, and thus the flicker is completely removed. There is a disadvantage that is not.

The present invention provides a flicker detection method using motion compensation that can accurately detect a flicker period and thereby completely eliminate flicker.

Further, another apparatus of the present invention is to provide a flicker detection apparatus using motion compensation capable of accurately detecting a flicker period and completely removing flicker.

Technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to the flicker detection method using the motion compensation according to an aspect of the present invention for achieving the above object of the present invention, the image of the n th and n + 1 th frame (where n is a natural number of 1 or more) provided continuously Obtaining a motion vector based on each vertical line average value of the nth frame and each vertical line of the n + 1th frame corresponding to each vertical line of the nth frame that compensates for the motion using the motion vector; Computing a sum of absolute difference (SAD) of the average value of the line and detecting a flicker period based on the calculated change in the plurality of SAD values.

In addition, according to the flicker detection apparatus using the motion compensation according to an aspect of the present invention for achieving another object of the present invention, the n th frame and n + 1 th frame provided continuously from the image sensor (where n is 1 or more) A motion vector extracting unit for obtaining a motion vector based on a natural number, and the n corresponding to each vertical line of the nth frame that compensates for the motion by using an average value of each vertical line of the nth frame and the motion vector. A flicker period detector for calculating a SAD of an average value of each vertical line of a + 1th frame, detecting a flicker period based on the calculated change of the plurality of SAD values, and a storage unit for storing the average value provided by the motion vector extractor; do.

According to a flicker detection method using a motion compensation and an apparatus for performing the same, a boundary detection area and a line average value calculation area are divided into at least two consecutive frames, and the boundary detection area is detected and based on the detected boundary. Estimate the motion vector. After the image data is removed by calculating SADs of the line average values on which the horizontal line motion compensation is performed using the estimated motion vector, the flicker period is detected based on the plurality of calculated SAD values.

Therefore, the flicker period can be accurately detected using motion compensation, and flicker can be completely removed by performing exposure compensation based on the accurately detected flicker period.

1 is a flowchart illustrating a flicker detection method using motion compensation according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating the motion vector acquisition process shown in FIG. 1 in more detail.
3 is a conceptual diagram illustrating a process of classifying image regions of a frame illustrated in FIG. 2.
4 is a conceptual diagram illustrating a flicker detection method using motion compensation according to an embodiment of the present invention.
5 is a block diagram showing the configuration of a flicker detection apparatus using motion compensation according to an embodiment of the present invention.

As the present invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description.

However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. In describing the present invention, in order to facilitate the overall understanding, the same reference numerals are used for the same elements in the drawings, and redundant description of the same elements is omitted.

1 is a flowchart illustrating a flicker detection method using motion compensation according to an embodiment of the present invention, and FIG. 2 is a flowchart illustrating a motion vector acquisition process shown in FIG. 1 in more detail. 3 is a conceptual diagram illustrating a process of classifying image regions of a frame illustrated in FIG. 2, and FIG. 4 is a conceptual diagram illustrating a flicker detection method using motion compensation according to an exemplary embodiment of the present invention.

1 to 4, a flicker detection method using motion compensation acquires a motion vector based on an image of two consecutive frames provided from an image sensor (step 100), and successively uses the obtained motion vector. After removing the image data by calculating the SAD of the average values of the vertical lines corresponding to each other in the two frames (step 200), the flicker period is detected based on the calculated change in the plurality of SAD values (step 300), and the detected flicker is detected. Flicker is removed by correcting the exposure based on the period (step 400).

Referring to FIG. 2, the motion vector obtaining step (step 100) illustrated in FIG. 1 will be described in more detail. The flicker detection apparatus using the motion vector (hereinafter, abbreviated as 'flicker detection apparatus') is provided from an image sensor. The first frame is divided into a boundary detection area and a line average value calculation area as shown in Fig. 3 (step 101).

The edge detection area is an edge detection area, and when a boundary is detected in the edge detection area of the n + 1th frame, the motion vector is estimated by comparing with the boundary detection area of the nth frame based on the detected edge position. Used for.

The line average value calculating area is an area used for calculating the period of the flicker component, and is divided into a vertical line average value calculating area, a horizontal line average value calculating area, and an additionally adjustable line average value calculating area.

Thereafter, the flicker detection apparatus calculates and stores an average value of vertical and horizontal lines included in the boundary detection area of the nth frame (step 103). Here, the vertical and horizontal line average values calculated in the boundary detection area of the n th frame are used to estimate the motion vector based on the boundary position information obtained in the boundary detection area of the n + 1 th frame.

In addition, the flicker detection apparatus calculates an average value of the vertical lines in the line average value calculation area (step 105). Here, the line average value calculated in each line may include a high frequency component to reduce the flicker detection probability when comparing the average value with the next frame n + 1 frame. The low frequency filtering is performed and then sequentially stored (step 107).

Thereafter, the flicker detection apparatus divides the image area for the n + 1th frame in the same manner as the nth frame (step 109), and detects an edge in the boundary detection area of the n + 1th frame (step 111). ). In operation 111, the flicker detection apparatus may detect the boundary through various known methods. For example, the flicker detection apparatus may compare the pixel values between adjacent pixels and determine the boundary pixel when the difference in the pixel values is greater than or equal to a preset threshold. Here, the boundary component means a position where the boundary occurs prominently in a plurality of vertical and horizontal lines.

Thereafter, the flicker detection apparatus calculates an average value of vertical and horizontal lines positioned at the boundary detected in the boundary detection area of the n + 1th frame (step 113).

As described above, when the average value of the vertical and horizontal lines corresponding to the boundary detection and the detected boundary in the boundary detection region of the n + 1th frame is finished, the flicker detection apparatus is configured to vertical and horizontal of the boundary detection region of the stored nth frame. Based on the boundary position obtained in the n + 1th frame among the line average values, vertical and horizontal line average values around a preset maximum motion vector moving distance are sequentially read from the storage unit (step 115).

Thereafter, the flicker detection apparatus takes a sum of absolute difference (SAD) with respect to the vertical and horizontal line average values of the n-th frame and the vertical and horizontal line average values corresponding to the boundary position of the n + 1-th frame, and the resulting value is The distance traveled by the vertical and horizontal lines of the n-th frame having the minimum value in the n + 1-th frame is calculated as the motion vector values in the vertical and horizontal directions (step 117).

The motion vector value calculated in step 117 is used as a correction value for compensating for motion in an area except the boundary detection area. For example, as shown in FIG. 4, when the motion vector has a horizontal value of -4 vertically to 0, the pixel starting position in which the pixel starting position for calculating the average value of the vertical line in the n + 1th frame is specified in the previous frame. When the line average value is calculated by starting reading four pixels earlier, the vertical line average value to which the horizontal line compensation is applied is calculated.

Thereafter, the flicker detection apparatus calculates a vertical line average value in the line average value calculation region of the n + 1th frame (step 119), performs low frequency filtering on the calculated vertical line average value to increase the flicker detection probability, and stores the same.

Referring back to FIG. 1, in operation 200, the flicker detection apparatus calculates SADs of vertical line average values of the n + 1th frame whose motion is compensated by applying the vertical line average values calculated in the nth frame and the motion vector. Here, the SADs of the vertical line average values sequentially calculated include only the flicker component from which image data is removed.

Further, in step 300, the flicker removal apparatus compares SADn-1 of the previous vertical line average value and SADn + 1 of the subsequent vertical line average value based on the SADn value of the predetermined vertical line average value to detect the flicker period, so that the SADn value is SADn−. If less than 1 and SADn + 1, the SADn value is determined as a valley, and the number of lines between the previous goal and the current goal is counted. The flicker period is detected by judging a count value larger than the flicker determination reference value among the counted values as a flicker period having a probabilistic representation.

In addition, the flicker detection method using motion compensation according to an embodiment of the present invention does not perform flicker detection when the illuminance is less than or equal to a predetermined reference illuminance. In other words, when the illuminance is low, because the frame speed is low, a large motion error occurs between two consecutive frames, and a large amount of system resources are required to extract the frame from such a frame.

5 is a block diagram showing the configuration of a flicker detection apparatus using motion compensation according to an embodiment of the present invention.

Referring to FIG. 5, the flicker detection apparatus according to the exemplary embodiment of the present invention may include an image sensor 510, a motion vector extractor 520, a flicker period detector 530, and a storage 540.

The image sensor 510 may be configured as a Charge Coupled Device (CCD) or a Complementary Metal Oxide (CMOS) image sensor, and converts and outputs an electrical signal corresponding to the brightness of incident light into a digital signal. Here, the image sensor may be configured as a Bayer pattern, and operate in an electronic rolling shutter method to output Bayer image data corresponding to brightness of incident light.

The motion vector extractor 520 obtains a motion vector based on the images of two consecutive frames provided from the image sensor 510. The motion vector extractor 520 may include an average value calculation module 521, a low frequency filter 523, a boundary detection module 525, and a motion vector estimation module 527.

Specifically, the average value calculation module 521 divides the frame detection area and the line average value calculation area for each frame provided from the image sensor 510, and calculates an average value of vertical and horizontal lines included in the boundary detection area of the nth frame. Calculate. The average value calculating module 521 calculates the average value of the vertical lines in the line average value calculating region of the n-th frame.

In addition, the average value calculating module 521 calculates an average value of vertical and horizontal lines positioned at the boundary detected in the boundary detection area of the n + 1th frame, and calculates a vertical line average value in the line average value calculation area of the n + 1th frame. Calculate.

As described above, since the line average value calculated by the average value calculation module 521 may include a high frequency component to reduce the flicker detection probability, low frequency filtering is performed.

The low frequency filter 523 performs low frequency filtering on the line average value provided from the average value calculation module 521 and stores the low frequency filter in the storage unit 540.

The edge detection module 525 detects an edge in the boundary detection area of the N + 1 th frame provided from the image sensor 510, and then estimates the position information of the detected boundary by the average value calculation module 521 and the motion vector estimation. To module 527. Here, the boundary detection module 525 may detect the boundary through various known methods. For example, the boundary detection module 525 may compare the pixel values between adjacent pixels and determine the boundary pixel when the difference between the pixel values is greater than or equal to a preset threshold.

The motion vector estimation module 527 is provided with the position information of the boundary detected by the boundary detection module 525 and n + 1th of the average value of the vertical and horizontal lines of the boundary detection area of the nth frame stored in the storage unit 540. Based on the boundary position obtained from the frame, the vertical and horizontal line average values around the preset maximum motion vector moving distance are sequentially read from the storage unit, and then the vertical and horizontal line average values of the n th frame are read and the n + 1 th frame. SAD is taken for the average value of the vertical and horizontal lines corresponding to the boundary positions of, and the distance of the vertical and horizontal lines of the n-th frame having the minimum value is moved from the n + 1th frame to the motion vector in the vertical and horizontal directions. Calculate by value.

The flicker period detector 530 removes the image data by calculating the SAD of the average value of each vertical line corresponding to each other in two consecutive frames using the motion vector provided from the motion vector extractor 520 to remove only the flicker component in each line. After leaving, the flicker period is detected based on the calculated change in the plurality of SAD values. Here, the detected flicker period may be an exposure correction for removing flicker.

In detail, the flicker period detector 530 calculates SADs of the vertical line average values of the n + 1th frame whose motion is compensated by applying the vertical line average values calculated in the nth frame and the motion vector. Here, the SADs of the vertical line average values sequentially calculated include only the flicker component from which image data is removed.

In addition, the flicker period detector 530 may determine the SADn-1 of the previous vertical line average value and the subsequent vertical line average value based on the SADn value which is the average value of the predetermined vertical line of the nth frame and the vertical line of the n + 1th frame corresponding thereto. If the SADn value is smaller than SADn-1 and SADn + 1 by comparing SADn + 1, the SADn value is determined as a goal, the number of lines between the previous goal and the current goal is counted, and the preset flicker determination reference value among the counted values is determined. A flicker period is detected by judging the largest and largest count value as a flicker period having a probabilistic representation.

The storage unit 540 may be configured as a nonvolatile memory, and stores an average value of each line provided from the motion vector extractor 520.

The motion vector extractor 520 and the flicker period detector 530 illustrated in FIG. 5 may be configured as one image signal processor chip that performs image signal processing. In this case, the image signal processor may include the image signal processor illustrated in FIG. In addition to the functional block, a component that performs various known functions may be further included. For example, the image signal processor may further include components that perform functions such as color interpolation, luminance processing, color processing color format, etc. in addition to the motion vector extractor 520 and the flicker period detector 530 illustrated in FIG. 5. have.

Although described with reference to the embodiments above, those skilled in the art will understand that the present invention can be variously modified and changed without departing from the spirit and scope of the invention as set forth in the claims below. Could be.

510: image sensor 520: motion vector extraction unit
521: average value calculation module 523: low frequency filter
525: boundary detection module 527: motion vector estimation module
530: flicker period detection unit 540: storage unit

Claims (11)

Obtaining a motion vector based on the images of the n th and n + 1 th frames, wherein n is a natural number greater than or equal to 1;
Sum of Absolute Difference of an average value of each vertical line of the n + 1th frame corresponding to each vertical line of the nth frame that compensates for the motion by using the motion vector and the average value of the vertical line of the nth frame Calculating c); And
A flicker detection method using motion compensation, the method comprising: detecting a flicker period based on the calculated change in the plurality of SAD values. The method of claim 1, wherein the obtaining of the motion vector comprises:
Dividing the n-th frame into a boundary detection area and a line average value calculation area;
Calculating an average value of vertical and horizontal lines of the boundary detection area of the nth frame; And
And calculating an average value of a vertical line of the line average value calculating region of the nth frame. The method of claim 2, wherein obtaining the motion vector
And performing low frequency filtering on the calculated average value of the vertical line and then storing the flicker. The method of claim 2, wherein obtaining the motion vector
After calculating the average value of the vertical lines of the line average value calculation area of the nth frame,
Dividing the n + 1th frame into a boundary detection area and a line average value calculation area;
Detecting an edge in the boundary detection region of the n + 1th frame;
Calculating an average value of vertical and horizontal lines existing at positions of boundaries detected in the boundary detection region of the n + 1th frame;
Reading an average value of vertical and horizontal lines corresponding to the detected boundary position among average values of vertical and horizontal lines of the boundary detection area of the nth frame; And
And obtaining a motion vector by taking an SAD of an average value of vertical and horizontal lines corresponding to each of the nth frame and the n + 1th frame, respectively. 5. The method of claim 4, wherein the obtaining of the motion vector by taking SAD of the average values of the vertical and horizontal lines corresponding to the nth frame and the n + 1th frame, respectively,
Based on the position of the boundary obtained in the n + 1th frame among the average values of the vertical and horizontal lines of the boundary detection area of the nth frame, the vertical and horizontal line average values corresponding to the preset maximum motion vector moving distance are sequentially read. Shipping step;
A SAD calculating step of taking an SAD for the average value of the vertical and horizontal lines of the n-th frame and the average value of the vertical and horizontal lines corresponding to the position of the boundary of the n + 1-th frame; And
And calculating a distance of the vertical and horizontal lines of the n-th frame having the minimum SAD value calculated in the SAD calculation step as the motion vector values in the vertical and horizontal directions. Flicker detection method using motion compensation. 5. The method of claim 4, wherein obtaining the motion vector
Calculating an average value of vertical lines of the line average value calculating region of the n + 1th frame; And
And performing low frequency filtering on the average value of the vertical lines of the calculated line average value region of the n + 1 th frame, and storing the average value of the vertical line. The method of claim 1, wherein the detecting of the flicker period based on the calculated change in the plurality of SAD values comprises:
Comparing a SAD value of a predetermined vertical line average value with a SAD value of a previous vertical line average value of the predetermined vertical line and a SAD value of a subsequent vertical line average value of the predetermined vertical line;
Determining the SAD value of the predetermined vertical line average value as a valley when the SAD value of the predetermined vertical line average value is smaller than the SAD value of the previous and subsequent vertical line average values;
Counting the number of lines between the determined plurality of goals; And
And determining a count value that is larger than a preset flicker determination reference value among the counted values as a flicker period. A motion vector extracting unit obtaining a motion vector based on the nth frame and n + 1th frame (where n is a natural number of 1 or more) continuously provided from the image sensor;
Calculating a SAD of an average value of each vertical line of the n + 1th frame corresponding to each vertical line of the nth frame that compensates for the motion by using the average value of each vertical line of the nth frame and the motion vector, A flicker period detector for detecting a flicker period based on the calculated change in the plurality of SAD values; And
And a storage unit for storing the average value provided by the motion vector extractor. The method of claim 8, wherein the motion vector extraction unit
The nth and n + 1th frames are divided into a boundary detection area and a line average value calculation area, and an average value of vertical and horizontal lines included in the boundary detection area and an average value of vertical lines included in the line average value calculation area are calculated. An average value calculating module;
A low frequency filter performing low frequency filtering on the average value of the vertical line provided from the average value calculating module;
An edge detection module for detecting an edge in an edge detection area of the n + 1th frame; And
And a motion vector estimation module for extracting a motion vector based on the detected position of the boundary. The method of claim 9, wherein the motion vector estimation module comprises:
The storage unit sequentially calculates vertical and horizontal line average values corresponding to a maximum motion vector moving distance that are preset based on the position of the detected boundary among the vertical and horizontal line average values of the boundary detection area of the nth frame stored in the storage unit. After reading, SAD is taken for the average value of the vertical and horizontal lines of the read n-th frame and the average value of the vertical and horizontal lines corresponding to the boundary position of the n + 1-th frame, and the resultant value of the n-th frame having the minimum value. A flicker detection apparatus using motion compensation, wherein the distance traveled by the vertical and horizontal lines in the n + 1th frame is calculated as a motion vector value. The method of claim 8, wherein the flicker period detection unit,
The SAD value of a predetermined vertical line average value is compared with the SAD value of a previous vertical line average value of the predetermined vertical line and the SAD value of a subsequent vertical line average value of the predetermined vertical line, and the SAD value of the predetermined vertical line average value is before and after When the SAD value of the vertical line average value is smaller than SAD, the SAD value of the predetermined vertical line average value is determined as a valley, and then the number of lines between the plurality of valleys is counted, and is greater than the preset flicker determination reference value among the counted values. A flicker detection apparatus using motion compensation, wherein the largest count value is determined as a flicker period.

Images