JP6462281B2 - Flicker reduction device - Google Patents

Description

  The present invention relates to a flicker reducing apparatus for reducing flicker appearing in a moving image taken under illumination whose light intensity varies in relation to an AC power supply frequency.

The intensity of the AC component of the light emitted from the luminaire driven by the AC power supply varies at a frequency twice as high as the power supply frequency f _A (Hz). Under this illumination, a moving image photographed at a frame frequency f _V (Hz) becomes a video including flicker having a flicker frequency f _F (Hz) represented by Expression (1).

With an ideal video signal when the flicker is not Vo (x, y, t) and the video signal V _F containing flicker (x, y, t) is the flicker component as shown in Equation (2) F _L It is represented by the sum of (x, y, t). Here, x, y, and t are coordinates in the horizontal direction, vertical direction, and time direction of the video signal, respectively.

Further, the flicker component F _L (x, y, t) is expressed by Expression (3). Here, θ represents a phase.

When the frame frequency f _{V of the} video is lower than the flicker frequency 2f _{A of the} illumination, a flicker is generated in the video signal by using a shutter having a time aperture of 1 / 2f _A (<1 / f _V ). be able to. However, when the video frame frequency f _V is higher than the lighting flicker frequency 2f _A , 1 / 2f _A > 1 / f _V , so it cannot be solved by the shutter, and flicker is eliminated by video signal processing. It is necessary to reduce and eliminate.

When shooting at a high speed for slow motion shooting, or when using a frame frequency higher than the conventional frame frequency to improve moving image quality, the frame frequency f _{V of the} video is higher than the flicker frequency 2f _{A of the} illumination. Can be. For example, when shooting at a frame frequency of 180 Hz under an AC power supply frequency of 60 Hz (lighting blinking frequency is 120 Hz), or when shooting at a frame frequency of 120 Hz under an AC power supply frequency of 50 Hz (lighting blinking frequency is 100 Hz) It corresponds to.

If the flicker component F _L (x, y, t), that is, the flicker frequency f _F , the phase θ, and the amplitude F _L can be detected from the video signal V _F (x, y, t), these are removed from the moving image. By doing so, a video signal V _O (x, y, t) without flicker can be obtained. If flicker is uniform within the screen, estimates the video signal V _F (x, y, t) from F _L and theta, are capable of removing flicker by equation (4).

Japanese Patent No. 4371108 Japanese Patent No. 4423889

  However, when shooting a moving image, the illumination conditions are not always constant. For example, it is not always illuminated uniformly within the screen, and the reflectance of the subject varies, and the influence of illumination within the screen changes from moment to moment depending on the operation of the camera and the movement of the subject. Therefore, the flicker does not appear uniformly on the screen and changes with time. Further, if a common AC power source is used for illumination, the frequency of change in light intensity emitted from a plurality of lighting fixtures is the same, but the phase is not necessarily the same. Therefore, the phase of the temporal fluctuation of flicker differs within the screen. In other words, the flicker in the equation (3) is actually a function of the position and time in the space as in the equation (5).

Then, from the video signal V _F (x, y, t) represented by the equation (2), an ideal video signal V _O (x, y, t) and a flicker component F _L (x, y, t) Is difficult to separate.

  An object of the present invention made in view of such circumstances is to provide a flicker reduction apparatus and program capable of solving the above-described problems and reducing / removing flicker that appears unevenly in a screen. .

In order to solve the above problems, a flicker reduction apparatus according to the present invention includes a frequency converter that converts a moving image into a signal in a spatio-temporal frequency domain including a temporal frequency and a spatial frequency , and a known flicker frequency among the temporal frequencies. Flicker component estimation in which the temporal frequency corresponding to the frequency component of the suppression frequency is determined from the frequency component of the temporal frequency near the suppression frequency in the signal in the spatio-temporal frequency domain And a flicker removing unit that generates a frequency domain signal in which a frequency component of a spatial frequency in a predetermined range is suppressed according to the suppression amount among the frequency components of the suppression frequency for the signal in the spatio-temporal frequency domain, and An inverse frequency transform unit that transforms the frequency domain signal generated by the flicker removal unit into a real space domain signal. The features.

Furthermore, in the flicker reducing apparatus according to the present invention, the frequency conversion unit converts the moving image into a signal in a three-dimensional space-time frequency domain of a horizontal frequency, a vertical frequency, and a time frequency, and the flicker removal unit includes : among the frequency components of the previous SL suppression frequency, characterized by suppressing the frequency components in the horizontal and vertical frequency in a predetermined range.

Further, the flicker reduction apparatus according to the present invention, the frequency conversion unit, the moving image is converted into a signal of when the spatial frequency domain two-dimensional horizontal frequency and temporal frequency, the flicker removal unit prior Symbol suppression frequency Among the frequency components, a frequency component of a horizontal frequency within a predetermined range is suppressed.

Further, the flicker reduction apparatus according to the present invention, the frequency conversion unit, the moving image is converted into a signal of when the spatial frequency domain two-dimensional vertical frequency and the temporal frequency, the flicker removal unit prior Symbol suppression Among the frequency components of the frequency, the frequency component of the vertical frequency within a predetermined range is suppressed.

  In order to solve the above problem, a program according to the present invention causes a computer to function as the flicker reduction apparatus.

  According to the present invention, it is possible to obtain a video signal that effectively reduces and eliminates flicker that appears unevenly in a screen.

It is a block diagram which shows the structural example of the flicker reduction apparatus which concerns on one Embodiment of this invention. It is a figure explaining the process which converts a moving image into the signal of the three-dimensional space-time frequency domain of a horizontal frequency, a vertical frequency, and a time frequency in the flicker reduction apparatus which concerns on one Embodiment of this invention. It is a figure explaining the process which converts a moving image into the signal of the two-dimensional space-time frequency domain of a horizontal frequency and a time frequency in the flicker reduction apparatus which concerns on one Embodiment of this invention. It is a figure explaining the process which converts a moving image into the signal of the two-dimensional space-time frequency domain of a vertical frequency and a time frequency in the flicker reduction apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the frequency component with respect to time frequency in the video signal containing a flicker. It is a figure which shows the three-dimensional space-time frequency area | region converted by the frequency conversion process in the flicker reduction apparatus which concerns on one Embodiment of this invention. It is a figure which shows the suppression area | region of the frequency component at the time of converting a moving image into the signal of a three-dimensional space-time frequency domain in the flicker reduction apparatus which concerns on one Embodiment of this invention. In the flicker reduction device according to an embodiment of the present invention, it is a diagram showing a frequency component suppression region when a moving image is converted into a signal in a two-dimensional spatio-temporal frequency region of horizontal frequency and temporal frequency. FIG. 5 is a diagram illustrating a frequency component suppression region when a moving image is converted into a two-dimensional spatio-temporal frequency domain signal of a vertical frequency and a temporal frequency in the flicker reduction apparatus according to an embodiment of the present invention.

The video signal V _F of Expression (2) is expressed as Expression (6) in the spatio-temporal frequency domain.

Since the spread in the spatial frequency domain when the flicker component F _L is small compared to the distribution of video _{signals, F L (X, Y,} T) is distributed in a narrow band centered around the flicker frequency f _F in the time frequency domain. Therefore, it is effective in reducing flicker to suppress a component centered on a specific time frequency corresponding to the flicker frequency f _F.

  Therefore, in the present invention, the video signal is converted into a two-dimensional or three-dimensional spatio-temporal frequency domain signal, the frequency component at a specific temporal frequency is reduced in the spatio-temporal frequency domain, Convert to Also, in reducing the frequency component at this specific time frequency, the frequency component of the original video signal is estimated by referring to the frequency component of the time frequency near the flicker frequency, and only the flicker component is reduced. The original video signal is retained.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a flicker reducing apparatus according to an embodiment of the present invention. In the example shown in FIG. 1, the flicker reduction apparatus 1 includes an input side frame memory (FM) 11, a frequency conversion unit 12, a flicker component estimation unit 13, a flicker removal unit 14, an inverse frequency conversion unit 15, And an output side frame memory (FM) 16.

Since the AC power supply frequency f _A and the moving image frame frequency f _V are known, the flicker frequency f _F is also known as shown in the equation (1). In the present embodiment, the AC power supply frequency f _A is 50 Hz, the frame frequency f _V is 120 Hz, and the flicker frequency f _F is 20 Hz.

  Each frame memory 11 sequentially accumulates video signals including flicker for each frame and outputs them to the frequency converter 12. For example, when processing is performed in units of 32 frames with the number of samples of Fourier transform in the time direction in the frequency converter 12 being 32, 32 frame memories 11 are stored and 32 frames are accumulated.

  The frequency conversion unit 12 performs frequency conversion processing (for example, FFT: Fast Fourier Transform processing) on the image input from the frame memory 11 and converts the image into a two-dimensional or three-dimensional spatio-temporal frequency domain signal including a temporal frequency. And output to the flicker component estimation unit 13 and the flicker removal unit 14. For example, when discrete Fourier transform is performed in units of 16 samples in the time direction, the time frequency resolution is 7.5 Hz (= 120/16), and when discrete Fourier transform is performed in units of 32 samples in the time direction, the time frequency resolution is 3.75 Hz (= 120/32). By increasing the number of discrete Fourier transform samples in the time direction, the time frequency resolution can be made finer, but this is a trade-off with the delay time. In addition to the discrete Fourier transform, a frequency analysis method such as a discrete cosine transform, a discrete sine transform, a discrete Walsh-Hadamard transform, or a discrete wavelet transform can be used as the frequency transform.

  FIG. 2 is a diagram for explaining processing when a moving image is converted into a three-dimensional space-time frequency domain signal of horizontal frequency, vertical frequency, and temporal frequency. In this case, the frequency conversion unit 12 inputs pixels of a two-dimensional array of columns and rows of each frame from the frame memory 11. Then, three-dimensional frequency conversion processing is performed in units of small blocks that are convenient for frequency conversion processing, and this processing is sequentially performed so as to cover the total number of pixels in the horizontal and vertical directions. FIG. 2 shows an example in which the number of Fourier transform samples in the time direction is 32, and the horizontal, vertical, and time direction three-dimensional frequency conversion processing blocks are 32 × 32 × 32 samples.

  FIG. 3 is a diagram for explaining processing when a moving image is converted into a signal in a two-dimensional spatio-temporal frequency domain having a horizontal frequency and a temporal frequency. In this case, the frequency conversion unit 12 inputs a one-dimensional array of pixels in the row direction of each frame from the frame memory 11. Then, in the horizontal (pixel) and time (frame) xt planes, two-dimensional frequency conversion processing is performed in units of small blocks convenient for horizontal frequency conversion processing, and this covers the total number of horizontal pixels. Process sequentially. FIG. 3 shows an example in which the number of discrete Fourier transform samples in the time direction is 32 and the two-dimensional frequency transform processing block in the horizontal and time directions is 32 × 32 samples.

  FIG. 4 is a diagram for explaining processing when a moving image is converted into a signal in a two-dimensional spatio-temporal frequency domain having a vertical frequency and a temporal frequency. In this case, the frequency conversion unit 12 inputs a one-dimensional array of pixels in the column direction of each frame from the frame memory 11. Then, on the yt plane of vertical (pixel) and time (frame), two-dimensional frequency conversion processing is performed in units of small blocks convenient for vertical frequency conversion processing, and this covers the total number of vertical pixels. Process sequentially. FIG. 4 shows an example in which the number of discrete Fourier transform samples in the time direction is 32 and the two-dimensional frequency transform processing block in the vertical and time directions is 32 × 32 samples.

The flicker component estimation unit 13 determines a frequency corresponding to a known flicker frequency f _F as a suppression frequency f _N among discrete time frequencies represented by integer multiples of the time frequency resolution in the frequency conversion unit 12. For example, when the flicker frequency f _F is 20 Hz, when the time frequency resolution is 7.5 Hz, 22.5 Hz and 15 Hz are the upper and lower frequencies closest to the flicker frequency f _F. When the time frequency resolution is 3.75 Hz, 22.5 Hz and 18.75 Hz are the upper and lower frequencies closest to the flicker frequency f _F. The frequency closest to the flicker frequency f _F or the closest upper and lower frequencies is determined as the suppression frequency f _N. (Hereinafter, the suppression frequency f _N is not necessarily one frequency.)

FIG. 5 is a diagram illustrating an example of frequency components with respect to time frequency in a video signal including flicker. The horizontal axis indicates the time frequency, and the vertical axis indicates the level of the frequency component. In FIG. 5, the time frequency resolution is 3.75 Hz, and the upper and lower frequencies closest to the flicker frequency f _F are the suppression frequencies f _N. Since the suppression frequency f _N in the vicinity of the flicker frequency f _F includes a flicker component, it has a larger frequency component than the original video signal without flicker indicated by a black circle.

Flicker component estimating unit 13, based on the frequency components of the time-frequency in the vicinity of the suppression frequency f _N, calculates a suppression amount of frequency components in the suppressed frequency f _N. For example, the average value of the frequency components F (f _N-1 ) and F (f _{N + 1} ) of the time frequency adjacent to and below the suppression frequency f _N or a weighted average value corresponding to the frequency difference with the flicker frequency is obtained. The estimated value of the component of the suppression frequency f _N included in the video signal when the subject is photographed when there is no flickering of the illumination, and the difference between the frequency component F (f _N ) of the suppression frequency f _N and the estimated value is suppressed. Amount. As a result, the influence on the same frequency component as the flicker frequency originally included in the video signal can be reduced.

Off Ricker removal unit 1 4, among the frequency components are time-frequency suppression frequency f _N, may be further suppressed only the frequency components of the spatial frequency of the predetermined range. Since flicker is considered to appear in a region where the spatial frequency is low, it is possible to obtain a video signal in which only the flicker component is reduced more effectively by suppressing the frequency component only in the region where the spatial frequency is low. Hereinafter, frequency component suppression processing in consideration of the spatial frequency will be described with reference to FIGS.

FIG. 6 is a diagram illustrating a three-dimensional space-time frequency region converted by the frequency conversion process of the frequency conversion unit 12. The X axis indicates the horizontal frequency, the Y axis indicates the vertical frequency, and the T axis indicates the time frequency. Here, the maximum value of the time frequency is f _V / 2 according to the sampling theorem.

FIG. 7 is a diagram illustrating an example of the suppression region of the frequency component when the frequency conversion unit 12 converts the moving image into a signal in the three-dimensional spatio-temporal frequency region. FIG. 7 shows a two-dimensional frequency plane XY in the horizontal and vertical directions in which the time frequency T is the suppression frequency f _N. In this two-dimensional frequency plane XY, flicker removal unit 1 4, the horizontal frequency 0 ≦ X ≦ X _LOW, and the vertical frequency to suppress only the frequency component of 0 ≦ Y ≦ Y _LOW. In the figure, the region where the frequency component is suppressed is indicated by hatching. However, the region where the frequency component is suppressed is not limited to a rectangle on the XY plane as in this example.

FIG. 8 is a diagram illustrating a frequency component suppression region when the frequency conversion unit 12 converts a moving image into a two-dimensional spatio-temporal frequency region signal of a horizontal frequency and a temporal frequency. FIG. 8 shows a two-dimensional frequency plane XT in the horizontal and time directions. In this two-dimensional frequency plane XT, flicker removal unit 1 4, time-frequency T is suppression frequency f _N, and a horizontal frequency suppresses only the frequency component of 0 ≦ X ≦ X _LOW. In the figure, a region where the frequency component is suppressed is indicated by a bold line.

FIG. 9 is a diagram illustrating a frequency component suppression region when the frequency conversion unit 12 converts a moving image into a two-dimensional spatio-temporal frequency region signal having a vertical frequency and a temporal frequency. FIG. 9 shows a two-dimensional frequency plane YT in the vertical and time directions. In the two-dimensional frequency plane YT, the flicker component estimation unit 13 suppresses only frequency components whose temporal frequency T is the suppression frequency f _N and whose vertical frequency is 0 ≦ Y ≦ Y _LOW . In the figure, a region where the frequency component is suppressed is indicated by a bold line. Note that X _LOW and Y _LOW described above are determined according to the resolution of the video system and shooting conditions.

The flicker removal unit 14 acquires the suppression frequency f _N and the suppression amount from the flicker component estimation unit 13. Then, the frequency component having the time frequency T being the suppression frequency f _N is suppressed according to the suppression amount with respect to the signal converted into the frequency domain signal by the frequency conversion unit 12, and is output to the inverse frequency conversion unit 15.

  The inverse frequency transform unit 15 performs an inverse frequency transform process that is the reverse of the process performed by the frequency transform unit 12, converts a spatio-temporal frequency domain signal into a real-time spatio-temporal signal, and outputs the signal to the frame memory 16. When the FFT process is performed by the frequency conversion unit 12, the inverse frequency conversion unit 15 performs an IFFT (Inverse Fast Fourier Transform) process. The result of inverse frequency conversion is an image in which flicker is suppressed.

  The flicker reducing apparatus 1 outputs a moving image by sequentially outputting images with the flicker suppressed from the frame memory 16. The flicker reduction apparatus 1 sequentially performs the above processing for each sample unit (32 frames in this embodiment) in the time direction.

As described above, the flicker reduction apparatus 1 according to the present invention converts an input moving image into a frequency domain signal including a time frequency by the frequency conversion unit 12, and supports a known flicker frequency by the flicker component estimation unit 13. to determine the time-frequency as suppression frequency f _N, the suppression amount of the frequency components of the suppression frequency f _N, determined from the frequency components of the time-frequency in the vicinity of the suppression frequency f _N, the suppression frequency by flicker removal unit 14 A frequency region in which the frequency component is suppressed according to the suppression amount is generated, and the frequency region generated by the flicker removing unit 14 is converted by the inverse frequency conversion unit 15 into a signal in the real space region. By performing such frequency analysis, the flicker reducing apparatus 1 can effectively reduce flicker that appears unevenly in the screen.

Further, since it is considered that flicker appears in a region where the spatial frequency is low, the flicker component estimation unit 13 suppresses only the frequency component whose spatial frequency is low among the frequency components whose temporal frequency is the suppression frequency f _N. A video signal with reduced flicker can be obtained even more effectively.

  Further, a computer can be suitably used to function as the above-described flicker reducing apparatus 1, and such a computer stores a program describing processing contents for realizing each function of the flicker reducing apparatus 1. This program can be realized by reading out and executing this program by the CPU of the computer. This program can be recorded on a computer-readable recording medium.

  Although the above embodiment has been described as a representative example, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiments can be combined into one, or one constituent block can be divided.

DESCRIPTION OF SYMBOLS 1 Flicker reduction apparatus 11,16 Frame memory 12 Frequency conversion part 13 Flicker component estimation part 14 Flicker removal part 15 Inverse frequency conversion part

Claims (5)

A frequency converter that converts a moving image into a spatio-temporal frequency domain signal including a temporal frequency and a spatial frequency ;
A time frequency corresponding to a known flicker frequency among the time frequencies is determined as a suppression frequency, and a suppression amount of a frequency component of the suppression frequency is determined as a time frequency in the vicinity of the suppression frequency in the signal in the space-time frequency domain. Flicker component estimator that is determined from the frequency components of
The signal of the space-time frequency domain, among the frequency components of the suppression frequency, and the flicker removing portion for generating a signal in the frequency domain a frequency component of the spatial frequency has been suppressed according to the suppression amount in a predetermined range,
An inverse frequency transform unit that transforms the frequency domain signal generated by the flicker removal unit into a real space domain signal;
A flicker reduction device comprising: The frequency conversion unit converts the moving image into a three-dimensional spatio-temporal frequency domain signal of a horizontal frequency, a vertical frequency, and a temporal frequency,
The flicker removal unit, among the frequency components of the previous SL suppression frequency, characterized by suppressing the frequency components in the horizontal and vertical frequency in a predetermined range, flicker reduction apparatus according to claim 1. The frequency conversion unit converts the moving image into a two-dimensional space-time frequency domain signal of a horizontal frequency and a temporal frequency,
The flicker removal unit, among the frequency components of the previous SL suppression frequency, characterized by suppressing the frequency component of the horizontal frequency of the predetermined range, flicker reduction apparatus according to claim 1. The frequency conversion unit converts the moving image into a two-dimensional space-time frequency domain signal of a vertical frequency and a time frequency,
The flicker removal unit, among the frequency components of the previous SL suppression frequency, characterized by suppressing the frequency components in the vertical frequency in a predetermined range, flicker reduction apparatus according to claim 1. The program for functioning a computer as a flicker reduction apparatus as described in any one of Claim 1 to 4 .

Images