WO2016129405A1

WO2016129405A1 - Image processing device, image processing method, and program

Info

Publication number: WO2016129405A1
Application number: PCT/JP2016/052581
Authority: WO
Inventors: 裕考篠崎
Original assignee: ソニー株式会社
Priority date: 2015-02-09
Filing date: 2016-01-29
Publication date: 2016-08-18
Also published as: JP2016146592A

Abstract

The present disclosure pertains to an image processing device, an image processing method, and a program with which it is possible to reduce the effect of a flicker during high-dynamic-range (HDR) image-capturing. A flicker detection unit calculates the amplitude and period of a flicker component from an image signal, among a short-time exposed image signal obtained from pixels exposed to light for a short time and a long-time exposed image signal obtained from pixels exposed to light for a long time, that includes the flicker component due to the flickering of a light source. A composition processing unit changes a blend ratio when combining the short-time exposed image signal and the long-time exposed image signal according to the amplitude of the flicker component, and combines the short-time exposed image signal and the long-time exposed image signal in accordance with the blend ratio. Furthermore, an image-capture control unit performs a control for changing the exposure time of the image signal that includes the flicker component according to the period of the flicker component. The present technology can be applied to, e.g., an image-capturing device with which it is possible to capture a HDR image.

Description

Image processing apparatus, image processing method, and program

The present disclosure relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program that can reduce the influence of flicker during HDR imaging.

Generally, when an image is taken in an environment where the light source flickers, flicker (flashing light) may occur in the image.

Conventionally, a technique for correcting flicker has been developed. For example, as one of the most widely used methods, there is a method of detecting the occurrence of flicker of a light source and changing the exposure time of the image sensor in accordance with the flicker frequency (see Patent Document 1). ). Specifically, when the half cycle time of the flicker of the light source is T, the exposure time of the image sensor is set to a constant multiple of the half cycle time T (exposure time = a × T: a is an arbitrary natural number). Thereby, the incident amount of the light source in each frame can always be kept constant, and the influence of the light source flicker can be eliminated.

Further, in recent years, as a technique for capturing an HDR (High Dynamic Range) image, an imaging method that combines image signals captured at a plurality of types of exposure times has been used.

By the way, when picking up an HDR image, the flicker correction as described above may not function well. In other words, when capturing an HDR image, it is necessary to set a sufficiently short exposure time, particularly in a short-exposure pixel, and to capture a bright region that saturates at an appropriate exposure time. For this reason, it is often impossible to set the exposure time of the short exposure pixel to be proportional to the half cycle of the flicker light source (exposure time = a × T). In such a case, the flicker of the light source may be corrected. could not.

On the other hand, a technique has been proposed in which a reference image that does not include a flicker component is generated, and an image that includes flicker is corrected using a difference from the reference image (see Patent Document 2). However, with such a method, flicker may be erroneously detected due to a moving subject or the like, and side effects such as overcorrection may occur.

Japanese Patent Laid-Open No. 2002-84466 JP 2012-178865 A

As described above, conventionally, it is difficult to appropriately correct flicker during HDR imaging, and it is required to reduce the influence of flicker during HDR imaging.

The present disclosure has been made in view of such a situation, and is intended to reduce the influence of flicker during HDR imaging.

An image processing apparatus according to an aspect of the present disclosure obtains at least an amplitude of the flicker component from the image signal including a flicker component due to flickering of a light source among a plurality of image signals captured at a plurality of types of exposure times. In accordance with the flicker detection unit and the amplitude of the flicker component obtained by the flicker detection unit, the blend rate when combining the plurality of image signals is changed, and the plurality of image signals are combined according to the blend rate. A synthesis processing unit.

According to an image processing method or program of one aspect of the present disclosure, an amplitude of the flicker component is calculated from the image signal including a flicker component caused by flickering of a light source among a plurality of image signals captured at a plurality of types of exposure times. The method includes a step of changing at least a blend rate when combining the plurality of image signals according to the obtained amplitude of the flicker component, and combining the plurality of image signals according to the blend rate.

In one aspect of the present disclosure, at least an amplitude of a flicker component is obtained from an image signal including a flicker component due to flickering of a light source among a plurality of image signals captured at a plurality of types of exposure times. Then, according to the obtained amplitude of the flicker component, the blend ratio when the plurality of image signals are combined is changed, and the plurality of image signals are combined according to the blend ratio.

According to one aspect of the present disclosure, the influence of flicker during HDR imaging can be reduced.

It is a block diagram showing an example of composition of a 1st embodiment of an image processing device to which this art is applied. It is a flowchart explaining an image process. It is a block diagram which shows the structural example of a flicker detection part. It is a figure explaining the method to detect the flicker by the inter-frame difference flicker detection part. It is a figure explaining the method to detect the flicker by the inter-frame difference flicker detection part. It is a figure explaining the method to detect the flicker by the difference flicker detection part between lines. It is a figure explaining the method to detect the flicker by the difference flicker detection part between lines. It is a figure explaining the HDR synthetic | combination process in an HDR synthetic | combination process part. It is a figure which shows control of general exposure time and analog gain. It is a figure which shows the histogram of a captured image. It is a figure which shows the histogram of the short accumulation signal in a HDR image, and a long accumulation signal. It is a figure which shows the relationship between the brightness of a to-be-photographed object, and exposure time. It is a figure explaining the minimum flickerless exposure time. It is a block diagram which shows the structural example of 2nd Embodiment of the image processing apparatus to which this technique is applied. It is a block diagram which shows the structural example of 3rd Embodiment of the image processing apparatus to which this technique is applied. It is a block diagram which shows the modification of a flicker detection part. It is a block diagram which shows the structural example of the imaging device mounted in an electronic device.

Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

FIG. 1 is a block diagram illustrating a configuration example of a first embodiment of an image processing apparatus to which the present technology is applied.

The image processing apparatus 11 shown in FIG. 1 performs image processing on an image picked up by the image pickup device 12 and outputs the processed image to a subsequent circuit (not shown). For example, the image sensor 12 performs a short-time exposure signal (short-time exposure image signal) that is an image signal obtained from a short-time exposure accumulation pixel that accumulates charges generated by performing a short-time exposure, and a long-time exposure. A long accumulation signal (long exposure image signal) which is an image signal obtained from the long exposure accumulation pixel for accumulating the generated charge is output. Then, the image processing device 11 performs image processing for generating an HDR image having a wide dynamic range by synthesizing the short accumulation signal and the long accumulation signal at a blend ratio (use ratio) according to brightness.

As shown in FIG. 1, the image processing apparatus 11 includes a flicker detection unit 21, an HDR synthesis processing unit 22, an imaging control unit 23, and a camera signal processing unit 24.

The flicker detection unit 21 acquires the short accumulation signal and the long accumulation signal output from the image pickup device 12 and observes the short accumulation signal and the long accumulation signal, whereby the flicker component of the light source is added to the short accumulation signal or the long accumulation signal. Detect if is included. That is, the flicker detection unit 21 performs a flicker detection process for detecting whether or not flicker occurs in any of an image composed of short accumulation signals and an image composed of long accumulation signals. Then, when it is detected that flicker has occurred, the flicker detection unit 21 obtains the amplitude and period of the flicker component included in the short accumulation signal or the long accumulation signal, and determines the amplitude of the flicker component as HDR. This is supplied to the composition processing unit 22, and the cycle of the flicker component is supplied to the imaging control unit 23. The processing performed by the flicker detection unit 21 will be described later with reference to FIGS.

As described later with reference to FIG. 8, the HDR synthesizing unit 22 performs a short accumulation signal and a long accumulation signal when performing the HDR synthesizing process according to the amplitude of the flicker component supplied from the flicker detection unit 21. Adaptively change the blend rate. Then, the HDR synthesis processing unit 22 generates an HDR image by synthesizing the short accumulation signal and the long accumulation signal at the adaptively changed blend rate, and supplies the HDR image to the imaging control unit 23 and the camera signal processing unit 24.

Based on the brightness of the HDR image supplied from the HDR synthesis processing unit 22, the imaging control unit 23 captures the exposure time and analog gain of the imaging device 12 so that an image with appropriate brightness is captured by the imaging device 12. To control. At this time, the imaging control unit 23 changes the exposure time of the image signal (one of the short accumulation signal and the long accumulation signal) containing the flicker component according to the cycle of the flicker component supplied from the flicker detection unit 21. Control. That is, as will be described later with reference to FIG. 12, the imaging control unit 23 sets the imaging element 12 so as to be a constant multiple of the minimum flickerless exposure time, which is the minimum exposure time that can suppress the influence of flicker. To control the exposure time.

The camera signal processing unit 24 performs general image processing on the HDR image supplied from the HDR synthesis processing unit 22, performs gamma correction, white balance adjustment, and the like, and supplies it to a subsequent circuit (not shown).

In the image processing apparatus 11 configured as described above, when the flicker detection unit 21 detects that the flicker is generated, the HDR composition processing unit 22 adaptively changes the blend rate and the imaging control unit. In 23, the exposure time of the image sensor 12 is controlled. Thereby, the flicker component contained in the short accumulation signal and the long accumulation signal can be reduced, and the influence of flicker in the HDR image can be suppressed.

Next, FIG. 2 is a flowchart for explaining image processing performed in the image processing apparatus 11.

For example, when the image pickup device 12 starts imaging and outputs a short accumulation signal and a long accumulation signal, the processing is started. In step S11, the flicker detection unit 21 and the HDR synthesis processing unit 22 perform one frame (time t ) Of short accumulation signal and long accumulation signal.

In step S12, the flicker detection unit 21 detects whether the flicker component of the light source is included in the short accumulation signal or the long accumulation signal based on the short accumulation signal and the long accumulation signal for one frame acquired in step S11. Flicker detection processing is performed.

In step S13, the HDR synthesizing unit 22 synthesizes the short accumulation signal and the long accumulation signal for one frame acquired in step S11 with a predetermined blend rate (for example, a setting value according to brightness). An HDR image is generated. Then, the HDR synthesis processing unit 22 supplies the generated HDR image to the imaging control unit 23 and the camera signal processing unit 24. For this frame, the HDR synthesizing unit 22 outputs the HDR image as it is without performing the process for the flicker detected by the flicker detection unit 21 in the immediately preceding step S12.

In step S14, the flicker detection unit 21 determines whether or not flicker has occurred as a result of performing the flicker detection process in step S12.

If it is determined in step S14 that flicker has not occurred, the process returns to step S11, and the same process is repeated thereafter. On the other hand, if it is determined in step S14 that flicker has occurred, the process proceeds to step S15.

In step S15, the flicker detection unit 21 obtains the cycle of the flicker component detected in step S12 and supplies it to the imaging control unit 23. The imaging control unit 23 adjusts the exposure time of the image sensor 12 according to the flicker cycle. To do. For example, when a flicker component is included in the long accumulation signal, the image sensor 12 performs adjustment so that the exposure time of the long-time exposure accumulation pixel becomes a constant multiple of the minimum flickerless exposure time (see FIG. 12).

In step S16, the flicker detection unit 21 and the HDR synthesis processing unit 22 respectively acquire a short accumulation signal and a long accumulation signal for one frame (time t + 1) next to the frame obtained in step S11.

In step S17, the flicker detection unit 21 detects whether the flicker component of the light source is included in the short accumulation signal or the long accumulation signal based on the short accumulation signal and the long accumulation signal for one frame acquired in step S16. Flicker detection processing is performed.

In step S18, the flicker detection unit 21 determines whether or not flicker has occurred as a result of performing the flicker detection process in step S17.

If it is determined in step S18 that no flicker has occurred, the process returns to step S11, and the same process is repeated thereafter. On the other hand, if it is determined in step S18 that flicker has occurred, the process proceeds to step S19.

In step S19, the flicker detection unit 21 obtains the amplitude of the flicker component detected in step S17 and supplies it to the HDR synthesis processing unit 22. The HDR synthesis processing unit 22 uses the blend rate based on the amplitude of the flicker component. Is adaptively changed. For example, if adjustment is performed in step S15 such that the exposure time of the long-time exposure accumulation pixel is a constant multiple of the minimum flickerless exposure time, the flicker detection process in step S17 includes the short accumulation signal. The flicker component being detected is detected. Therefore, the flicker detection unit 21 supplies the amplitude of the flicker component of the short accumulation signal to the HDR synthesis processing unit 22, and the HDR synthesis processing unit 22 reduces the usage ratio of the short accumulation signal (see FIG. 8). Change the blend rate.

In step S20, the HDR synthesis processing unit 22 generates an HDR image by synthesizing the short accumulation signal and the long accumulation signal for one frame acquired in step S16 with the blend rate changed in step S19, and performs imaging control. To the unit 23 and the camera signal processing unit 24. And after the process of step S20, a process returns to step S11 and the same process is repeated hereafter.

Next, FIG. 3 is a block diagram showing a configuration example of the flicker detection unit 21 of FIG.

As shown in FIG. 3, the flicker detection unit 21 includes inter-frame difference

flicker detection units

31 and 32 and an inter-line difference flicker detection unit 33.

The inter-frame difference flicker detection unit 31 obtains a difference between frames of an image constructed from the short accumulation signal supplied from the image sensor 12, thereby using a time change amount between frames and flickering the short accumulation signal. Detect if a component is included. As a result of the detection, the inter-frame difference flicker detection unit 31 obtains the amplitude and period of the flicker component of the short accumulation signal when it is detected that the short accumulation signal contains the flicker component. Then, the inter-frame difference flicker detection unit 31 supplies the flicker component amplitude of the short accumulation signal to the HDR synthesis processing unit 22 and supplies the cycle of the flicker component of the short accumulation signal to the imaging control unit 23.

The inter-frame difference flicker detection unit 32 obtains a difference between frames of an image constructed from the long accumulation signal supplied from the image sensor 12, and uses the time change amount between frames to flicker the long accumulation signal. Detect if a component is included. As a result of the detection, the inter-frame difference flicker detection unit 32 obtains the amplitude and period of the flicker component of the long accumulation signal when it is detected that the flicker component is included in the long accumulation signal. Then, the inter-frame difference flicker detection unit 32 supplies the amplitude of the flicker component of the long accumulation signal to the HDR synthesis processing unit 22 and supplies the cycle of the flicker component of the long accumulation signal to the imaging control unit 23.

The inter-line difference flicker detection unit 33 uses the amount of change for each row of the image in each of the images constructed from the short accumulation signal and the long accumulation signal supplied from the image sensor 12 to store the short accumulation signal and the long accumulation signal. It is detected whether or not a flicker component is included in each signal. As a result of the detection, for example, when it is detected that the short accumulation signal contains a flicker component, the inter-line difference flicker detection unit 33 obtains the amplitude and period of the flicker component of the short accumulation signal, The flicker component amplitude of the accumulated signal is supplied to the HDR synthesizing unit 22, and the cycle of the flicker component of the short accumulation signal is supplied to the imaging control unit 23. Similarly, the inter-line difference flicker detection unit 33 obtains the amplitude and period of the flicker component of the long accumulation signal, for example, when it is detected that the long accumulation signal contains the flicker component, and flickers the long accumulation signal. The amplitude of the component is supplied to the HDR synthesis processing unit 22, and the cycle of the flicker component of the long accumulation signal is supplied to the imaging control unit 23.

Here, with reference to FIG. 4 and FIG. 5, a method in which the inter-frame difference

flicker detection units

31 and 32 detect flicker using the amount of time change between frames will be described.

For example, as shown in the upper side of FIG. 4, the inter-frame difference flicker detection unit 31 sequentially images (

frames

1, 2,..., Based on short accumulation signals supplied from the image sensor 12 as time passes. N). Similarly, the inter-frame difference flicker detection unit 32 sequentially constructs images (

frames

1, 2,..., N) based on the long accumulation signal supplied from the image sensor 12 as time elapses.

When flicker occurs in the image, when the brightness of the image for each frame is observed, the observed signal has a waveform as shown in the lower side of FIG. That is, if flicker occurs due to blinking of the light source, the observed signal in the image in which the subject is imaged is obtained by adding the flicker component to the subject's own brightness. Therefore, in order to obtain the flicker component, first, it is necessary to calculate the brightness unique to the subject.

Therefore, the inter-frame difference

flicker detection units

31 and 32 add up the brightness of a plurality of frames and calculate the average value Ave (= (frame 1 + frame 2+... + Frame N) / N). Calculate as unique brightness. Here, the influence of flicker can be eliminated by setting the number of frames used for calculating the average value Ave to a time proportional to the light source flicker cycle. With this method, if the subject is not moving, the subject's own brightness excluding the influence of flicker can be calculated.

Next, the inter-frame difference

flicker detection units

31 and 32 calculate a flicker component in each frame from the brightness unique to the subject. For example, the inter-frame difference

flicker detection units

31 and 32 can obtain a flicker component (= frame / average value Ave) by dividing the signal of each frame by the brightness unique to the subject. That is, a portion having a large ratio to the average value Ave is a flicker component.

As described above, the inter-frame difference

flicker detection units

31 and 32 calculate the flicker component, and if the flicker component is equal to or greater than a certain threshold, it is determined that the short accumulation signal and the long accumulation signal include the flicker component. can do.

For example, as shown in the upper side of FIG. 5, the inter-frame difference

flicker detection units

31 and 32 divide the observed signal (signal value) by the subject's own brightness to obtain a flicker component ratio (signal ratio). ) Is calculated.

The inter-frame difference

flicker detection units

31 and 32 approximate the calculated flicker ratio with an arbitrary sine wave, as shown on the lower side of FIG. Accordingly, the inter-frame difference

flicker detection units

31 and 32 determine that flicker has occurred in the image if the amplitude of the sine wave (approximate waveform) obtained by approximation is equal to or greater than a predetermined threshold. If it cannot be approximated to any sine wave, it is assumed that the brightness varies due to factors other than flicker (for example, the subject has moved). In this case, it is detected as flicker. Not.

By the way, in the method of detecting flicker using the amount of time change between frames, when the average value Ave of a plurality of frames is calculated, if the subject or the imaging apparatus is moving, the brightness is increased by the movement. It is difficult to accurately detect flicker. Thus, by obtaining the flicker component from only one frame signal, it is possible to eliminate the influence of such a change in brightness.

Next, with reference to FIG. 6 and FIG. 7, a method in which the inter-line difference flicker detection unit 33 detects flicker using the amount of change for each line of the image will be described.

As shown on the left side of FIG. 6, when the image sensor 12 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a rolling shutter for sequentially reading out image signals is performed for each row of the image. Therefore, there is a time difference in the exposure timing between the uppermost row and the lowermost row. Therefore, as shown on the right side of FIG. 6, the CMOS image sensor has a characteristic that when the light source includes flicker, a phenomenon in which the brightness varies depending on the row occurs.

Therefore, the inter-line difference flicker detection unit 33 calculates the average value of each line in the image using such characteristics, and detects the flicker using the change amount of the average value. In addition to using the average value of the image signal for each row, the integrated value of the image signal for each row may be used.

For example, when the average value of the image signal is calculated for each row and plotted in the row direction, the average value of the image signal includes the original brightness (subject brightness) and flicker component as shown in FIG. It is included. Here, after controlling the exposure time so that flicker does not occur in either the short accumulation signal or the long accumulation signal (see FIG. 12), the original brightness of the subject is determined from the image signal whose exposure time is controlled. Can be sought.

FIG. 7 shows an example in which the flicker component is not included in the image configured using the long accumulation signal. In this case, the inter-line difference flicker detection unit 33 can obtain the flicker component by calculating the signal ratio between the short accumulation signal and the long accumulation signal.

For example, the inter-line difference flicker detection unit 33 calculates the flicker component based on the signal ratio of the short accumulation signal and the long accumulation signal in a state where the flicker component is not included in the long accumulation signal, and the flicker component is constant. If it is equal to or greater than the threshold, it is detected that the flicker component is included in the short accumulation signal. When the flicker is detected, the inter-line difference flicker detection unit 33 approximates the signal ratio of the flicker component with a sine wave, similarly to the inter-frame difference

flicker detection units

31 and 32, and obtains the flicker component amplitude and period. Can be supplied to the HDR synthesis processing unit 22. That is, with the horizontal axis in FIG. 5 described above as the row coordinate, the amplitude and period of the flicker component included in one frame of the short accumulation signal can be obtained.

Similarly, the inter-line difference flicker detection unit 33 is configured so that the flicker included in one frame of the long accumulation signal is based on the signal ratio of the short accumulation signal and the long accumulation signal in a state where the flicker component is not included in the short accumulation signal. The amplitude and period of the component can be determined. In other words, in order for the inter-line difference flicker detection unit 33 to detect flicker, flicker occurs in either one of the image configured using the short accumulation signal and the image configured using the long accumulation signal. It is assumed that it is not.

Next, the HDR synthesizing process in the HDR synthesizing unit 22 will be described with reference to FIG.

First, in a normal HDR synthesis process, an HDR image is configured using the short accumulation signal S, the long accumulation signal L, the black level B _opb , the exposure ratio gain, and the blend ratio α as in the following equation (1). The signal HDR to be calculated is calculated.

As shown in FIG. 8A, the blend ratio α between the short accumulation signal and the long accumulation signal is set so as to be switched according to the brightness of the subject. The signal HDR is calculated by calculating Expression (1). In general, when the subject is dark, the blend ratio α uses a long accumulation signal that can be imaged up to a dark part by long exposure. The signal is set to be used. That is, when the brightness of the subject is equal to or less than the threshold value TH0, the blend ratio α is set to 0. When the brightness of the subject is equal to or greater than the threshold value TH1, the blend ratio α is set to 1, and the threshold value TH0 to the threshold value TH1. Then, the blend ratio α is changed linearly.

Then, the HDR synthesizing unit 22 switches the blend ratio α between the short accumulation signal and the long accumulation signal according to the flicker detection result by the flicker detection unit 21, that is, according to the magnitude of the amplitude A of the flicker component. Change the setting.

For example, when the flicker component is included in the short accumulation signal, the HDR synthesis processing unit 22 updates the threshold value TH0 and the threshold value TH1 based on the amplitude A of the flicker component according to the following equation (2). In equation (2), c0 and c1 are arbitrary coefficients proportional to the brightness of the subject.

Accordingly, as shown in FIG. 8B, when the flicker component is included in the short accumulation signal, the threshold value TH0 and the threshold value TH1 are updated to a large value, and the long accumulation signal is often used. That is, the use of the short accumulation signal including the flicker component is reduced, and the HDR synthesis processing unit 22 can generate an HDR image in which flicker is suppressed.

On the other hand, when the flicker component is included in the long accumulation signal, the HDR synthesis processing unit 22 updates the threshold value TH0 and the threshold value TH1 based on the amplitude A of the flicker component according to the following equation (3). In equation (3), d0 and d1 are arbitrary coefficients proportional to the brightness of the subject.

Therefore, as shown in FIG. 8C, when the flicker component is included in the long accumulation signal, the threshold value TH0 and the threshold value TH1 are updated to small values, and the short accumulation signal is often used. That is, the use of the long accumulation signal including the flicker component is reduced, and the HDR synthesis processing unit 22 can generate an HDR image in which flicker is suppressed.

Next, the control of the exposure time and analog gain of the image pickup device 12 by the image pickup control unit 23 will be described with reference to FIGS.

FIG. 9 shows general exposure time and analog gain control. In FIG. 9, the horizontal axis indicates the brightness of the subject, the right vertical axis indicates the exposure time, and the left vertical axis indicates the analog gain.

As shown in FIG. 9, the exposure time and analog gain are controlled according to the brightness of the subject. That is, when the subject is very bright, the exposure time is set to a minimum unit (Min speed), and the analog gain is also set to a minimum value (Min Gain). Then, as the subject becomes darker, the exposure time is first controlled to gradually increase, and after the exposure time reaches the longest (1/60), the analog gain is controlled to increase.

The brightness of the subject is determined by the histogram of the captured image as shown in FIG. In FIG. 10, the horizontal axis indicates the brightness, and the vertical axis indicates the histogram value.

For example, the imaging control unit 23 calculates the exposure level by summing up the product of the histogram value and the signal value, and determines that the exposure is insufficient when the value is darker than the preset brightness of the target. Control to increase the exposure time or analog gain.

In addition, AE (Automatic Exposure) transition in the HDR image is performed in the same manner, and the imaging control unit 23 first selects an image configured using the short accumulation signal and an image configured using the long accumulation signal. Calculate the histogram.

For example, as shown in FIG. 11, for an image configured using a long accumulation signal, the image is clipped to 1023 and acquired for an image configured using a short accumulation signal. Calculate by dividing. Thereafter, the imaging control unit 23 adjusts the exposure level of each image. At this time, the exposure level target can be set to a different value between the short accumulation signal and the long accumulation signal.

Here, when flicker is detected by the flicker detection unit 21, the imaging control unit 23 controls the exposure time based on the cycle of the flicker component supplied from the flicker detection unit 21, as shown in FIG. Do.

FIG. 12 shows the relationship between the brightness of the subject and the exposure time. The thick line represents the exposure time control when flicker correction is not performed, and the thin line is the exposure time when flicker correction is performed. Represents the control.

As shown in FIG. 12, when flicker correction is not performed, the exposure time is controlled linearly, whereas when flicker correction is performed, a constant multiple (2 times, 4 times) of the minimum flickerless exposure time is used. (Exposure times, 6 times, etc.).

Here, as shown in FIG. 13, the minimum flickerless exposure time is set to a half cycle T of the brightness of the light source (flicker component). In other words, if the exposure time is a half cycle T of the brightness cycle of the light source, it is possible to avoid being affected by flicker regardless of the timing of imaging. Specifically, when the brightness cycle of the light source is 60 Hz, the half cycle T is set to 8.333 milliseconds.

As described above, the imaging control unit 23 controls at least one of the short accumulation signal and the long accumulation signal in which the flicker component is detected to be a constant multiple of the minimum flickerless exposure time. The occurrence of flicker is suppressed in the image. Thereby, as described with reference to FIG. 7 described above, the original brightness of the subject can be obtained based on an image in which no flicker occurs.

If the exposure time is controlled so as to be a constant multiple of the minimum flickerless exposure time (2 times, 4 times, 6 times, etc.), an arbitrary exposure ratio cannot be set, and the exposure ratio is It is limited to an integral multiple (for example, 1 ×, 2 ×, 3 ×, 8 ×, etc.). Therefore, the imaging control unit 23 increases the analog gain of the image signal for which the exposure time is controlled in accordance with the exposure ratio that is insufficient by controlling the exposure time, so that an arbitrary exposure ratio is obtained. Can be adjusted.

As described above, the image processing apparatus 11 can generate flicker-free HDR images by suppressing the occurrence of flicker when capturing HDR images. Further, since the image processing apparatus 11 only controls the exposure time of the image sensor 12 and the blend ratio α when generating the HDR image, for example, the image processing is performed on the image itself based on the detected flicker. It is possible to avoid the occurrence of side effects such as overcorrection compared to the correction method that is performed.

Next, FIG. 14 is a block diagram illustrating a configuration example of the second embodiment of the image processing apparatus to which the present technology is applied.

In the image processing apparatus 11A shown in FIG. 14, the same reference numerals are given to components common to the image processing apparatus 11 in FIG. 1, and detailed description thereof is omitted. That is, the image processing apparatus 11A has the same configuration as the image processing apparatus 11 of FIG. 1 in that it includes a flicker detection unit 21, an HDR synthesis processing unit 22, an imaging control unit 23, and a camera signal processing unit 24.

The image processing apparatus 11A is different from the image processing apparatus 11 in FIG. 1 in that it includes a flicker correction unit 25.

The flicker correction unit 25 is provided in the previous stage of the HDR synthesizing unit 22, and the short accumulation signal and the long accumulation signal are reduced so that the flicker component included in the short accumulation signal and the long accumulation signal output from the image sensor 12 is reduced. Correct. For example, when the flicker detection unit 21 detects that a short accumulation signal or a long accumulation signal contains a flicker component, the flicker component is approximated by a sine wave as described above, and the cycle and amplitude of the sine wave are calculated. The flicker correction unit 25 is supplied. As a result, the flicker correction unit 25 reduces the flicker component by correcting the signal value of the short accumulation signal or the long accumulation signal containing the flicker component with the sine wave supplied from the flicker detection unit 21. Then, the corrected short accumulation signal or long accumulation signal is supplied to the HDR synthesis processing unit 22.

Also in the image processing apparatus 11A configured as described above, an HDR image in which the occurrence of flicker is suppressed can be generated in the same manner as the image processing apparatus 11 in FIG.

FIG. 15 is a block diagram illustrating a configuration example of the third embodiment of the image processing apparatus to which the present technology is applied.

In the image processing apparatus 11B shown in FIG. 15, the same reference numerals are given to the same components as those in the image processing apparatus 11 in FIG. 1, and detailed description thereof will be omitted. That is, the image processing apparatus 11B has the same configuration as that of the image processing apparatus 11 of FIG. 1 in that it includes an imaging control unit 23 and a camera signal processing unit 24.

In the image processing apparatus 11B, unlike the image processing apparatus 11 of FIG. 1, the flicker detection unit 21B and the HDR synthesis processing unit 22B can perform processing on signals exposed with four different exposure times. It is configured.

That is, in the image processing apparatus 11B, the short accumulation signals 1 to 3 and the long accumulation signal exposed at four different exposure times are output from the image sensor 12, and are supplied to the flicker detection unit 21B and the HDR synthesis processing unit 22B, respectively. The

The flicker detection unit 21B detects whether or not a flicker component is included from each of the short accumulation signals 1 to 3 and the long accumulation signal. The HDR synthesis processing unit 22B can generate an HDR image with a wider dynamic range using the short accumulation signals 1 to 3 and the long accumulation signal.

Therefore, the image processing apparatus 11B can suppress flicker occurring in the HDR image generated from the short accumulation signals 1 to 3 and the long accumulation signal exposed at four different exposure times.

In addition, the present technology performs processing on image signals exposed at two different types of exposure times as in the image processing apparatus 11 of FIG. 1 and four types of different exposure times as in the image processing apparatus 11B of FIG. In addition to processing on the exposed image signal, the present invention can be applied to processing on an image signal exposed with a plurality of different types of exposure times.

Next, FIG. 16 is a block diagram showing a modification of the flicker detection unit 21.

As shown in FIG. 16, the flicker detection unit 21C includes inter-frame difference

flicker detection units

31 and 32. That is, the flicker detection unit 21 in FIG. 3 includes the inter-line difference flicker detection unit 33, whereas the flicker detection unit 21C does not include the inter-line difference flicker detection unit 33.

Thus, even if the flicker detection unit 21C includes only the inter-frame difference

flicker detection units

31 and 32, as described above, it is possible to detect flicker using the amount of time change between frames. That is, the inter-line difference flicker detection unit 33 is not an essential component for detecting flicker. Further, the flicker may be detected only by the inter-line difference flicker detection unit 33. That is, as long as the flicker component contained in the short accumulation signal and the long accumulation signal can be detected, the configuration is not limited to the configuration of the flicker detection unit 21 or the flicker detection unit 21C.

The image processing apparatus 11 of each embodiment as described above is, for example, an imaging system such as a digital still camera or a digital video camera, a mobile phone having an imaging function, or another device having an imaging function. It can be applied to various electronic devices.

FIG. 17 is a block diagram illustrating a configuration example of an imaging device mounted on an electronic device.

As shown in FIG. 17, the imaging apparatus 101 includes an optical system 102, an imaging element 103, a signal processing circuit 104, a monitor 105, and a memory 106, and can capture still images and moving images.

The optical system 102 includes one or more lenses, guides image light (incident light) from a subject to the image sensor 103, and forms an image on a light receiving surface (sensor unit) of the image sensor 103.

The image sensor 103 corresponds to the image sensor 12 described above. In the image sensor 103, electrons are accumulated for a certain period according to an image formed on the light receiving surface via the optical system 102. Then, a signal corresponding to the electrons accumulated in the image sensor 103 is supplied to the signal processing circuit 104.

The signal processing circuit 104 can apply the above-described image processing apparatus 11 as part of its function, and performs various signal processing on the image signal output from the image sensor 103. An image (image data) obtained by performing signal processing by the signal processing circuit 104 is supplied to the monitor 105 and displayed, or supplied to the memory 106 and stored (recorded).

In the imaging apparatus 101 configured as described above, the occurrence of flicker at the time of capturing an HDR image is suppressed by applying the image processing apparatus 11 according to each of the above-described embodiments. For example, a higher-quality HDR image is generated. An image can be taken.

In addition, this technique can also take the following structures.
(1)
A flicker detection unit that obtains at least the amplitude of the flicker component from the image signal including flicker components caused by flickering of a light source among a plurality of image signals captured at a plurality of types of exposure times;
A blending processing unit configured to change a blend rate when the plurality of image signals are combined according to the amplitude of the flicker component obtained by the flicker detection unit, and to combine the plurality of image signals according to the blend rate. An image processing apparatus.
(2)
The flicker detection unit determines whether or not the flicker component is included in each of the plurality of image signals based on a temporal change amount between the frames for each image composed of the plurality of image signals. The image processing apparatus according to (1), wherein the image processing apparatus detects and obtains at least an amplitude of the flicker component from the image signal including the flicker component.
(3)
Whether the flicker detection unit includes the flicker component in each of the plurality of image signals based on the amount of change for each row of each image in each image of one frame composed of each of the plurality of image signals. The image processing apparatus according to (1) or (2), wherein the image processing device detects whether or not and obtains at least an amplitude of the flicker component from the image signal including the flicker component.
(4)
The flicker detection unit further obtains a cycle of the flicker component,
From the above (1) to (3), further comprising: an imaging control unit that performs control to change an exposure time of the image signal including the flicker component according to the cycle of the flicker component obtained by the flicker detection unit An image processing apparatus according to any one of the above.
(5)
The image processing apparatus according to (4), wherein the imaging control unit sets an exposure time of the image signal including the flicker component to a constant multiple of a half cycle of the flicker component.
(6)
The imaging control unit performs control to increase an analog gain of the image signal according to an insufficient exposure ratio by setting the exposure time of the image signal including the flicker component (4) Or the image processing apparatus as described in (5).
(7)
The flicker detection unit and the composition processing unit are supplied with a short exposure image signal obtained from a pixel that performs short exposure and a long exposure image signal obtained from a pixel that performs long exposure,
The imaging control unit determines an exposure time of the image signal in which the flicker component is detected by the flicker detection unit of the short-time exposure image signal and the long-time exposure image signal as a half cycle of the flicker component period. The image processing device according to any one of (4) to (6).
(8)
The flicker detection unit obtains the original brightness of the subject from the image signal whose exposure time is controlled after the exposure time is controlled by the imaging control unit. The image processing apparatus described.
(9)
When detecting the flicker component from the image signal whose exposure time is not controlled by the imaging control unit, the flicker detection unit obtains the amplitude of the flicker component and supplies the amplitude of the flicker component to the synthesis processing unit. The image processing apparatus according to any one of (4) to (8).
(10)
Among the plurality of image signals picked up at a plurality of types of exposure times, obtain at least the amplitude of the flicker component from the image signal containing the flicker component due to flickering of the light source,
An image processing method comprising the steps of: changing a blend ratio when combining a plurality of the image signals according to the obtained amplitude of the flicker component; and combining the plurality of image signals according to the blend ratio.
(11)
Among the plurality of image signals picked up at a plurality of types of exposure times, obtain at least the amplitude of the flicker component from the image signal containing the flicker component due to flickering of the light source,
According to the obtained amplitude of the flicker component, the blend ratio when combining the plurality of image signals is changed, and the computer is caused to execute image processing including a step of combining the plurality of image signals according to the blend ratio. program.

Note that the present embodiment is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present disclosure.

11 image processing device, 12 image sensor, 21 flicker detection unit, 22 HDR synthesis processing unit, 23 image capture control unit, 24 camera signal processing unit, 25 flicker correction unit, 31 and 32 inter-frame difference flicker detection unit, 33 inter-line difference Flicker detector

Claims

A flicker detection unit that obtains at least the amplitude of the flicker component from the image signal including flicker components caused by flickering of a light source among a plurality of image signals captured at a plurality of types of exposure times;
A blending processing unit configured to change a blend rate when the plurality of image signals are combined according to the amplitude of the flicker component obtained by the flicker detection unit, and to combine the plurality of image signals according to the blend rate. An image processing apparatus.
The flicker detection unit determines whether or not the flicker component is included in each of the plurality of image signals based on a temporal change amount between the frames for each image composed of the plurality of image signals. The image processing apparatus according to claim 1, wherein the image processing apparatus detects and obtains at least an amplitude of the flicker component from the image signal including the flicker component.
Whether the flicker detection unit includes the flicker component in each of the plurality of image signals based on the amount of change for each row of each image in each image of one frame composed of each of the plurality of image signals. The image processing apparatus according to claim 1, wherein at least an amplitude of the flicker component is obtained from the image signal including the flicker component.
The flicker detection unit further obtains a cycle of the flicker component,
The image processing apparatus according to claim 1, further comprising: an imaging control unit that performs control to change an exposure time of the image signal including the flicker component in accordance with a cycle of the flicker component obtained by the flicker detection unit. .
The image processing apparatus according to claim 4, wherein the imaging control unit sets an exposure time of the image signal including the flicker component to a constant multiple of a half period of the flicker component.
The imaging control unit performs control to increase an analog gain of the image signal in accordance with an exposure ratio deficient by setting the exposure time of the image signal including the flicker component. The image processing apparatus described.
The flicker detection unit and the composition processing unit are supplied with a short exposure image signal obtained from a pixel that performs short exposure and a long exposure image signal obtained from a pixel that performs long exposure,
The imaging control unit determines an exposure time of the image signal in which the flicker component is detected by the flicker detection unit of the short-time exposure image signal and the long-time exposure image signal as a half cycle of the flicker component period. The image processing apparatus according to claim 6, wherein the image processing apparatus is set to a constant multiple of.
The image processing apparatus according to claim 7, wherein the flicker detection unit obtains the original brightness of the subject from the image signal whose exposure time is controlled after the exposure time is controlled by the imaging control unit.
When detecting the flicker component from the image signal whose exposure time is not controlled by the imaging control unit, the flicker detection unit obtains the amplitude of the flicker component and supplies the amplitude of the flicker component to the synthesis processing unit. The image processing apparatus according to claim 8.
Among the plurality of image signals picked up at a plurality of types of exposure times, obtain at least the amplitude of the flicker component from the image signal containing the flicker component due to flickering of the light source,
An image processing method comprising the steps of: changing a blend ratio when combining a plurality of the image signals according to the obtained amplitude of the flicker component; and combining the plurality of image signals according to the blend ratio.
Among the plurality of image signals picked up at a plurality of types of exposure times, obtain at least the amplitude of the flicker component from the image signal containing the flicker component due to flickering of the light source,
According to the obtained amplitude of the flicker component, the blend ratio when combining the plurality of image signals is changed, and the computer is caused to execute image processing including a step of combining the plurality of image signals according to the blend ratio. program.