JP7558702B2 - Image blur correction device and control method thereof, and imaging device - Google Patents

Description

The present invention relates to a technology for image blur correction processing in an imaging device, etc.

In imaging devices with image stabilization functions, inertial sensors such as angular velocity sensors are used to detect the amount of camera shake, and control is performed to drive part or all of the imaging optical system based on the shake detection information of the imaging device. The output signal of the inertial sensor contains DC components such as variations in reference voltage due to individual differences, and drift occurs due to temperature changes.

When using an angular velocity sensor, DC components and drift can cause angle errors when determining the angle through time integration. There is a method for suppressing the deterioration of image blur correction accuracy due to angle errors by removing DC components and drift from the output signal of the angular velocity sensor through filtering. Patent Document 1 discloses a method for removing DC components and drift contained in the output signal of an inertial sensor using a Kalman filter.

JP 2018-105938 A

In conventional technology, if the parameter values of the Kalman filter are not appropriate, there is a possibility that the impact on the image stabilization performance, etc. will be significant. For example, if the parameter values are determined by the experience or trial and error of the person setting the filter so as to achieve the results that the person intends, the quality of the Kalman filter will depend on the skill of the person setting the filter.

Furthermore, a large drift may cause fluctuations in the output of the inertial sensor over time for each individual sensor (hereinafter referred to as "fluctuations"). In this case, it is not realistic for a setter to set the parameter values for each individual sensor for a gyro sensor in which the appropriate parameter values for the Kalman filter differ for each individual sensor. Therefore, even when a gyro sensor with large fluctuations is used, the parameter values of the Kalman filter are set uniformly, not for each individual sensor. As a result, if there is variation in the image stabilization effect for each individual imaging device, there is a possibility that differences in image quality may occur even for the same model.
An object of the present invention is to suppress variations in the effect of image blur correction and stabilize image quality.

An image blur correction device of one embodiment of the present invention is an image blur correction device for an image blur correction device that performs image blur correction by acquiring a shake detection signal from a detection means that detects shake in a main body of the image blur correction device equipped with an image blur correction means or an external device that can be attached to the main body, and is characterized in that the image blur correction device comprises: an image blur correction means that performs optical image blur correction by moving the image blur correction means or a correction lens in a direction different from a direction of an imaging optical axis, or an electronic image blur correction by applying image processing to an image captured by the image blur correction means; an image processing means that calculates a motion vector between a plurality of images captured by the image blur correction means and calculates an amount of shake in the image blur correction device based on the motion vector; an estimation means that estimates an offset component as a fluctuation including a DC component contained in the shake detection signal by performing filtering processing with a Kalman filter on a signal obtained by removing the shake amount from the shake detection signal; and a control means that controls the image blur correction means based on a signal obtained by subtracting the offset component from the shake detection signal, and the estimation means changes a parameter value of the Kalman filter using statistical data that indicates a relationship between an angular error and a measurement time interval of the output of the detection means.

The present invention makes it possible to reduce variation in the image stabilization effect and stabilize image quality.

FIG. 1 is a diagram illustrating an example of the configuration of an imaging apparatus according to an embodiment. 2 is a block diagram showing a functional configuration of a control unit included in the imaging device. FIG. 11A and 11B are diagrams illustrating the relationship of an angle error to a measurement time interval of a shake detection output. 4 is a flowchart illustrating a process in the first embodiment. FIG. 13 is a diagram showing changes in angular velocity over time. FIG. 11 is a block diagram showing a functional configuration of a control unit in a second embodiment. FIG. 13 is a diagram for explaining a method of calculating a table representing Allan variance. 10 is a flowchart illustrating a process according to a second embodiment. FIG. 13 is a control block diagram of an imaging apparatus according to a third embodiment. FIG. 4 is a block diagram for explaining the operation of an offset correction unit. 5A and 5B are diagrams illustrating an example of an output waveform of an angular velocity sensor when statically settled. FIG. 4 is a diagram showing a change in an angle signal over time. 1 is an explanatory diagram of a method for estimating an angular fluctuation due to an output error of an angular velocity sensor. 13 is a flowchart illustrating a process according to a third embodiment. 13 is a flowchart illustrating a process according to a first modified example of the third embodiment. 13 is a flowchart illustrating a process according to a second modified example of the third embodiment. 13 is a flowchart illustrating a process in Modification 3 of the third embodiment. FIG. 13 is a control block diagram of an imaging apparatus according to a fourth embodiment. FIG. 13 is a control block diagram of an imaging apparatus according to a fifth embodiment.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
[First embodiment]
FIG. 1A is a central cross-sectional view of an imaging device 1000 according to the present embodiment, and FIG. 1B is a block diagram showing the electrical configuration of the imaging device 1000. The direction perpendicular to the paper surface of FIG. 1A is defined as the x direction, and two mutually orthogonal directions within the paper surface are defined as the y direction and the z direction, respectively. FIG. 1 shows an example of an imaging system consisting of a camera body unit (hereinafter simply referred to as the body unit) 1 and an interchangeable lens 2 that can be attached to the body unit 1. The body unit 1 and the interchangeable lens 2 can be connected via an electrical contact 11. For example, the body unit 1 has the configuration of a so-called mirrorless camera. Note that the present invention is not limited to mirrorless cameras, and can be applied to lens-interchangeable single-lens reflex cameras and various electronic devices having an imaging function.

The main body 1 includes an image sensor 6, an image blur correction unit 14, a shake detection unit 15, and a shutter mechanism 16. The interchangeable lens 2 includes optical members that constitute an imaging optical system, and a shake detection unit 18. When the interchangeable lens 2 is attached to the main body 1, the imaging device 1000 can perform image blur correction of the captured image based on the shake detection signals acquired by the shake detection units 15 and 18.

The interchangeable lens 2 is equipped with an optical system 3, and focus adjustment, aperture drive, image blur correction, etc. are performed by a lens drive unit 13. As a result, light from the subject passes through the optical system 3 and a good image is formed on the image sensor 6 of the main body unit 1. Lens drive control is performed by a lens system control unit (hereinafter referred to as the lens control unit) 12. The lens control unit 12 is equipped with a CPU (Central Processing Unit) and controls the interchangeable lens 2.

The shake detection unit 18 detects shake of the interchangeable lens 2 in a first direction, which is the direction of the imaging optical axis 4, and in a second direction and a third direction that are perpendicular to the first direction and perpendicular to each other, and outputs multiple detection signals to the lens control unit 12. For example, the first direction is parallel to the z direction, the second direction is parallel to the x direction, and the third direction is parallel to the y direction. The lens control unit 12 controls the correction lens (shift lens, tilt lens, etc.) provided in the optical system 3 via the lens drive unit 13.

The main body 1 is equipped with an operation detection unit 10 that detects an instruction operation by the photographer using a release button or the like. For example, in accordance with the instruction operation, the image sensor 6 performs photoelectric conversion on the light from the subject. The image processing unit 7 acquires the output of the image sensor 6, performs image processing, and stores the image data in the memory unit 8. At this time, the image processing unit 7 can perform vector calculations to detect the amount of movement of the subject image. Vector calculations are calculations that determine the direction and distance of movement of feature points from images taken at different times. The amount of movement of the subject image and the amount of camera shake can be obtained from the results of vector calculations using known technology. Specific calculation methods are omitted.

The camera system control unit (hereinafter referred to as the camera control unit) 5 has a CPU and controls the imaging device 1000. The camera control unit 5 communicates with the lens control unit 12 via electrical contacts 11 when the interchangeable lens 2 is attached to the main body unit 1, and cooperates with them to control the imaging system. A live view display before shooting is performed by the display unit 9 under the control of the camera control unit 5. The display unit 9 is composed of a liquid crystal display unit 9a provided on the back of the main body unit 1, and an electronic viewfinder 9b having a display device within the finder. The memory unit 8 has a storage medium, and image data acquired by shooting is stored in the storage medium.

The shutter mechanism 16 is driven and controlled by the camera control unit 5 via the shutter drive unit 17. The exposure time of the image sensor 6 is controlled according to the shooting time set by the photographer or the shooting time determined by the camera control unit 5.

The image blur correction unit 14 performs image blur correction by moving the image sensor 6 according to a control command from the camera control unit 5. The image blur correction unit 14 moves the image sensor 6, for example, in a second or third direction perpendicular to the photographing optical axis 4 of the interchangeable lens 2, based on an output correction value calculated from the output of the shake detection unit 15 of the main body 1. This makes it possible to correct image blur caused by camera shake due to the photographer's hand shake, etc.

The shake detection unit 15 detects the shake of the main body unit 1 in three directions and outputs multiple shake detection signals to the camera control unit 5. For example, the shake detection unit 15 has an angular velocity sensor. The camera control unit 5 acquires the output of the shake detection unit 15, calculates the output correction value, and controls the movement of the image sensor 6 relative to the shooting optical axis 4.

In the configuration example of FIG. 1, image blur correction based on movement control of the image sensor 6 has been described, but the present invention is not limited to this and can be applied to the following embodiments.
(1) Optical image blur correction based on drive control of a correction lens provided in the interchangeable lens 2.
(2) Electronic image blur correction based on image processing of the captured image (image area extraction processing).
(3) A composite image blur correction using the configuration of FIG. 1 or a combination of (1) and (2).

Next, the offset estimation unit 5002 provided in the camera control unit 5 will be described in detail with reference to FIG. 2. FIG. 2 is a block diagram showing a functional configuration for explaining the operation of the camera control unit 5. The camera control unit 5 includes adders 5001 and 5003, an offset estimation unit 5002, an integral filter 5004, and a memory unit 5100. In the following, it is assumed that the calculations performed by the adder include the addition (subtraction) of negative values.

The adder 5001 acquires the output signal of the angular velocity sensor of the shake detection unit 15 and the signal of the camera shake amount calculated by vector calculation in the image processing unit 7, and calculates the difference between the two signals. The adder 5001 outputs the difference signal to the offset estimation unit 5002.

Data stored in the memory unit 5100 is input to the offset estimation unit 5002 in order to estimate the fluctuation of the output of the shake detection unit 15 from the differential signal. Data of the observation noise variance 5101 and the system noise variance 5102 are stored in the memory unit 5100. These data are calculated from a table showing the relationship of the angular error to the measurement time interval of the output of the shake detection unit 15 when stationary, and will be described in detail later.

The offset estimation unit 5002 executes an offset estimation process based on the input signal. That is, the offset estimation unit 5002 acquires a fluctuation signal superimposed on the output of the shake detection unit 15. The fluctuation signal acquired by the offset estimation unit 5002 is sent to the adder 5003.

The adder 5003 obtains the fluctuation signal of the output of the shake detection unit 15 and the output signal of the shake detection unit 15 and performs subtraction. By subtracting the fluctuation signal from the output signal of the shake detection unit 15, it is possible to obtain the output signal of the shake detection unit 15 in which the influence of the fluctuation has been removed or reduced. The integral filter 5004 obtains the output correction value of the shake detection unit 15 by performing time integration on the output signal of the adder 5003. This output correction value is input to the image blur correction unit 14 and is used to control the amount of movement of the image sensor 6.

Next, the estimation process performed by the offset estimation unit 5002 will be described in detail. The offset is a fluctuation including a DC component included in the output of the shake detection unit 15. An example in which the offset estimation unit 5002 is configured as a linear Kalman filter and performs filter processing will be shown. The linear Kalman filter can be expressed by the following formulas (1) to (7).

Equation (1) represents an operation model in a state space representation, and equation (2) represents an observation model. The meanings of each symbol are as follows:
A: System matrix in the operating model.
B: Input matrix.
C: Output matrix in the observation model.
ε _t : process noise.
δ _t : Observation noise.
t: discrete time.
u _t : control input.

式（３）は予測ステップにおける事前推定値を表し、式（４）は事前誤差共分散を表す。式（４）中の、Σ_ｘは動作モデルのノイズの分散を表す。

Equation (3) represents the a priori estimate in the prediction step, and equation (4) represents the a priori error covariance, where Σ _x represents the variance of the noise in the motion model.

式（５）はフィルタリングステップにおいてカルマンゲインの算出式を表し、上付き添え字のＴは転置行列を表している。さらに式（６）はカルマンフィルタによる事後推定値を表し、式（７）は事後誤差共分散を表す。また式（５）中のΣ_ｚは観測モデルのノイズの分散を表す。

Equation (5) represents the calculation formula for the Kalman gain in the filtering step, and the superscript T represents the transposed matrix. Equation (6) represents the posterior estimate by the Kalman filter, and Equation (7) represents the posterior error covariance. In addition, Σ _z in Equation (5) represents the noise variance of the observation model.

In this embodiment, in order to estimate the fluctuation of the output of the shake detection unit 15, the offset value of the output of the shake detection unit 15 is set to _xt , and the output of the shake detection unit 15 is set to _zt . Furthermore, if _εt is system noise and _δt is observation noise, the model representing the offset value does not have _ut , which is the control input term in formula (1), and can be expressed by the following first-order linear model in which A=C=1 in formulas (1) and (2).

で表す。また、式（５）における観測モデルのノイズの分散Σ_ｚを、

で表す。 The noise variance Σ _x of the operation model in the above formula (4) is expressed as the system noise variance

The noise variance Σ _z of the observation model in equation (5) is expressed as follows:

It is expressed as:

とすると、時刻ｔにおけるシステムノイズの事前誤差分散推定値は、

となる。さらに時刻ｔにおけるカルマンゲインを、

とし、振れ検出部１５の出力を、

とすると、以下の式でカルマンフィルタを構成することができる。

The offset is estimated in advance as

Then, the a priori error variance estimate of the system noise at time t is

Furthermore, the Kalman gain at time t is

The output of the shake detection unit 15 is expressed as follows:

Then, the Kalman filter can be constructed using the following equation.

と、振れ検出部１５により出力されるシステムノイズの分散

と、時刻「ｔ－１」での誤差分散推定値

により、オフセット事前推定値

および事前誤差分散推定値

が算出される。そして事前誤差分散推定値

および振れ検出部１５により出力される観測ノイズの分散

に基づいてカルマンゲイン

が算出される。 The offset estimation unit 5002 performs estimation calculations using the above-mentioned formulas (10) to (14). The offset estimation value at time "t-1" of the update period of the estimation calculation is

and the variance of the system noise output by the shake detection unit 15

and the error variance estimate at time "t-1"

Therefore, the offset is estimated

and the prior error variance estimate

is calculated. Then the prior error variance estimate

and the variance of the observation noise output by the shake detection unit 15

Based on the Kalman gain

is calculated.

と、オフセット事前推定値

との誤差にカルマンゲイン

を乗じた値によってオフセット事前推定値

が修正され、オフセット推定値

が算出される。 According to the formula (13),
Observed output of the shake detector 15

and the offset pre-estimate

The error with the Kalman gain

Offset the prior estimate by multiplying

is corrected and the offset estimate

is calculated.

が修正されて、誤差分散推定値

が算出される。これらの演算によって事前推定更新と修正を演算周期ごとに繰り返すことでオフセット推定値が算出される。 Furthermore, by using equation (14), the prior error variance estimate

is modified to give the error variance estimate

By repeating the advance estimation update and correction using these calculations every calculation cycle, an offset estimation value is calculated.

および観測ノイズの分散

がカルマンフィルタのパラメータに相当する。パラメータ値の設定方法として、従来の方法では、推定の収束速度とフィルタの安定性とを考慮して観測ノイズの分散

を定めるという指針しか示されていない。設定者はその意図した結果が得られるようにパラメータ値を設定するが、ドリフトといった慣性センサ出力の揺らぎが大きいジャイロセンサに対して、設定者がセンサ個体ごとに値を設定することは現実的ではない。例えば、揺らぎの大きい慣性センサを使用する場合にカルマンフィルタのパラメータ値をセンサ個体によらず一律に定める場合を想定する。この場合、個々の撮像装置ごとに像ブレ補正効果にばらつきが発生すると、同一機種でも画像品位に差異が生じる可能性がある。 In the Kalman filter configured as above, the variance of the system noise is

and the variance of the observation noise

are the parameters of the Kalman filter. In the conventional method, the variance of the observation noise is set by taking into account the convergence speed of the estimation and the stability of the filter.

The only guideline given is to determine the parameter values. The designer sets the parameter values so that the intended results are obtained, but for gyro sensors, whose inertial sensor output has large fluctuations such as drift, it is not realistic for the designer to set the values for each individual sensor. For example, when using an inertial sensor with large fluctuations, it is assumed that the parameter values of the Kalman filter are set uniformly regardless of the individual sensor. In this case, if there is variation in the image stabilization effect for each individual imaging device, there is a possibility that the image quality will differ even for the same model.

Referring to Figure 3, a table based on statistical data showing the relationship of angular error to the measurement time interval of the output of the shake detection unit 15 and the Allan variance used in the table will be described in detail. Figure 3 (A) shows an example of a table showing the relationship of angular error to the measurement time interval of the output of the shake detection unit 15. The value of the Allan variance σ(τ) corresponding to each measurement time interval τ and the slope between adjacent data are shown. The measurement time interval τ in this example is the sampling time Δt or a power of 2 multiple of that.

式（１６）のＫは、データ数ｍ個のときのグループの数を表す。式（１７）のτはデータ数ｍ個のグループにおける測定時間間隔τを表す。式（１５）では、データ数ｍ個のグループにおいて隣り合うグループの平均値の差の平均を算出する演算が行われる。よって、アラン分散は、データを測定時間間隔τで測定した際の測定データの標準偏差（以下、偏差ともいう）を示していることがわかる。 FIG. 3B is a double logarithmic graph, with the horizontal axis representing the measurement time interval τ and the vertical axis representing the Allan variance σ(τ). A solid line 301 indicates the noise characteristics of the output signal of the shake detection unit 15 when stationary. The Allan variance σ(τ) is widely used as an index of the frequency stability of the output of an inertial sensor such as an angular velocity sensor, and represents the degree of variation in the output during the measurement time interval τ. Although it is called variance, it is actually a standard deviation. However, σ(τ) is generally called the Allan variance. The Allan variance σ(τ) when the sampling time Δt and the total number of data N (here, ²ⁿ ) are divided into m groups (m=1, 2, 4, 8, ..., N/2) can be calculated by the following formula (15).

In formula (16), K represents the number of groups when the number of data pieces is m. In formula (17), τ represents the measurement time interval τ in a group with m pieces of data. In formula (15), a calculation is performed to calculate the average of the differences in the average values of adjacent groups in a group with m pieces of data. Therefore, it can be seen that the Allan variance indicates the standard deviation (hereinafter also referred to as deviation) of the measurement data when the data is measured at the measurement time interval τ.

The measurement time interval τ is the time interval for acquiring data, so it can be rephrased as the sampling time for a group of m pieces of data. Therefore, from the Allan variance graph in Figure 3(B), the deviation of the measurement data when data is sampled at the measurement time interval τ can be read. In this case, the measurement time interval τ of the data plotted at the left end of the solid line 301 is the sampling time Δt when the data is acquired, and deviations for measurement time intervals τ shorter than this cannot be acquired. The table shown in Figure 3(A) lists the Allan variance values at τ when m = 1 to N/2 in equation (17). The solid line 301 in Figure 3(B) shows the table data on a double logarithmic graph.

From the graph showing the noise characteristics due to Allan variance in FIG. 3B, the characteristics of the noise in a signal measured at a sampling time τ can be known. In general, noise contained in the output of an inertial sensor includes white noise, 1/f noise, and random walk noise, and the noise that exhibits the main characteristics changes by changing the measurement time interval τ. The straight line portion with a slope of "-1/2" shown in the first range 302 in FIG. 3B is a range that exhibits the characteristics of white noise. Also, the straight line portion with a slope of "+1/2" shown in the second range 303 in FIG. 3B is a range that exhibits the characteristics of random walk noise. Although not shown, it is known that when a signal contains 1/f noise, a straight line range with a slope of 0 appears between ranges 302 and 303.

In Fig. 3B, as shown by the solid line 301, a single continuous curve is formed, but in reality, it is a sum of lines representing the deviations of the white noise and random walk noise contained in the noise of the signal, i.e., the Allan variance. Therefore, by extending each line of inclination in Fig. 3B, it is possible to read the Allan variance when the noise corresponding to that inclination is sampled at a measurement time interval τ. The solid line 301 in Fig. 3B is a line representing the noise characteristics of the output signal of the shake detection unit 15 when the shake detection unit 15 is stationary, but the output signal itself can be considered as the observation noise. Therefore, the solid line 301 is a line representing the Allan variance of the observation noise of formula (12), i.e., the deviation σ _z of the observation noise with respect to the sampling time.

3B indicates the random walk noise contained in the output signal of the shake detection unit 15 when the camera is stationary, i.e., the variance of the fluctuations. This line indicates the Allan variance of the system noise in equation (11), i.e., the deviation σ _x of the system noise with respect to the sampling time.

The value of the Allan variance of τ corresponding to the desired sampling time can be read from these lines. By using the obtained Allan variance value in equations (10) to (14), it becomes possible to set each variance corresponding to the parameters of the Kalman filter from the Allan variance corresponding to the inertial sensor output.

Next, a method for obtaining each variance when the sampling time is Δt will be described with reference to Fig. 3. First, in the case of the deviation _σz of the observation noise representing the output noise of the inertial sensor, the measurement time interval τ coincides with the sampling time Δt of the output noise. Therefore, the deviation _σz of the observation noise can be obtained from the Allan variance when τ = Δt in equation (15).

式（１８）の右辺に示すＫはレートランダムウォーク係数と呼ばれ、τ＝３のときに傾き１／２の線が取るアラン分散の値である。この値は、実線３０１で傾きが１／２になる範囲を延長した破線３０４の切片を算出することで求まる値である。図３（Ａ）のテーブルからＫ値を求める場合、アラン分散の傾きが１／２になる範囲の中から任意の１点を選び、式（１８）を用いてＫについて解けばよい。傾き１／２の範囲内であれば、システムノイズの偏差σ_ｘはアラン分散と一致する。尚、傾きが１／２と一致しない場合には、傾き０および１と区別できる範囲内で傾き１／２に対して許容誤差を設け、その範囲内から選択してもよい。 In addition, in the case of a deviation σ _x of system noise representing fluctuations in the output noise of the inertial sensor, the Allan variance can be calculated from the value when the measurement time interval τ=Δt in the following formula.

K on the right side of formula (18) is called the rate random walk coefficient, and is the value of the Allan variance of the line with a slope of 1/2 when τ=3. This value is obtained by calculating the intercept of the dashed line 304, which is an extension of the range where the slope of the solid line 301 is 1/2. When calculating the K value from the table in FIG. 3(A), an arbitrary point is selected from the range where the slope of the Allan variance is 1/2, and K is solved using formula (18). If it is within the range of the slope of 1/2, the deviation σ _x of the system noise coincides with the Allan variance. If the slope does not coincide with 1/2, a tolerance is set for the slope of 1/2 within a range that can be distinguished from slopes of 0 and 1, and a value may be selected from within that range.

The system noise deviation σ _x and the observation noise deviation σ _z obtained by equations (15), (18), and (19) are actual deviations obtained from a signal obtained by removing the amount of shake of the main body 1 superimposed on the output of the shake detection unit 15. Therefore, by using this method, it is possible to set the parameter values of the Kalman filter according to an individual gyro sensor even for a gyro sensor with large fluctuations.

For the parameter values of the Kalman filter in this embodiment, a process is carried out in advance at a factory or the like to calculate a table from data acquired when the angular velocity sensor of the shake detection unit 15 is in a stationary state. The variance data of each noise calculated from the table is stored in the storage unit 5100. In addition to this method, there is also a method of generating a table using data from the angular velocity sensor acquired when the imaging device 1000 is in a stationary state to calculate the parameter values of the Kalman filter.

The process of acquiring the output correction value of the shake detection unit 15 by the offset estimation unit 5002 will be described in detail with reference to Figures 4 and 5. The output correction value of the shake detection unit 15 is the output obtained by removing the offset from the output of the shake detection unit 15, and corresponds to the target value of the shake correction amount of the image shake correction unit 14. Note that, in this embodiment, offset estimation is performed on the output of the shake detection unit 15 in the main body 1, but the target of the offset estimation may also be the output of the shake detection unit 18 in the interchangeable lens 2.

Figure 4 is a flow chart explaining the process from when the power of the imaging device 1000 is turned on until the start of shooting, which starts when the photographer turns on the power of the main body 1. Also, Figure 5 is a diagram showing output signals, with the horizontal axis showing the time axis and the vertical axis showing the angular velocity. Figure 5 (A) shows the output signal 51 of the shake detection unit 15. Figure 5 (B) shows the amount of shake 52 of the main body 1 acquired from the motion vector information of the image processing unit 7. Figure 5 (C) shows a signal 53 obtained by removing the amount of shake 52 of the main body 1 from the output signal 51 of the shake detection unit 15, and an estimated waveform 54 of the offset component acquired by the offset estimation unit 5002 (see the white line). Figure 5 (D) shows a waveform 55 obtained by removing the estimated waveform 54 of the offset component from the output signal 51 of the shake detection unit 15.

In S401 of FIG. 4, the camera control unit 5 acquires the output signal 51 of the shake detection unit 15 of the main body unit 1, and proceeds to processing of S402. In S402, the camera control unit 5 performs processing to detect the amount of shake 52 of the main body unit 1 from the motion vector information using the image processing unit 7, and then proceeds to processing of S403. The motion vector information is shake detection information based on a comparison between multiple images acquired by the image sensor 6. The amount of shake of the main body unit 1 can be detected from this motion vector information.

In S403, the camera control unit 5 obtains a signal 53 that removes or reduces the amount of shake 52 of the main body unit 1 detected in S402 from the output signal 51 of the shake detection unit 15 of the main body unit 1 detected in S401, and then proceeds to processing of S404.

In S404, the offset estimation unit 5002 estimates the offset component contained in the signal 53 acquired in S403. The offset estimation unit 5002 executes an offset component estimation process using the signal 53 acquired in S403 and the data of the observation noise variance 5101 and the system noise variance 5102 pre-stored in the memory unit 5100. The method of estimating the offset component using the Kalman filter and the method of calculating the observation noise variance 5101 and the system noise variance 5102 have been described above, so their description will be omitted. The waveform 54 indicated by the white line in Figure 5 (C) represents the time change of the offset component estimated by the offset estimation unit 5002.

After calculating the estimated waveform 54 of the offset component, the process proceeds to S405, where the camera control unit 5 subtracts the signal of the estimated waveform 54 of the offset component from the output signal 51 of the shake detection unit 15 of the main body unit 1 to obtain the angular velocity waveform 55 of FIG. 5(D). By removing the estimated waveform 54 of the offset component from the output signal 51, the angular velocity waveform 55 becomes a waveform that represents the amount of shake of the main body unit 1 detected by the shake detection unit 15.

Next, the process proceeds to S406, where the camera control unit 5 performs time integration processing on the angular velocity waveform 55 acquired in S405 using the integral filter 5004. The output correction value of the shake detection unit 15, which represents the shake caused by the amount of angular displacement of the main body unit 1, is acquired, and the process ends.

In this embodiment, the above series of operations allows the parameter values of the Kalman filter to be set according to the fluctuations of each individual inertial sensor in offset estimation using filter processing. By suppressing the variation in the image stabilization effect of imaging devices of the same model, it is possible to stabilize image quality.

[Second embodiment]
Next, a second embodiment of the present invention will be described. In this embodiment, a processing example in which parameter values are switched according to the time change of the Allan variance is shown. The main difference between this embodiment and the first embodiment is that in this embodiment, processing is performed taking into consideration the case in which the fluctuation contained in the output of the shake detection unit 15 changes over time. By reusing the symbols and signs already used for matters similar to those in the first embodiment, detailed explanations of those matters will be omitted and differences will be mainly described. The method of omitting such explanations will be the same in the embodiments described later.

The operation of the camera control unit 5 will be described with reference to FIG. 6. FIG. 6 is a block diagram showing an example of the configuration for explaining the operation of the camera control unit 5. The difference from FIG. 2 is that a table calculation unit 5201 and a determination unit 5202 in the storage unit 5100 are provided.

The adder 5001 of the camera control unit 5 calculates the difference between the output signal of the shake detection unit 15 and the signal of the camera shake amount acquired by the vector calculation of the image processing unit 7. The acquired difference data is input to the table calculation unit 5201. The details of the table calculation process will be described later.

The data of the observed noise variance and the data of the system noise variance acquired by the table calculation unit 5201 are stored in the memory unit 5100. When storing, the determination unit 5202 in the memory unit 5100 compares the value of the system noise variance calculated by the table calculation unit 5201 with the value of the system noise variance 5102 stored in the memory unit 5100, and determines which value to store and hold. The reason why only the variance value of the system noise is compared here is because only the variance value of the system noise changes due to the influence of fluctuations.

The data of the observation noise variance 5101 and the system noise variance 5102 stored in the memory unit 5100 and the difference data acquired by the adder 5001 are input to the offset estimation unit 5002. The offset estimation unit 5002 executes an estimation process of the offset component superimposed on the output of the shake detection unit 15.

The adder 5003 subtracts the offset component of the shake detection unit 15 acquired by the offset estimation unit 5002 from the output of the shake detection unit 15 to acquire an output signal of the shake detection unit 15 in which the influence of the fluctuation has been removed or reduced. The integral filter 5004 performs time integration of this output signal, and the output correction value of the shake detection unit 15 is acquired and input to the image shake correction unit 14.

Next, referring to FIG. 7, a method for calculating the table representing the Allan variance in the table calculation unit 5201 will be described in detail. This table represents the relationship between the measurement time interval of the difference waveform of the adder 5001 and the angle error. Below, a method for determining the measurement time required to obtain the Allan variance and a method for updating the Allan variance will be described.

Figure 7 is a graph showing a differential waveform 71 of the adder 5001 immediately after the imaging device 1000 is powered on. The horizontal axis represents time, and the vertical axis represents angular velocity. A temperature drift called startup drift occurs in the angular velocity sensor of the shake detection unit 15 immediately after power is turned on, and low-frequency components are superimposed on the sensor output as shown in range 72. The effect of this startup drift decreases over time, so the superimposition of low-frequency components also disappears over time. Until the startup drift is eliminated, the magnitude of the fluctuation due to the drift continues to change, so the deviation of the system noise also changes over time.

In order to obtain the system noise deviation that changes over time, the table calculation unit 5201 starts collecting differential waveform data from the adder 5001 immediately after the imaging device 1000 is turned on. After a predetermined time (denoted as s seconds) has elapsed, the system noise deviation is started to be obtained from the Allan variance of the differential waveform data. Here, s seconds is the measurement time required to obtain the system noise deviation. When obtaining the system noise deviation, the observed noise deviation is also obtained. The maximum measurement time interval for the Allan variance obtained by measurement for s seconds is s/2 seconds. However, since the Allan variance obtained at the maximum measurement time interval is a value obtained from the difference between only two groups, the accuracy of the value representing the deviation is lower than the predetermined accuracy. Therefore, the measurement time of s seconds is set to a time that is sufficient to obtain the accuracy required to obtain the system noise deviation. In addition, since the magnitude of the fluctuation changes over time, the value of the system noise deviation must be updated in accordance with the change. Time T in FIG. 7 represents the elapsed time from the point when table calculation unit 5201 started acquiring differential waveform data from adder 5001, and has a relationship of "T≧s" with respect to s. After T seconds have elapsed, table calculation unit 5201 calculates the Allan variance for T-Δs seconds at any time in accordance with the sampling time of shake detection unit 15, acquires the deviation of system noise and observation noise, and continues updating.

Referring to FIG. 8, a detailed description will be given of a series of processes until the camera control unit 5 acquires the output correction value of the shake detection unit 15 by the offset estimation unit 5002. FIG. 8 is a flowchart that explains the processes from when the power of the imaging device 1000 is turned on until the start of shooting. For the same processes as those shown in the flowchart of FIG. 6, the same step numbers as in FIG. 6 are used, and detailed descriptions thereof will be omitted.

After the processes of S401 to S403, the process proceeds to S801. In S801, after the imaging device 1000 is powered on, the camera control unit 5 compares the elapsed time T with the measured time s seconds. The elapsed time T is measured by a timer in the camera control unit 5. A determination is made regarding the elapsed time T as to whether the measured time s seconds has elapsed. If it is determined that "T≧s", the process proceeds to S802, and if it is determined that "T<s", the determination process of S801 is repeated.

In S802, the table calculation unit 5201 calculates the Allan variance using the differential waveform data of the adder 5001 for s seconds immediately preceding the elapsed time T, and obtains each variance from the deviation of the observed noise and the system noise. The method of calculating the Allan variance and obtaining the variance of each noise is as described above. After obtaining the variance of the observed noise and the variance of the system noise, the process proceeds to S404.

In S803 following S406, the determination unit 5202 obtains the difference value between the system noise variance obtained in S802 and the system noise variance 5102 stored in the storage unit 5100, and determines whether the difference value is equal to or less than a threshold value. This threshold value is set to a value at which it can be determined that the system noise variance value obtained in S802 has converged to the system noise variance value in the storage unit 5100. If it is determined that the obtained difference value is greater than the threshold value, the process proceeds to S804, and if it is determined that the difference value is equal to or less than the threshold value, the series of processes ends. Note that the method is not limited to comparing the difference value between the currently obtained variance value and the previously obtained variance value with the threshold value, and any method can be used as long as it can determine the convergence to the above variance value.

In S804, a process is executed to update the data values of the observation noise variance 5101 and the system noise variance 5102 stored in the memory unit 5100 with the variance values obtained in S802, and then the process proceeds to S802.

In this embodiment, when the fluctuations contained in the output of the shake detection unit 15 change over time, a process is performed to update the parameter values of the filter means (Kalman filter) as needed in accordance with the output of the shake detection unit 15. By suppressing the variation in the image stabilization effect of imaging devices of the same model, it is possible to stabilize image quality.

[Third embodiment]
A third embodiment of the present invention will be described with reference to Fig. 9 to Fig. 17. In this embodiment, an example of an imaging apparatus having an image stabilization device similar to that of the above-mentioned embodiment and capable of suppressing degradation of image stabilization performance in long exposure or super telephoto shooting is shown. In the conventional technology, it is difficult to control image stabilization more accurately in accordance with the characteristics of the vibration detection sensor, and it is also not possible to perform accurate control only for the output error of the vibration detection sensor.

In this embodiment, as a measure against degradation of image blur correction performance caused by output error of the angular velocity sensor, a blurred image generation prevention process is performed when a predicted error is equal to or greater than a threshold value based on the Allan variance, exposure time, and focal length. An example of an imaging device equipped with an imaging optical system and an image processing device is described below.

Figure 9 is a block diagram showing the configuration of the imaging device 100. The imaging device 100 is a digital camera, and is composed of a main body 101 and an interchangeable lens 102 that can be attached to the main body 101. The main body 101 includes an image sensor, an image blur correction unit, a shutter mechanism, etc.

The camera CPU 201 provided in the main body 101 controls the imaging device 100 and controls each component shown in FIG. 9 in response to an instruction operation by the photographer using the release button 104, etc.

Light from the subject passes through the imaging optical system 213 along the imaging optical axis 214 shown in FIG. 9 and enters the imaging element 210. The imaging element 210 is a CMOS (complementary metal-oxide semiconductor) image sensor or the like, and outputs an electrical signal corresponding to an optical image in response to an input light beam. The signal output from the imaging element 210 is subjected to image processing in an image processing unit, and the generated image data is stored in a memory unit. The focal plane shutter 212 controls the blocking state of the light entering the imaging element 210. The focal plane shutter 212 and the imaging element 210 are driven according to a control signal from the camera CPU 201.

The main body 101 includes a vibration sensor unit 202. The angular velocity sensor in the vibration sensor unit 202 detects angular velocity from vibrations applied to the main body 101. The angular velocity detection signal is input to an offset correction unit 203. The offset correction unit 203 refers to motion vector information acquired from multiple captured images and corrects the offset that occurs in the angular velocity sensor.

The calculation unit 204 acquires the angular velocity signal corrected by the offset correction unit 203, and performs first-order time integration, etc. to calculate a target value. The calculation unit 204 calculates the amount of correction for the image blur correction mechanism from the signal converted into an angle signal. The control unit 205 controls the correction unit 211, which drives the image blur correction mechanism, based on the amount of correction (angle information) calculated by the calculation unit 204. The correction unit 211 drives the image sensor 210 in a direction perpendicular to the imaging optical axis 214, as indicated by arrow 211(a), in accordance with a control signal from the control unit 205. It is possible to correct image blur that occurs on the imaging surface of the image sensor 210 due to vibration of the main body unit 101.

The vibration sensor characteristic calculation unit 206 performs a calculation process of the noise characteristics of the vibration sensor based on the angular velocity signal corrected by the offset correction unit 203. The vibration sensor integrated error calculation unit 207 calculates an integrated error based on the noise characteristics calculated by the characteristic calculation unit 206 and the information of the exposure time and focal length set by the photographer, and predicts the amount of blur of the image under the shooting conditions. The information of the exposure time and focal length set by the photographer is obtained from the camera CPU 201.

The determination unit 208 performs a process of determining whether or not a blurred image is generated. For example, the determination unit 208 compares the predicted value of the amount of blur of the image calculated by the integrated error calculation unit 207 with a threshold value, and determines whether the predicted value is equal to or greater than the threshold value. When the determination unit 208 determines that the amount of image blur is equal to or greater than the threshold value based on the predicted result of the amount of blur of the image obtained by the integrated error calculation unit 207, the blurred image prevention processing unit 209 sends a command to the camera CPU 201 to suppress or prevent the generation of a blurred image. The camera CPU 201 executes the process of suppressing or preventing the generation of a blurred image in accordance with the command from the blurred image prevention processing unit 209.

Figures 10 to 12 are explanatory diagrams regarding offset correction of an angular velocity sensor. Figure 10 is a block diagram for explaining the operation of the offset correction unit 203. The offset correction unit 203 includes a motion vector detection unit 1301, a multiplication unit 1302, and a division unit 1304.

The multiplication unit 1302 acquires and multiplies the output signal of the angular velocity sensor of the vibration sensor unit 202 by the motion vector information output from the motion vector detection unit 1301. The deviation 1303 (denoted as e) calculated by the multiplication unit 1302 corresponds to the offset amount superimposed on the angular velocity sensor. The division unit 1304 acquires information on the deviation e, and performs division according to the deviation e on the output of the angular velocity sensor to generate an angular velocity signal divided by the offset amount.

11(A) and 11(B) are graphs showing examples of output waveforms of an angular velocity sensor in a static state. The vertical axis represents the angular velocity signal, and the horizontal axis represents the time axis. Graph line 305 in FIG. 11(A) shows a state in which an offset occurs in the angular velocity sensor. In a state in which an offset occurs, the angular velocity sensor may output a signal with fluctuations that would not normally occur even in a static state in which the image capture device has little shake. Therefore, if image blur correction is performed based on a detection signal that includes unnecessary signal components, an image with image blur will be obtained due to erroneous image blur correction. In contrast, graph line 306 in FIG. 11(B) shows the waveform of an angular velocity sensor that has been subjected to offset correction processing. The waveform of graph line 306 is a waveform with less fluctuation than the waveform of graph line 305 in FIG. 11(A).

It is possible to output a signal related to the angular velocity sensor generated by offset correction (see graph line 306), i.e., a signal from the angular velocity sensor with reduced offset, thereby suppressing degradation of image stabilization performance.

Figure 12 (A) shows the change in the angle signal over time, with the horizontal axis being the time axis and the vertical axis being the angle axis. The waveform shown by graph line 307 shows the angle signal obtained by integrating the waveform of the angular velocity signal after offset correction shown by graph line 306 in Figure 11 (B) by the calculation unit 204 (Figure 9). If a delay occurs in the calculation process during offset correction, correction residues will occur in the angular velocity sensor, and errors may occur in the output signal. The waveform shown by graph line 307 in Figure 12 (A) shows the value obtained by integrating the angular velocity signal including the output error, and angle fluctuations that would not normally occur can be confirmed. This angle fluctuation corresponds to the error after integration with respect to the integration time. The impact of angle fluctuations caused by output errors in the angular velocity sensor will be described in detail with reference to Figure 12 (B).

12B is a shooting sequence diagram when still image shooting is performed by the imaging device 100. The meanings of the symbols are as follows.
RLS: A signal that is output when the release button 104 is pressed.
SIG_A: a front curtain drive signal for the focal plane shutter 212 in FIG.
SIG_B: rear curtain drive signal for the focal plane shutter 212 in FIG.
SIG_deg: represents an angular waveform obtained by integrating a signal including an output error of the angular velocity, and corresponds to the waveform of graph line 307 in FIG. 12(A).
A plurality of times t1 to t4 are shown on the time axis. A first period Tv_A is the period from time t2 to time t3, and a second period Tv_B is the period from time t2 to time t4. It is assumed that the period length of Tv_B is longer than the period length of Tv_A.

When the user presses the release button 104 at time t1 in FIG. 12(B), the RLS signal changes, and the imaging device 100 transitions from a non-shooting state to a shooting state. After a certain time has passed since the RLS signal changed, the SIG_A signal changes, and driving of the front curtain begins. Time t2 indicates the timing at which driving of the front curtain of the focal plane shutter 212 begins. A certain time after the output of the SIG_A signal, the SIG_B signal changes, and driving of the rear curtain of the focal plane shutter 212 begins.

The focal plane shutter 212 is controlled and driven according to the exposure time set by the photographer. Time t3 in FIG. 12(B) indicates the timing at which the rear curtain drive starts in accordance with the setting of the first exposure time for a shooting condition of several seconds, such as camera shake. The period Tv_A corresponds to the first exposure time from time t2 to time t3.

On the other hand, time t4 in FIG. 12(B) indicates the timing of starting rear curtain drive in accordance with the setting of a second exposure time that is longer than the first exposure time. For example, the second exposure time is an exposure time assuming image capture performed with the imaging device 100 fixed to a tripod. Period Tv_B is a period corresponding to the second exposure time from time t2 to time t4.

For example, if a user takes a handheld shot with a lens with a focal length of 100 mm, and the exposure conditions are relatively short (several seconds) as shown in period Tv_A, the fluctuations in the angle signal caused by the output error of the angular velocity signal are small and do not affect image blur correction. However, when shooting under relatively long exposure conditions with the camera fixed to a tripod or the like, as in period Tv_B, the fluctuations in the angle signal caused by the output error of the angular velocity signal cannot be ignored. In other words, if incorrect image blur correction is performed due to the influence of the fluctuations in the angle signal, an image with image blur may be generated.

Even under the exposure conditions shown in period Tv_A, if a user takes a photograph with a super telephoto lens with a large focal length, even a slight movement of the image stabilization mechanism is magnified when the subject is photographed. Therefore, an image with image blur may be generated due to the movement of the image stabilization mechanism based on the fluctuation of the angle signal caused by the output error of the angular velocity signal.

In this way, image blurring due to output errors from the angular velocity sensor can occur when shooting with a long exposure time using a fixed tripod or when shooting with a super telephoto lens. In order to prevent degradation of image blurring correction performance caused by output errors from the angular velocity sensor, it is necessary to estimate the angular fluctuation due to output errors from the angular velocity sensor in advance, predict the amount of image blurring according to the shooting conditions, and handle it appropriately.

A method for estimating angular fluctuations due to output errors of the angular velocity sensor will be described with reference to FIG. 13. Graph line 401 in FIG. 13(A) shows the Allan standard deviation for the angular velocity sensor output after offset correction shown by graph line 306 in FIG. 11(B). The horizontal axis in FIG. 13(A) represents the measurement time interval τ of the angular velocity sensor output signal after offset correction. The vertical axis in FIG. 13(A) represents the standard deviation value στ of the angular velocity sensor output signal after offset correction at the measurement time interval τ. Both axes are shown on a logarithmic scale.

The Allan standard deviation corresponds to a quantity that represents the noise characteristics of an angular velocity sensor. The Allan standard deviation is widely used as an index when evaluating inertial sensors such as angular velocity sensors. With the Allan standard deviation, the standard deviation value of the sensor output signal to be evaluated can be evaluated on the time axis. In this embodiment, this characteristic can be used to estimate the angular fluctuation. Note that the Allan standard deviation is generally known as the Allan variance, but in order to clearly state that it is a standard deviation as described above, in this embodiment it is referred to as the Allan standard deviation.

13A, the horizontal axis is the measurement time interval τ of the angular velocity sensor output signal after offset correction. Therefore, as shown in the following formula (20), the value obtained by multiplying the Allan standard deviation value στ corresponding to the measurement time interval τ by the measurement time interval τ can be treated as the angular fluctuation obtained by integrating the angular velocity.
τ(sec)×στ(dps)=Angle fluctuation estimated value (degree) Equation (20)
In Fig. 13A, the unit of the measurement time interval τ is "seconds", and the unit of the Allan standard deviation στ corresponding to τ is "degrees per second", that is, angles per second. As an example, on graph line 401 in Fig. 13A, at the position where the measurement time interval τ is 20 seconds, the Allan standard deviation value στ of the angular velocity sensor output signal after offset correction corresponding to the measurement time interval is 0.0016 dps. In this case, the calculation result using equation (20) is as follows.
20 (sec) x 0.0016 (dps) = estimated angular fluctuation value 0.032 (degrees)
That is, the estimated angle fluctuation value calculated by integrating for 20 seconds is 0.032 (degrees).

Graph line 402 in FIG. 13B shows data obtained when the angle fluctuation estimate is obtained by the above-mentioned method based on graph line 401 shown in FIG. 13A. The horizontal axis shows the measurement time interval τ, and the vertical axis shows the angle fluctuation estimate. Both axes are shown in logarithmic scale. As described above, by using the Allan standard deviation, the standard deviation value of the angular velocity sensor output signal after offset correction on the time axis can be evaluated. As shown by graph line 402 in FIG. 13B, the angle fluctuation estimate on the time axis can be calculated. In other words, graph line 402 shows the angle fluctuation estimate over time. By using the curve data shown by graph line 402, the angle fluctuation estimate at the exposure time can be obtained. The data of the angle fluctuation estimate at the exposure time is stored in advance as table data in the storage unit of the imaging device. Therefore, it is possible to predict the amount of image blur based on the shooting conditions set by the photographer.

A processing example will be described with reference to FIG. 14. FIG. 14 is a flowchart illustrating a camera sequence for predicting the amount of image blur and suppressing degradation of image blur correction performance. When the power supply of the imaging device 100 is turned on in S501, the process proceeds to S502. In S502, the offset correction unit 203 executes offset correction processing of the angular velocity sensor.

In S503, the characteristic calculation unit 206 calculates the Allan standard deviation of the angular velocity sensor output based on the output signal of the angular velocity sensor that was offset corrected in S502. Table data of the estimated angular fluctuation value during the exposure time is created. In S504, a shooting instruction determination process is executed. The camera CPU 201 determines whether the photographer has pressed the release button 104 to issue a shooting instruction operation. If a shooting instruction operation is detected, the process proceeds to S505. If a shooting instruction operation is not detected, the process returns to S503 and continues creating the table data.

In S505, the camera CPU 201 determines whether the shooting conditions are such that image stabilization performance is degraded due to an output error of the angular velocity sensor. Specifically, these are shooting conditions with a long exposure time using a tripod or the like (exposure time is longer than a threshold value), or shooting conditions using a super telephoto lens (focal length is greater than a threshold value). These shooting conditions correspond to shooting conditions in which the characteristics of the vibration detection means are dominant. If it is determined in S505 that the shooting conditions are not as described above, the process proceeds to S506 and ends. If it is determined in S505 that the shooting conditions are as described above, the process proceeds to S507.

In S507, the integrated error calculation unit 207 calculates a predicted value of image blur based on the exposure conditions and focal length information set by the photographer. In the next S508, the judgment unit 208 performs judgment processing by comparing the predicted value of image blur with a threshold value. If it is determined that the predicted value is equal to or less than the threshold value, the process proceeds to S506 and ends the operation. On the other hand, if it is determined in S508 that the predicted value is greater than the threshold value, the process proceeds to S509.

In S509, the blurred image prevention processing unit 209 outputs a command to the camera CPU 201 to perform processing to warn that image blur will occur. The camera CPU 201 executes processing to warn of the generation of a blurred image, for example, by a message on the display screen or a sound from a speaker. After processing in S509, the process proceeds to S506 and the operation ends.

In this embodiment, an imaging device can be provided that can make a user aware of a degradation in image stabilization performance caused by an output error of the angular velocity sensor during long exposures, super telephoto shooting, and the like. A prediction table for angular error is created based on the output result of the angular velocity sensor after offset correction. At that time, there may be cases where the time required for offset correction is insufficient and sufficient correction is not possible. In such cases, the prediction table for angular error may be created from the output signal of the angular velocity sensor before offset correction. Although fluctuations due to offset occur in the angular velocity sensor information before offset correction, even when a predicted value of image blur is calculated based on that information, it is possible to process the user to be aware of a degradation in image stabilization performance.

[Modification 1 of the third embodiment]
Modification 1 of the third embodiment will be described with reference to FIG. 15. The main difference from the third embodiment is that signal processing is performed on the angular velocity signal. FIG. 15 is a flowchart for explaining a specific camera sequence for suppressing degradation of image stabilization performance in modification 1. In FIG. 15, step numbers already used for the same processes (S501 to S508) as in FIG. 14 are reused, and detailed descriptions thereof will be omitted. Such an omission method is the same in the modifications described below.

If it is determined in S508 that the predicted value of image blur is greater than the threshold value, the process proceeds to S601. In S601, the blurred image prevention processing unit 209 instructs the camera CPU 201 to execute processing for shifting the time constant of the high pass filter (HPF) to the high frequency side for the output signal of the angular velocity sensor. This processing switches the signal processing method of the vibration detection means. This cuts off the low frequency characteristics that cause offsets, so that it is possible to suppress angular fluctuations caused by output errors of the angular velocity sensor. Note that when the processing for shifting the time constant of the HPF to the high frequency side is performed, low frequency components such as camera shake cannot be detected, but the control of this modified example is performed when shooting under conditions where camera shake is unlikely to occur, such as when the camera is fixed to a tripod. Therefore, there is little harm caused by the processing for shifting the HPF to the high frequency side.

According to this modified example, it is possible to suppress the deterioration of image stabilization performance by changing the signal processing method of the vibration detection means.

[Modification 2 of the third embodiment]
A second modification of the third embodiment will be described with reference to Fig. 16. The main difference from the third embodiment is that image processing is performed on a captured image. Fig. 16 is a flowchart showing a specific camera sequence for suppressing a decrease in image blur correction performance in the second modification.

If it is determined in S508 that the predicted value of image blur is greater than the threshold value, the process proceeds to S701. In this case, it is known in advance that the captured image will be an image with image blur. Therefore, in S701, before the user checks the captured image, the blurred image prevention processing unit 209 instructs the camera CPU 201 to execute image processing to correct the blurred image. Image processing to correct the blurred image includes image processing to increase the sharpness of the image. This operation corresponds to the operation of switching the image processing method.

According to this modified example, it is possible to suppress the deterioration of image stabilization performance by changing the image processing method.

[Modification 3 of the third embodiment]
Modification 3 of the third embodiment will be described with reference to Fig. 17. The main difference from the third embodiment is that a process for changing the exposure time is executed. Fig. 17 is a flowchart showing a specific camera sequence for suppressing a decrease in image blur correction performance in Modification 3.

If it is determined in S508 that the predicted value of image blur is greater than the threshold value, the process proceeds to S801. In S801, the blurred image prevention processing unit 209 instructs the camera CPU 201 to execute automatic setting of the exposure time. Based on the angle error prediction table, a process is executed to automatically set a short exposure time so that the predicted value of image blur is equal to or less than the threshold value, and image capture is performed with the set exposure time. This process corresponds to a process of switching the exposure conditions of the imaging device.

According to this modified example, it is possible to suppress deterioration of image stabilization performance by automatically changing the exposure conditions.

[Fourth embodiment]
A fourth embodiment of the present invention will be described with reference to Fig. 18. An example of an image capture device capable of suppressing degradation of image blur correction performance even when various lens devices are attached to the main body of the image capture device and used in a lens-interchangeable camera system will be described. A process of calculating Allan variance data based on the output of the vibration sensor and the motion vector will be described.

Figure 18 is a block diagram related to the control of the imaging device 100 composed of the main body 101 and the interchangeable lens 102. The interchangeable lens 102 has a lens CPU 1202, electrical contacts 1203, an imaging optical system 213, a lens driving unit 1215, and a vibration sensor 1216.

When the main body 101 and the interchangeable lens 102 are mechanically connected, the camera CPU 1201 and the lens CPU 1202 are electrically connected via the electrical contacts 1203, and are able to communicate with each other. The lens CPU 1202 controls the lens drive unit 1215 according to a signal from the vibration sensor 1216 and instructions from the camera CPU 1201. The lens drive unit 1215 drives the aperture and image blur correction lens of the imaging optical system 213 according to instructions from the lens CPU 1202. The imaging optical system 213 is equipped with a focus lens, zoom lens, shift lens, aperture, etc. on the imaging optical axis 214, and forms an image from the light from the subject.

The vibration sensor 1216 detects the vibration of the interchangeable lens 102, and the detection signal is sent to the lens CPU 1202. The vibration sensor 1216 is an inertial sensor such as an angular velocity sensor. The main body 101 includes a camera CPU 1201, an image sensor 210, a focal plane shutter 212, an image processing unit 1204, a memory unit 1205, a motion vector detection unit 1206, and an Allan variance calculation unit 1207.

The camera CPU 1201 controls the entire system. The camera CPU 1201 controls each component of the interchangeable lens camera system in response to the photographer's operation to issue a shooting instruction using the release button 104, etc.

The focal plane shutter 212 controls the blocking state of light incident on the image sensor 210 based on instructions from the camera CPU 1201. The image sensor 210 performs photoelectric conversion on the light that passes through the image sensor optical system 213 and forms an image, and outputs an electrical signal corresponding to the optical image to the image processor 1204.

The image processing unit 1204 includes an A/D converter, a white balance adjustment circuit, a gamma correction circuit, an interpolation calculation circuit, and the like. The image processing unit 1204 executes predetermined image processing and generates digital data of an image (video) to be stored. The image processing unit 1204 also compresses still image data, moving image data, and audio data using known methods.

The storage unit 1205 includes a semiconductor storage device that stores programs executed by the camera CPU 1201, a memory card that stores various data generated by the image processing unit 1204, etc. The semiconductor storage device is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory).

The motion vector detection unit 1206 detects motion vectors from multiple image signals captured at different times and processed by the image processing unit 1204. In detecting a motion vector, image movement can be detected by comparing feature points of two temporally consecutive images. The method of detecting a motion vector is a well-known technique, and therefore a detailed description thereof will be omitted.

The Allan variance calculation unit 1207 calculates the difference between the detection signal of the vibration sensor 1216 and the detection signal of the motion vector detection unit 1206, and calculates the Allan variance using the difference. At this time, the detection signal of the vibration sensor 1216 and the detection signal of the motion vector detection unit 1206 are each converted to an angular velocity and then the difference calculation is performed. When converting the motion vector to an angular velocity, information such as the focal length of the interchangeable lens 102, the shooting interval (or frame rate) of the image used to detect the motion vector, and the pixel pitch of the image sensor 210 is used. The Allan variance data calculated by the Allan variance calculation unit 1207 can be referenced by the camera CPU 1201.

Let us assume that the imaging device 100 is not fixed to a tripod or the like, for example, when shaking occurs in the imaging device 100 while preparing for handheld shooting. In this case, by removing the signal of the shaking of the imaging device 100 detected by the motion vector detection unit 1206 from the detection signal of the vibration sensor 1216, it becomes possible to detect the noise and drift inherent to the vibration sensor 1216. The camera CPU 1201 can more accurately perform signal processing to correct the offset of the shake detection signal and suppress drift.

In this embodiment, even after the interchangeable lens 102 is attached to the main body 101, the camera CPU 1201 can obtain Allan variance data specific to the vibration sensor 1216 from the Allan variance calculation unit 1207. Therefore, it is possible to provide an imaging device that can suppress a decrease in image blur correction performance when various lens devices are attached to the main body. Note that while a lens device is shown as an example of an external device that can be attached to the main body of the imaging device, the present invention is applicable to a system in which an external device such as an accessory having a vibration sensor can be attached to the main body.

[Fifth embodiment]
An image pickup apparatus 300 according to a fifth embodiment of the present invention will be described with reference to Fig. 19. Differences from the image pickup apparatus 100 according to the fourth embodiment will be mainly described. The difference between Fig. 19 and Fig. 18 is that a correction unit 211 and a correction control unit 1208 are provided. In addition, the main body 101 is provided with a first vibration sensor 1209, and the interchangeable lens 102 is provided with a second vibration sensor 1216. That is, the vibration sensor 1209 detects vibration of the main body 101 and outputs a detection signal to the camera CPU 1201. The vibration sensor 1216 detects vibration of the interchangeable lens 102 and outputs a detection signal to the lens CPU 1202.

The correction control unit 1208 controls the correction unit 211 in accordance with a control command from the camera CPU 1201, and the correction unit 211 drives the image sensor 210. The drive mechanism of the image sensor 210 is configured to translate the image sensor 210 in a plane perpendicular to the image sensor optical axis 214, and to rotate the image sensor 210 around the image sensor optical axis 214. The correction control unit 1208 sends a control command to the correction unit 211 to drive the image sensor 210 so as to cancel out the vibrations detected by the vibration sensor 1209.

The Allan variance calculation unit 1307 determines whether the magnitude of the detection signal of the vibration sensor 1209 is equal to or less than a predetermined threshold. For example, the predetermined threshold is set to a threshold for the detection signal of the vibration sensor 1209 used when determining that the imaging device 300 is supported on a tripod. If the magnitude of the detection signal of the vibration sensor 1209 is equal to or less than the threshold, the Allan variance calculation unit 1307 calculates the Allan variance using the detection signal of the vibration sensor 1216. Note that, as a method for determining whether the imaging device is supported on a tripod using the detection signal of the vibration sensor, there is a method for determining whether a state in which the magnitude of the detection signal of the vibration sensor is smaller than a predetermined threshold continues for a predetermined threshold time or more. This technology is well known, so a detailed description will be omitted. The same applies to the determination of whether the imaging device is placed on a desk or the like.

Assume that the imaging device 300 is in a static state after the interchangeable lens 102 is attached to the main body 101. A static state is a state in which there is almost no shaking, such as when the imaging device is attached to a tripod or placed on a desk. In such a case, it is possible to obtain the detection signal of the vibration sensor 1216 and calculate the Allan variance. In other words, since it is possible to calculate the Allan variance when the main body 101 of the imaging device 300 is not vibrating, it becomes possible to detect the inherent noise and drift of the vibration sensor 1216. The lens CPU 1202 can more accurately perform offset correction of the shake detection signal and signal processing to suppress drift.

According to this embodiment, it is possible to suppress deterioration of image blur correction performance in an imaging system in which various lens devices can be attached to the main body of the imaging device. In this embodiment, an example has been shown in which the detection signal of the vibration sensor 1209, which is originally used to control the correction unit 211, is used to determine the static state of the imaging device 300. However, without being limited to this, the static state of the imaging device 300 may also be determined using the detection signal of another vibration sensor provided in the main body 101.

According to the above embodiment, offset correction can be performed by appropriately setting filter processing parameter values based on statistical data to account for fluctuations in each individual inertial sensor, thereby suppressing variation in the image stabilization effect of the imaging device.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the present invention. The present invention can be implemented in a configuration in which the above-mentioned embodiments and modifications are appropriately combined. In addition, in the above-mentioned embodiment, an example was shown in which a data table of Allan variance is used as statistical data showing the relationship of the angle error to the measurement time interval of the shake detection output. However, the present invention is not limited to this, and can be applied to embodiments that utilize various types of statistical processing.

5 Camera control unit 7 Image processing unit 14 Image blur correction unit 15 Shake detection unit 5002 Offset estimation unit 5100 Storage unit

Claims (19)

An image stabilization device for an imaging device that performs image stabilization by acquiring a detection signal of shake from a detection unit that detects shake in a main body of the imaging device having an imaging unit or in an external device that can be attached to the main body, the image stabilization device comprising:
an image blur correction unit that performs optical image blur correction by moving the image capture unit or the correction lens in a direction different from the direction of the photographing optical axis, or performs electronic image blur correction by performing image processing on the image captured by the image capture unit;
an image processing means for calculating a motion vector between a plurality of images captured by the imaging means and calculating an amount of shake in the imaging device based on the motion vector;
an estimation means for estimating an offset component as a fluctuation including a direct current component contained in the shake detection signal by performing a filtering process using a Kalman filter on a signal obtained by removing the shake amount from the shake detection signal;
a control unit that controls the image blur correction unit based on a signal obtained by subtracting the offset component from the shake detection signal ,
the estimation means changes parameter values of the Kalman filter using statistical data indicating a relationship between an angular error and a measurement time interval of the output of the detection means. 2. The image blur correction device according to claim 1, further comprising an integrating means for obtaining an output correction value of said detecting means by performing time integration on a signal obtained by subtracting the offset component from the shake detection signal, and said control means controls said image blur correction means by using said output correction value . 3. The image stabilization apparatus according to claim 1, wherein the estimation means changes the parameter value by using data on a standard deviation related to the offset component as the statistical data. 4. The image stabilization apparatus according to claim 2, wherein the parameter value is a variance value of observation noise and a variance value of system noise. an update means for acquiring the shake detection signal and the motion vector and updating the statistical data;
5. The image stabilization apparatus according to claim 1, wherein the estimation means obtains updated statistical data and changes the parameter value. a storage means for storing data on the variance of the observation noise and the variance of the system noise;
5. The image stabilization device according to claim 4, further comprising an update unit that acquires the shake detection signal and the motion vector and updates the variance value of the system noise stored in the storage unit. 7. The image stabilization device according to claim 6, wherein the updater updates the variance value of the system noise when a difference between the variance value of the system noise currently acquired and the variance value of the system noise previously acquired is greater than a threshold value. 8. The image stabilization device according to claim 1, wherein the estimation means changes the parameter value by using the statistical data generated from the detection signal acquired in a stationary state. An imaging apparatus comprising the image blur correction device according to claim 1 . a calculation means for calculating a noise characteristic related to a signal in which the offset component is suppressed with respect to the vibration detection signal ;
a calculation means for generating data indicating a relationship between an integrated error and an integral time for the shake detection signal by using the noise characteristic calculated by the calculation means;
and a processing means for acquiring the data generated by the calculation means, information on an exposure time of an image sensor and a focal length of an image capturing optical system, and executing a process for suppressing or preventing generation of a blurred image when the integrated error is equal to or greater than a threshold value. The imaging device according to claim 10, characterized in that the processing means executes one or more of a process of issuing a warning about generation of a blurred image, a process of switching a signal processing method for the shake detection signal, a process of switching an image processing method for a captured image, and a process of switching exposure conditions. 12. The imaging apparatus according to claim 10, wherein the calculation means calculates a standard deviation related to the shake detection signal to generate table data. 13. The imaging apparatus according to claim 10, wherein the processing means executes the processing when a predetermined shooting condition is satisfied and the integrated error is equal to or greater than a threshold value. The imaging device according to claim 13 , wherein the calculation means predicts an image blur of the captured image from information on an exposure time and a focal length when the photographing condition is satisfied. 15. The imaging device according to claim 13, wherein the photographing condition is a photographing condition in which the exposure time is longer than a threshold value, or a photographing condition in which the focal length is greater than a threshold value. The detection means includes an angular velocity sensor,
16. The imaging apparatus according to claim 10, wherein the calculation means calculates data representing a fluctuation in angle by multiplying a standard deviation value corresponding to a measurement time interval of the shake detection signal by the measurement time interval. An external device having a detection means for detecting vibration can be attached to the main body,
10. The imaging device according to claim 9, further comprising a calculation means for calculating a difference between a detection signal of a detection means included in the external device and the motion vector when the external device is attached to the main body, to calculate the statistical data. An external device can be attached to the main body,
a first detection means for detecting vibration of the main body portion;
and a calculation means for calculating the statistical data using a detection signal of a second detection means included in the external device when the external device is attached to the main body and a magnitude of the detection signal of the first detection means is equal to or smaller than a threshold value. A control method executed by an image stabilization device of an imaging device equipped with an imaging means, the image stabilization device comprising an image stabilization means for performing optical image stabilization by moving the imaging means or a correction lens in a direction different from a direction of an optical axis of an image capture, or electronic image stabilization by performing image processing on an image captured by the imaging means, the method comprising:
an image processing step of calculating a motion vector between a plurality of images captured by the imaging means and calculating an amount of shake in the imaging device based on the motion vector;
an estimation step of estimating an offset component as a fluctuation including a DC component contained in the shake detection signal by performing a filtering process using a Kalman filter on a signal obtained by removing the shake amount from the shake detection signal;
a control step of controlling the image blur correction means based on a signal obtained by subtracting the offset component from the shake detection signal ,
a process of changing a parameter value of the Kalman filter using statistical data indicating a relationship between an angle error and a measurement time interval of the output of the detection means, the process comprising: changing a parameter value of the Kalman filter using the statistical data indicating a relationship between an angle error and a measurement time interval of the output of the detection means;

Images