JP2008118570A - Shake correcting device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suitably correct image deterioration due to a pixel defect of an imaging element even when performing electronic shake correction. <P>SOLUTION: Disclosed is a shake correcting device which superposes a plurality of images obtained by the imaging element (12) having a plurality of pixels one over another into one image, the blur correcting device having a position shift detecting circuit (80) which detects position shift quantities among the plurality of images to be put together, and an image composing circuit (82) which adds data of respective pixels of the plurality of images to be put together while positioning them based upon the position shift quantities detected by the position shift detecting circuit, and also corrects data of pixels of one image by using data of corresponding pixels of other images when the former data are output from a defect pixel of the imaging element. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a shake correction apparatus and method, and more particularly, a shake correction apparatus having a function of correcting image quality degradation caused by an imaging output from a defective pixel included in a solid-state imaging device typified by a CCD or CMOS sensor, and Regarding the method.

  Conventionally, an electronic camera shake correction technique has been proposed as a camera shake correction technique for an electronic still camera (see, for example, Patent Document 1). In the electronic camera shake correction technology, a plurality of images continuously shot with short-time exposure that does not cause camera shake are superimposed and combined while aligning positions to obtain a camera shake corrected image with a desired exposure time.

  In such electronic image stabilization technology, image degradation caused by pixel defects in the image sensor must be corrected correctly because the degradation positions in the multiple images to be combined are shifted from each other in order to correct after image alignment. I could not. In Patent Document 1, white scratches due to pixel defects tend to be noticeable in long-time exposure, but may function as normal pixels in short-time exposure. Therefore, when the above-described electronic camera shake correction is performed. Says that white scratches can be reduced without correction.

JP 2000-69352 A

  In this way, at the time of electronic camera shake correction, appropriate correction is not performed for image degradation due to pixel defects of the image sensor, or correction is not performed as described in Patent Document 1, so that high-quality is achieved. An image could not be obtained.

  The present invention has been made in view of the above problems, and an object thereof is to appropriately correct image deterioration caused by pixel defects of an image sensor even when electronic camera shake correction is performed.

  In order to achieve the above object, the blur correction apparatus of the present invention that superimposes and combines a plurality of images obtained by an imaging means including a plurality of pixels on a single image is a positional shift between the plurality of images to be combined. A detection unit for detecting the amount and the data of each pixel of the plurality of images to be combined are added while being aligned based on the amount of displacement detected by the detection unit, and the pixel data of one image is defective When the data is output from the pixel, the image forming apparatus includes a combining unit that performs correction using the data of the corresponding pixel of another image.

  In addition, a shake correction method for superimposing and combining a plurality of images obtained by an imaging unit including a plurality of pixels on a single image includes a detection step of detecting a positional deviation amount between a plurality of images to be combined, When the data of each pixel of a plurality of images to be combined is added while being aligned based on the amount of positional deviation detected in the detection step, and the pixel data of one image is data output from a defective pixel And a combining step of performing correction using data of corresponding pixels of another image.

  According to another configuration of the invention, there is provided a shake correction apparatus that superimposes and combines a plurality of images obtained by an imaging unit including a plurality of pixels on a single image. A storage unit that stores position information; a correction unit that corrects data output from the defective pixel using data of peripheral pixels based on the position information of the defective pixel stored in the storage unit; and a composition Detecting means for detecting a positional deviation amount between a plurality of images, and aligning data of each pixel of the plurality of synthesized images corrected by the correcting means based on the positional deviation amount detected by the detecting means. And adding means for adding.

  In addition, a shake correction method for superimposing and combining a plurality of images obtained by an imaging unit including a plurality of pixels on a single image includes: a storage unit storing position information of defective pixels of the imaging unit; An acquisition step of acquiring position information, a correction step of correcting data output from the defective pixel based on data of peripheral pixels based on the position information acquired in the acquisition step, and between a plurality of images to be combined A detection step of detecting the amount of misalignment, and a composition for adding the pixel data of the plurality of images to be synthesized corrected in the correction step while aligning them based on the amount of misalignment detected in the detection step Process.

  According to the present invention, even when electronic camera shake correction is performed, it is possible to appropriately correct image degradation caused by pixel defects in the image sensor.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus such as an electronic still camera according to the first embodiment of the present invention. This image processing apparatus has an electronic camera shake correction function for performing camera shake correction by synthesizing a plurality of continuous images taken with short-time exposure.

  In FIG. 1, reference numeral 100 denotes an image processing apparatus. Reference numeral 10 denotes an image pickup lens for forming an object optical image on an image sensor 12 described later, and 11 denotes a shutter for adjusting the exposure of the object optical image. Reference numeral 12 denotes an image sensor that photoelectrically converts an object optical image into an analog electric signal and outputs it, and reference numeral 14 denotes an A / D converter that converts an analog signal output from the image sensor 12 into a digital signal.

  Reference numeral 16 denotes a D / A converter, and 18 denotes an image display unit comprising a TFT LCD or the like. The display image data written in the memory 40 is converted from digital data to analog data via the D / A converter 16 and displayed by the image display unit 18. Further, an electronic viewfinder (EVF) function can be realized by continuously reading out image signals from the image sensor 12 and sequentially displaying them on the image display unit 18.

  Reference numeral 20 denotes a recording medium such as a memory card or a hard disk, and photographed image data and the like are recorded on the recording medium 20.

  Reference numeral 30 denotes an image processing circuit, which performs predetermined image processing such as development processing such as pixel interpolation processing and color conversion processing, and resizing processing on captured or recorded image data.

  Reference numeral 40 denotes a memory for expanding the data stored in the ROM 94 and temporarily storing captured image data. The memory 40 has a recording capacity sufficient to store a predetermined number of still images and a predetermined time of moving images. ing. For example, the image data output from the A / D converter 14 is written into the memory 40 via the memory control circuit 50, the image processing circuit 30, the compression / decompression circuit 60, or directly via the memory control circuit 50.

  A memory control circuit 50 includes an A / D converter 14, a D / A converter 16, a recording medium 20, an image processing circuit 30, a memory 40, a compression / decompression circuit 60, a flaw mark circuit 70, a flaw correction circuit 72, a position. The data flow to the deviation detection circuit 80 and the image composition circuit 82 is controlled.

  A compression / decompression circuit 60 compresses / decompresses image data.

  70 is a flaw mark circuit that reads out defective pixel position information of the image sensor 12 from the memory 40 via the memory control circuit 50 and replaces the pixel value at the defective pixel position with a predetermined mark value, and 72 corrects the corresponding pixel from neighboring pixels. This is a scratch correction circuit.

  Reference numeral 80 denotes a misregistration detection circuit for detecting the misregistration amount between two images, and 82 denotes an image composition circuit for synthesizing two images.

  Reference numeral 92 denotes an operation unit such as a mode dial switch, which can switch and set various function modes such as power-off, shooting mode, and playback mode. Reference numeral 90 denotes a system control circuit that controls the operation of the entire image processing apparatus 100 and each circuit constituting the image processing apparatus 100 in accordance with the setting by the operation unit 92 and the contents stored in the ROM 94 (read only memory). The ROM 94 stores in advance programs used by the system control circuit 50, flaw data, and the like.

  FIG. 2 is a diagram showing a schematic configuration of the flaw correction circuit 72. Reference numerals 402, 404, and 406 denote flip-flops, 408 denotes an adder, 410 denotes a data selector, and 412 denotes a flaw mark detection circuit. If it is not a flaw mark, the input pixel data is output as it is, and if a flaw mark is detected, the average value of the left and right pixel data is output as a correction value. Note that the method of correcting the flaw mark pixel in the flaw correction circuit 72 of the present invention is not limited to this, and the flaw mark pixel is corrected with reference to a plurality of neighboring pixels such as upper and lower pixels and diagonal pixels. Also good.

● Scratch pixel data correction without electronic image stabilization.
At the time of normal photographing (when electronic camera shake correction is not performed), first, the image data output from the A / D converter 14 is detected by the scratch mark circuit 70 based on the defective pixel position information stored in the ROM 94 in advance. A flaw mark is attached to the data output from the pixel. The flaw correction circuit 72 detects a flaw mark from the output of the flaw mark circuit 70 and corrects data output from the defective pixel (hereinafter referred to as “flaw pixel data”). However, in the present invention, the flaw data correction processing during normal photographing is not limited to this, and the flaw correction circuit 72 may directly acquire defective pixel position information from the ROM 94 and detect and correct flaw pixel data. Absent.

[Scratch Pixel Data Correction During Electronic Camera Shake Correction] When electronic camera shake correction is performed, in the first embodiment, two images of an image obtained by adding a scratch mark to the output image of the A / D converter 14 by the scratch mark circuit 70 are used. A positional shift between images is detected and images are synthesized. However, the present invention is not limited to this, and the order of processing for marking a flaw mark and detecting misalignment is not limited. In addition, the detection method for misregistration is not particularly limited in the present invention, and may be configured to detect using a known method.

  FIG. 3 is a diagram showing a schematic configuration of the image composition circuit 82 in the first embodiment. X and Y indicate two pieces of input image data to be combined, and Z indicates the combined image data. 202 and 204 are flaw mark detection circuits, 206 and 208 are data selectors, 203 and 205 are multipliers for doubling input image data, and 210 is an adder. The image data X and Y are input in a state where the position shift is canceled by the memory control circuit 50 based on the detection result of the position shift detection circuit 80.

  FIG. 4 is a diagram for explaining an image composition method at the time of electronic camera shake correction in the first embodiment. In order to make the explanation easy to understand, 5 × 5 pixels are shown as one image, but in actuality, they are composed of a larger number of pixels. The example shown in FIG. 4 shows a case where the misalignment detection circuit 80 detects misalignment and synthesizes eight images by aligning the positions. The shaded pixels are marked with scratch marks. A pixel is shown. In order to perform alignment so as to cancel out the displacement, as shown in FIG. 4A, the positions of pixels corresponding to the same defective pixel of the image sensor 12 are different for each image. Further, it is assumed that there is no pixel having a pixel value of 0 in the image data, and the pixel value of the scratch mark is 0.

  First, the first image P (0) and the second image P (1) are synthesized. At this time, the image data of each image is input to the image composition circuit 82 shown in FIG. 3 for each pixel. The flaw mark detection circuits 202 and 204 determine whether or not a flaw mark is attached to the image data of each pixel (hereinafter referred to as “pixel data”). If there is no flaw mark, the flaw mark detection circuits 202 and 204 control the data selectors 208 and 206 to select the terminal “0”, and if there is a flaw mark, the terminal “1” is set. Control to select.

  By this control, for example, a flaw mark is detected in the Y pixel data, and if no flaw mark is detected in the X pixel data, the X pixel data is doubled and output from the image composition circuit 82. The reverse is true when a scratch mark is detected only in the X pixel data. In addition, when a scratch mark is detected in the pixel data of both pixels, the pixel value 0 of the scratch mark is output, so that it remains as a scratch mark.

  If there is no flaw mark, the corresponding pixel data is added by the above synthesis process. If there is a flaw mark, pixel data without a flaw mark is doubled and output. When the pixel data of the first image P (0) and the second image P (1) are sequentially added in this way, the image P (0, 1) in FIG. The image data has various scratch marks. Similarly, the third image P (2), the fourth image P (3), the fifth image P (4), the sixth image P (5), and the seventh image P ( The pixel data of 6) and the eighth image P (7) are sequentially added. As a result, four images having scratch marks as shown in FIG. 4B are obtained. In the example shown in FIG. 4, at this time, most of the pixel data with the flaw mark is corrected by the pixel data without the flaw mark.

  Further, the process of adding each pixel data described above is repeated for the four images shown in FIG. 4B, and two images shown in FIG. 4C are acquired. Further, when the above-described processing is repeated for these two images, one image P (0 to 6) shown in FIG. 4D can be acquired.

  In this manner, the flawed pixel data can be corrected by correcting the pixel data with the flaw mark using the aligned pixel data to be combined. Although the number of flaw pixel data in the image synthesized by this processing can be reduced, the flaw pixel data finally remaining is corrected by the flaw correction circuit 72 using surrounding pixels.

Next, a more specific control example for realizing flaw pixel data correction at the time of correcting the electronic camera shake will be described with reference to FIGS. Here, as a result of continuously shooting 16 (= 2 ⁴ ) images at a shutter speed of 1/128 seconds and synthesizing these images, a camera shake-corrected image corresponding to a shutter speed equivalent to 1/8 second is obtained. And

FIG. 5 is a flowchart for explaining the control by the system control circuit 90, and FIG. 6 is a table showing changes in variables in the flowchart of FIG. In FIG. 5, P (i) indicates an image to be synthesized, and i indicates an image number. Here, since 16 sheets are continuously shot, i = 0 to 15. Also, a constant M, which will be described later, is set to a value obtained by subtracting 1 from the index when the number of images to be combined is expressed as a power of 2 (16 = 2 ⁴ , M = 4-1 = 3).

First, variables m and n are initialized to 0 (steps S1 and S2). Next, the positional deviation detection circuit 80 detects the positional deviation amount of the two images P (2n ^{m + 1} ) and P (2n ^{m + 1} +2 ^m ) to be combined (step S4). Since m = n = 0 here, the amount of displacement between the images P (0) and P (1) is detected (see also the table in FIG. 6). Next, the pixel data of the images P (0) and P (1) are combined by the image combining circuit 82 while being aligned based on the amount of displacement, and the images are combined and replaced with P (0) (step) S6). In the image composition processing by the image composition circuit 82, as described above, since the flawed pixel data with the flaw mark is synthesized while being corrected, the number of the uncorrected flawed pixel data is larger than that of the image before the synthesis. Will be reduced.

Thereafter, after increasing n by 1 (step S8), until n = 2 ^(Mm) = 8 (during YES in step S10), the process returns to step S4 and the above process is repeated. Since m = 0 and n = 1 in the second loop, the third and fourth images P (2) and P are derived from P (2n ^{m + 1} ) and P (2n ^{m + 1} +2 ^m ). The image of (3) is synthesized and replaced with the image P (2).

  Two columns of m = 0 in FIG. 6 indicate image numbers synthesized in a loop of n = 1-7.

  When n = 8 (NO in step S10), the process proceeds to step S12, m is increased by 1, and compared with M. If M or less (during NO in step S14), the process returns to step S2 and the above processing is repeated. .

  Here, since m = 1, an image having an image number indicated by two columns of m = 1 in FIG. 6 is synthesized. That is, the images P (0) and P (2), P (4) and P (6), P (8) and P (10), P (12) and P (14) synthesized by the process of m = 0. ) Are respectively synthesized and replaced with P (0), P (4), P (8), and P (14), respectively. This process further corrects the damaged pixel data.

  The above processing is repeated for m = 2 and m = 3, the images shown in the two columns m = 2 and m = 3 in FIG. 6 are combined, and finally the image P (16) in which the 16 images are combined. 0) is obtained.

  Then, the scratch mark remaining after the synthesis of all the images is corrected by the scratch correction circuit 72 (step S16), and the process ends.

  The control shown in FIG. 5 and FIG. 6 is a specific example for realizing the method of FIG. 4. In this example, the algorithm using the original image number is used, but the present invention is not limited to this. For example, any algorithm that realizes the method shown in FIG. 4 may be used, such as re-assigning the number of the image after the composition sequentially from 1 and repeating the image composition processing until the last image is obtained.

  As described above, according to the first embodiment, it is possible to appropriately correct the image deterioration caused by the pixel defect of the image sensor while performing the electronic camera shake correction.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. In the flaw pixel data correction method at the time of electronic camera shake correction according to the first embodiment, flaw pixel data can be appropriately corrected when images of a power of 2 are captured and combined. However, when the number is not a power of 2, the corrected data does not have an appropriate value as will be described later.

  Therefore, in the second embodiment, an image composition circuit and method capable of correcting flaw pixel data with appropriate weighting regardless of the number of images to be synthesized will be described. Since the configuration other than the detailed configuration of the image composition circuit 82 is the same as the configuration described in the first embodiment, the description is omitted. Hereinafter, the configuration and method of the image composition circuit 82 characteristic of the second embodiment, which is performed instead of the processing described with reference to FIGS. 4, 5, and 6 of the first embodiment, will be described. To do.

FIG. 7 is a diagram illustrating a schematic configuration of the image composition circuit 82 in the second embodiment. X and Y indicate two pieces of input image data to be combined, and Z indicates the combined image data. X is image data having a smaller image number, and Y is image data having a larger image number. 302 and 304 are flaw mark detection circuits, 306 and 308 are remainder units, 314 and 316 are data selectors, and 318 is an adder. The image data X and Y are input in a state where the position shift is canceled by the memory control circuit 50 based on the detection result of the position shift detection circuit 80.

  FIG. 8 is a diagram for explaining an image composition method at the time of electronic camera shake correction in the second embodiment. In order to make the explanation easy to understand, 5 × 5 pixels are shown as one image, but in actuality, they are composed of a larger number of pixels. The example shown in FIG. 8 shows a case where the positional deviation detection circuit 80 detects positional deviation and combines six images with the same position. The shaded pixels are marked with scratch marks. A pixel is shown. In order to perform alignment so as to cancel out the positional deviation, as shown in FIG. 8A, the positions of pixels corresponding to the same defective pixel of the image sensor 12 are different for each image. Further, it is assumed that there is no pixel having a pixel value of 0 in the image data, and the pixel value of the scratch mark is 0.

  First, the first image P (0) and the second image P (1) are synthesized. In the case of the first and second images, each pixel data is pixel data of one image. In this case, when any one of the pixel data is marked with a flaw mark, the other pixel data may be doubled and output, so the coefficients of the remainders 306 and 308 are set to A = B = 2. Each pixel data is input to the image composition circuit 82 shown in FIG. The flaw mark detection circuits 302 and 304 determine whether or not each pixel data has a flaw mark. If there is no flaw mark, the flaw mark detection circuits 302 and 304 control the data selectors 314 and 316 to select the terminal “0”, and if there is a flaw mark, the terminal “1” is set. Control to select.

  By this control, for example, a flaw mark is detected in the Y pixel data, and if no flaw mark is detected in the X pixel data, the X pixel data is doubled and output from the image composition circuit 82. The reverse is true when a scratch mark is detected only in the X pixel data. In addition, when a scratch mark is detected in the pixel data of both pixels, the pixel value 0 of the scratch mark is output, so that it remains as a scratch mark.

  If there is no flaw mark, the corresponding pixel data is added by the above synthesis process. If there is a flaw mark, pixel data without a flaw mark is doubled and output. When the pixel data of the first image P (0) and the second image P (1) are sequentially added in this way, the image P (0, 1) in FIG. The image data has various scratch marks. Similarly, the pixel data of the third image P (2), the fourth image P (3), the fifth image P (4), and the sixth image P (5) are sequentially added. To do. As a result, three images having scratch marks as shown in FIG. 8B are obtained.

  Further, the process of adding each pixel data described above is repeated for the three images shown in FIG. In this case, each pixel data of the first and second images has pixel data of two images, and thus becomes pixel data of four images by synthesis. In this case, when any one of the pixel data is marked with a flaw mark, the other pixel data may be doubled and output, so the coefficients of the remainders 306 and 308 are set to A = B = 2. Each pixel data is input to the image composition circuit 82 shown in FIG. 3 for each pixel and synthesized. Here, the third image (P4, 5) in FIG. 8B does not have a fourth image and cannot be combined at this stage, and therefore is not combined. As a result, two images shown in FIG. 8C are obtained.

  Further, the above-described processing is repeated for the two images. The first image P (0 to 3) is a composite image of four images, whereas the second image P (4, 4). 5) is an image obtained by combining two images. In this case, if the coefficient of the remainder unit is fixed to 2 as in the first embodiment and the synthesis process is performed, the flawed pixel data cannot be corrected correctly. For example, when a scratch mark is attached only to the pixel data of P (0 to 3), if P (4, 5) is doubled and output, it corresponds to four (= 2 × 2) images. Level pixel data is output. On the other hand, if a scratch mark is attached only to the pixel data of P (4, 5), if P (0-3) is doubled and output, it corresponds to an image of 8 sheets (= 2 × 4). Pixel data at a level to be output.

  Accordingly, the coefficients A and B are set as follows. That is, when the images P (0 to 3) and P (4, 5) are combined, pixel data of a level corresponding to six images should be obtained. Therefore, after the coefficients of the remainder units 306 and 308 are set to A = 6/4 = 3/2 and B = 6/2 = 3, the above-described image composition processing is performed.

  By setting the coefficient in this way, for example, when a scratch mark is detected only in the Y image data, the X pixel data is multiplied by 3/2 and output. On the other hand, if a flaw mark is detected only in the X image data, the Y pixel data is tripled and output. In addition, when a scratch mark is detected in the pixel data of both pixels, the pixel value 0 of the scratch mark is output, so that it remains as a scratch mark.

  Thereby, one image P (0 to 5) shown in FIG. 8D obtained by synthesizing six images can be obtained.

  In this manner, the flawed pixel data can be corrected by correcting the pixel data with the flaw mark using the aligned pixel data to be combined. Although the number of flaw pixel data in the image synthesized by this processing can be reduced, the flaw pixel data finally remaining is corrected by the flaw correction circuit 72 using surrounding pixels.

  Thus, prior to image synthesis, the number of images synthesized to obtain each image to be synthesized is used as the denominator (one for unsynthesized images), and the number of images synthesized to obtain each image. Are set as the coefficients of the remainders 306 and 308.

  Next, a more specific control example for realizing flaw pixel data correction at the time of electronic camera shake correction will be described with reference to FIG. 9 and FIG. Here, 14 images are continuously shot at a shutter speed of 1/140 seconds, and as a result of combining these images, a camera shake-corrected image corresponding to a shutter speed equivalent to 1/10 seconds is obtained.

FIG. 9 is a flowchart for explaining control by the system control circuit 90 in the second embodiment, and FIG. 10 is a table showing changes in variables in the flowchart of FIG. In FIG. 9, P (i) indicates an image to be synthesized, and i indicates an image number. Here, since 14 (2 ³ <14 <2 ⁴ ) continuous shooting, i = 0 to 13 is set. Further, the constant M is a value that exceeds the number of images to be synthesized among values that can be expressed by powers of 2, and is obtained by subtracting 1 from an index that is closest to the number of images (M = 4-1 = 1 = Set to 3).

First, variables m and n are initialized to 0 (steps S101 and S102). Next, the positional deviation detection circuit 80 detects the positional deviation amount of the two images P (2n ^{m + 1} ) and P (2n ^{m + 1} +2 ^m ) to be combined (step S104). Since m = n = 0 here, the amount of displacement between the images P (0) and P (1) is detected (see also the table in FIG. 10). Next, coefficients A and B are set as described above (step S110). Here, A = B = 2/1 = 2 is set.

  Next, the pixel data of the images P (0) and P (1) are combined by the image combining circuit 82 while being aligned based on the amount of displacement, and the images are combined and replaced with P (0) (step) S112).

Thereafter, after increasing n by 1 (step S114), it is determined whether n <2 ^(Mm) = 8 (step S116). If n <2 ^(Mm) = 8, is there a pair of images to be combined? It is determined whether or not (step S118). If there is a pair of images to be combined, the process returns to step S104 and the above is repeated. Since m = 0 and n = 1 in the second loop, the third and fourth images P (2) and P are derived from P (2n ^{m + 1} ) and P (2n ^{m + 1} +2 ^m ). The image of (3) is synthesized and replaced with the image P (2).

  Two columns of m = 0 in FIG. 10 indicate image numbers synthesized in a loop of n = 1 to 6.

  If n = 8 (NO in step S116) or there are no more pairs of images to be combined (NO in step S118), the process proceeds to step S120. Then, m is incremented by 1, and compared with M. If it is equal to or less than M (during NO in step S122), the process returns to step S102 and the above process is repeated.

  Here, since m = 1, an image having an image number indicated by two columns of m = 1 in FIG. 10 is synthesized. That is, the images P (0) and P (2), P (4) and P (6), P (8) and P (10) synthesized by the process of m = 0 are respectively synthesized, and P (0 ), P (4), and P (8). This process further corrects the damaged pixel data. However, since there is no paired image for P (12), the composition process is not performed and it remains as it is.

  Next, m = 2, and an image having an image number indicated by two columns of m = 2 in FIG. 10 is synthesized. That is, the images P (0) and P (4), P (8) and P (12) synthesized by the process of m = 1 are respectively synthesized and replaced with P (0) and P (8), respectively. Here, since the number of images synthesized so far is different between P (8) and P (12) (4 and 2), the coefficients of the remainders 306 and 308 are A = 6/4 = 3 / 2, B = 6/2 = 3.

  Further, when m = 3, the images P (0) and P (8) having the image numbers indicated by the two columns of m = 3 in FIG. 10 are combined. At this time, there are 8 images synthesized so far as the image P (0), whereas there are 6 images synthesized so far as P (8). The coefficients are A = 14/8 = 7/4 and B = 14/6 = 7/3, respectively.

  In this way, an image P (0) in which 14 images are finally combined is obtained.

  Then, the scratch mark remaining after the synthesis of all the images is corrected by the scratch correction circuit 72 (step S124), and the process is terminated.

  The control shown in FIGS. 9 and 10 is a specific example for realizing the method of FIG. 8. In this example, the algorithm using the original image number is used, but the present invention is not limited to this. For example, any algorithm that realizes the method shown in FIG. 8 may be used, such as re-assigning the number of the image after the composition sequentially from 1 and repeating the image composition processing until the last image is obtained.

  As described above, according to the second embodiment, it is possible to appropriately correct the image deterioration caused by the pixel defect of the image sensor while performing the electronic camera shake correction.

<Third Embodiment>
Next, a third embodiment of the present invention will be described. Since the configuration of the image processing apparatus according to the third embodiment is the same as that of the second embodiment, description thereof is omitted. The third embodiment and the second embodiment differ in the order in which the images are combined. Hereinafter, an image composition method according to the third embodiment will be described.

Blemish Pixel Data Correction During Electronic Camera Shake Correction FIG. 11 is a diagram for explaining an image composition method during electronic camera shake correction in the third embodiment. In order to make the explanation easy to understand, 5 × 5 pixels are shown as one image, but in actuality, they are composed of a larger number of pixels. The example shown in FIG. 11 shows a case where the position shift detection circuit 80 detects the position shift and combines five images to synthesize the positions. The shaded pixels are pixels marked with a flaw mark. Is shown. Since alignment is performed so as to cancel out the positional deviation, as shown in FIG. 11A, the positions of pixels corresponding to the same defective pixel of the image sensor 12 are different for each image. Further, it is assumed that there is no pixel having a pixel value of 0 in the image data, and the pixel value of the scratch mark is 0.

  First, the first image P (0) and the second image P (1) are synthesized. In the case of the first and second images, each pixel data is pixel data of one image. Also in the third embodiment, the coefficients of the surplus units 306 and 308 are set to A = B = 2/1, that is, 2 in the same way as in the second embodiment. Each pixel data is input to the image composition circuit 82 shown in FIG. The flaw mark detection circuits 302 and 304 determine whether or not each pixel data has a flaw mark. If there is no flaw mark, the flaw mark detection circuits 302 and 304 control the data selectors 314 and 316 to select the terminal “0”, and if there is a flaw mark, the terminal “1” is set. Control to select.

  By this control, for example, a flaw mark is detected in the Y pixel data, and if no flaw mark is detected in the X pixel data, the X pixel data is doubled and output from the image composition circuit 82. The reverse is true when a scratch mark is detected only in the X pixel data. In addition, when a scratch mark is detected in the pixel data of both pixels, the pixel value 0 of the scratch mark is output, so that it remains as a scratch mark.

  Next, in the third embodiment, the image P (1 + 2) obtained by combining the first image and the second image is combined with the third image P (3). In this case, the pixel data of the image P (1 + 2) is pixel data of two images, and the pixel data of the image P (3) is pixel data of one image. The coefficients are set to A = 3/2 and B = 3/1 = 3. Then, each pixel data is synthesized in the same manner as when the first and second images are synthesized.

  For example, when a flaw mark is detected only in the Y pixel data, the X pixel data is output by being multiplied by 3/2. On the other hand, when a flaw mark is detected only in the X pixel data, the Y pixel data is tripled and output. When a flaw mark is detected in the pixel data of both pixels, the pixel value 0 of the flaw mark is output, so that it remains as a flaw mark.

  Similarly, the images are sequentially synthesized. At that time, prior to image synthesis, the number of images synthesized to obtain each image to be synthesized is set as the denominator (one in the case of an unsynthesized image), and the number of images synthesized to obtain each image. Are set as the coefficients of the remainders 306 and 308.

  In this way, the composition is continued until one last image remains (FIG. 11E). In this way, the flawed pixel data can be corrected by interpolating the pixel data to which the flaw mark is attached using the aligned pixel data to be synthesized. Although the number of flaw pixel data in the image synthesized by this processing can be reduced, the flaw pixel data finally remaining is corrected by the flaw correction circuit 72 using surrounding pixels.

  Next, a more specific control example for realizing the flaw pixel data correction at the time of the electronic camera shake correction will be described with reference to FIG. Here, a case will be described in which N images obtained by continuous shooting in a short time are combined while correcting camera shake.

  First, the variable n is initialized to 0 (step S302). Next, the positional deviation detection circuit 80 detects the positional deviation amount of the two images P (n) and P (n + 1) to be combined (step S304). Since n = 0 here, the amount of positional deviation between the images P (0) and P (1) is detected. Next, coefficients A and B are set as described above (step S306). Here, A = B = 2/1 = 2 is set.

  Next, the pixel data of the images P (0) and P (1) are combined by the image combining circuit 82 while being aligned based on the amount of displacement, and the image is combined to replace P (1) (step) S308). Thereafter, n is incremented by 1 (step S310), and it is determined whether n is smaller than N. If it is smaller, the process returns to step S304 and the above process is repeated.

  If n is greater than or equal to N, it is determined that the synthesis of all images has been completed, and the process proceeds to step S314 where the scratch correction circuit 72 corrects the scratch marks still remaining after the synthesis of all the images. Exit.

  As described above, according to the third embodiment, it is possible to appropriately correct the image deterioration caused by the pixel defect of the image sensor while performing the electronic camera shake correction.

  In the first to third embodiments, the case has been described in which a flaw mark is added to the flaw pixel data by the flaw mark circuit 70 and the flaw mark is detected by the image composition circuit 82. However, the present invention is not limited to this. For example, the misregistration detection circuit 80 and the image composition circuit 82 may directly acquire the flaw data and detect the flaw position. In that case, the image composition circuit 82 may acquire the position shift information from the position shift detection circuit 80 and detect the scratch position in consideration of the position shift.

<Fourth Embodiment>
Next, a third embodiment of the present invention will be described. In the first to third embodiments described above, the case where the defect pixel data is corrected when the image is synthesized is described. However, in the fourth embodiment, after the defect pixel data is corrected first, A case where images are combined will be described.

  Since the configuration other than the detailed configuration of the image composition circuit 82 is the same as the configuration described in the first embodiment, the description is omitted. Hereinafter, a configuration and method of an image synthesis circuit characteristic of the fourth embodiment, which is performed in place of FIGS. 4, 5, and 6 of the first embodiment, will be described.

FIG. 13 is a diagram illustrating a schematic configuration of the image composition circuit 82 in the fourth embodiment. X and Y indicate two pieces of input image data to be combined, and Z indicates the combined image data. Reference numeral 502 denotes an adder. The image data X and Y are input in a state where the position shift is canceled by the memory control circuit 50 based on the detection result of the position shift detection circuit 80.

  At the time of electronic camera shake correction, in the fourth embodiment, a positional shift is made with respect to an image obtained by applying a flaw mark to the output image of the A / D converter 14 by the flaw mark circuit 70 and flaw corrected by the flaw correction circuit 72. Detect and composite images.

  FIG. 14 is a flowchart for explaining the operation of the system control circuit 90 in the fourth embodiment. Here, a case will be described in which N images obtained by continuous shooting in a short time are combined while correcting camera shake.

  First, the variable n is initialized to 0 (step S402). Next, the positional deviation detection circuit 80 detects the positional deviation amounts of the two images P (n) and P (n + 1) to be combined after the flaw correction is performed by the flaw correction circuit 72 (step S404). ). Since n = 0 here, the amount of positional deviation between the images P (0) and P (1) is detected. Next, the image composition circuit 82 adds the respective pixel data of the images P (0) and P (1) while aligning based on the amount of displacement, and the image composition is performed to replace P (1) (step). S406).

Then, after n is incremented by 1 (step S408), the process returns to step S404 and repeats the above process until n becomes N or more (during YES in step S410).
When n is equal to or greater than N, it is determined that all images have been combined, and the process ends.

  As described above, according to the fourth embodiment, by correcting the damaged pixel data before synthesizing the image, it is possible to appropriately correct the image deterioration caused by the pixel defect of the image sensor even during the electronic camera shake correction. Can do.

<Other embodiments>
The object of the present invention can also be achieved as follows. First, a storage medium (or recording medium) that records a program code of software that implements the functions of the above-described embodiments is supplied to a system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the following can be achieved. That is, when the operating system (OS) running on the computer performs part or all of the actual processing based on the instruction of the read program code, the functions of the above-described embodiments are realized by the processing. It is. Examples of the storage medium for storing the program code include a flexible disk, hard disk, ROM, RAM, magnetic tape, nonvolatile memory card, CD-ROM, CD-R, DVD, optical disk, magneto-optical disk, MO, and the like. Can be considered. Also, a computer network such as a LAN (Local Area Network) or a WAN (Wide Area Network) can be used to supply the program code.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to an embodiment of the present invention. It is a figure which shows schematic structure of the flaw correction circuit in embodiment of this invention. 1 is a diagram illustrating a schematic configuration of an image synthesis circuit according to a first embodiment of the present invention. It is a figure for demonstrating the image composition method at the time of electronic camera-shake correction in the 1st Embodiment of this invention. It is a flowchart for demonstrating control by the system control circuit in the 1st Embodiment of this invention. It is a figure which shows the table | surface which showed the change of the variable in the flowchart of FIG. It is a figure which shows schematic structure of the image composition circuit in the 2nd and 3rd embodiment of this invention. It is a figure for demonstrating the image composition method at the time of the electronic camera-shake correction in the 2nd Embodiment of this invention. It is a flowchart for demonstrating control by the system control circuit in the 2nd Embodiment of this invention. It is a figure which shows the table | surface which showed the change of the variable in the flowchart of FIG. It is a figure for demonstrating the image composition method at the time of the electronic camera-shake correction in the 3rd Embodiment of this invention. It is a flowchart for demonstrating control by the system control circuit in the 3rd Embodiment of this invention. It is a figure which shows schematic structure of the image composition circuit in the 4th Embodiment of this invention. It is a flowchart for demonstrating control by the system control circuit in the 4th Embodiment of this invention.

Explanation of symbols

  10: imaging lens, 11: shutter, 12: imaging device, 14: A / D converter, 16: D / A converter, 18: image display unit, 20: recording medium, 30: image processing circuit, 40: memory 50: Memory control circuit, 60: Compression / decompression circuit, 70: Scratch mark circuit, 72: Scratch correction circuit, 80: Misalignment detection circuit, 82: Image composition circuit, 90: System control circuit, 92: Operation unit, 96 : ROM, 100: Imaging device, 202, 204, 302, 304: Scratch detection circuit, 203, 205: Multiplier, 206, 208, 314, 316: Data selector, 210, 318, 502: Adder, 306, 308 : Remainder unit, 402, 404, 406: flip-flop, 408: adder, 410: data selector, 412: flaw mark detection circuit

Claims (15)

A shake correction apparatus that superimposes and combines a plurality of images obtained by an imaging unit including a plurality of pixels on a single image,
Detecting means for detecting a positional deviation amount between a plurality of images to be combined;
The data of each pixel of the plurality of images to be combined are added while being aligned based on the amount of positional deviation detected by the detection means, and the pixel data of one image is data output from the defective pixel And a compositing means for performing correction using data of corresponding pixels of another image. Storage means for storing position information of defective pixels of the imaging means;
Further comprising means for replacing the data output from the defective pixel with a preset fixed value based on the positional information of the defective pixel stored in the storage means;
2. The composition unit according to claim 1, wherein when the pixel data of one image is data replaced with the fixed value, correction is performed using data of a corresponding pixel of another image. The described blur correction device. A storage means for storing position information of defective pixels of the imaging means;
The synthesizing unit determines whether or not the data of each pixel of the plurality of images to be synthesized is data output from the defective pixel based on the position information of the defective pixel stored in the storage unit. The shake correction apparatus according to claim 1. It further comprises correction means for correcting the data output from the defective pixel using the data of the peripheral pixels,
When the pixels corresponding to each other between the plurality of images to be combined are defective pixels, the combining unit does not correct the data of the pixels, and the correcting unit performs correction after combining the plurality of images. The blur correction apparatus according to claim 1, wherein data output from a defective pixel that has not been corrected is corrected.   The synthesizing unit includes a multiplying unit that multiplies the input pixel data by a coefficient, and the coefficient is set according to the number of images already synthesized in order to obtain the plurality of images to be synthesized. The correction is performed by multiplying the corresponding pixel data of the other image by the coefficient when the pixel data is data output from the defective pixel. 2. The blur correction device according to claim 1. A shake correction apparatus that superimposes and combines a plurality of images obtained by an imaging unit including a plurality of pixels on a single image,
Storage means for storing position information of defective pixels of the imaging means;
Correction means for correcting data output from the defective pixel based on data of peripheral pixels based on the position information of the defective pixel stored in the storage means;
Detecting means for detecting a positional deviation amount between a plurality of images to be combined;
A blur correction unit comprising: a combining unit that adds the data of each pixel of the plurality of images to be combined corrected by the correction unit while aligning the data based on the amount of displacement detected by the detection unit. apparatus.   An imaging apparatus comprising the shake correction apparatus according to claim 1. A blur correction method for superimposing and synthesizing a plurality of images obtained by continuous exposure in a short time by an imaging means including a plurality of pixels on a single image,
A detection step of detecting a positional deviation amount between a plurality of images to be combined;
Data of each pixel of the plurality of images to be combined is added while being aligned based on the amount of displacement detected in the detection step, and pixel data of one image is data output from a defective pixel And a combining step of performing correction using data of corresponding pixels of another image. An acquisition step of acquiring the position information from the storage means storing the position information of the defective pixel of the imaging means;
Based on the position information acquired in the acquisition step, further comprising the step of replacing the data output from the defective pixel with a preset fixed value,
9. The composition step according to claim 8, wherein when pixel data of one image is data replaced with the fixed value, correction is performed using data of a corresponding pixel of another image. The described blur correction method. An acquisition step of acquiring the position information from storage means storing the position information of the defective pixel of the imaging means;
In the synthesis step, it is determined whether data of each pixel of the plurality of images to be synthesized is data output from a defective pixel based on the position information acquired in the acquisition step. Item 9. The blur correction method according to Item 8. In the synthesis step, when pixels corresponding to each other between the plurality of images to be synthesized are both defective pixels, correction of the data of the pixels is not performed.
11. The method according to claim 8, further comprising a correction step of correcting data output from a defective pixel that has not been corrected after combining the plurality of images by using data of peripheral pixels. 2. The blur correction method according to item 1.   The synthesizing step includes a step of multiplying input pixel data by a coefficient, and the coefficient is set according to the number of images already synthesized in order to obtain the plurality of images to be synthesized. 12. The correction is performed by multiplying the data of the corresponding pixel of the other image by the coefficient when the data is the data output from the defective pixel. 2. The blur correction method according to item 1. A blur correction method for superimposing and synthesizing a plurality of images obtained by an imaging means including a plurality of pixels on a single image,
An acquisition step of acquiring the position information from the storage means storing the position information of the defective pixel of the imaging means;
Based on the position information acquired in the acquisition step, a correction step of correcting data output from the defective pixel using data of peripheral pixels;
A detection step of detecting a positional deviation amount between a plurality of images to be combined;
And a combining step of adding the pixel data of the plurality of images to be combined corrected by the correcting step while aligning them based on the amount of displacement detected in the detecting step. Method.   A program for causing a computer to execute each step of the shake correction method according to any one of claims 8 to 13.   A computer-readable storage medium storing the program according to claim 14.

Images