JP5371470B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for processing an image signal output from an image sensor such as a CCD or a CMOS image sensor, and more particularly to an improvement in image quality of a captured image.

  Electronic image processing apparatuses such as digital cameras that record and reproduce still images and moving images captured by a solid-state imaging device (hereinafter referred to as an imaging device) such as a CCD or CMOS are commercially available. In the imaging device, a defect (scratch) occurs for each pixel in a manufacturing process or the like, and a pixel or the like that outputs an abnormal level may appear. In order to reduce the influence of such defective pixels (hereinafter referred to as defective pixels), it is common to perform pixel defect correction (hereinafter referred to as defect correction) in recent imaging apparatuses.

  Mainly, at the time of shipment of the image sensor, the sensor output at the standard charge accumulation time under a predetermined condition is evaluated, and the defective pixel is determined based on the evaluation result. Then, a method is adopted in which data such as the defective pixel address and scratch level is acquired and stored in the memory. This data describes the position data (x, y) of the defective pixel and its level.

In defect correction, it is possible to further reduce image quality degradation by performing interpolation calculation processing based on image data of pixels adjacent to the defective pixel. In addition, in order to interpolate for each same color of RGB, correction values are often created using adjacent same color pixels. (For example, see Patent Documents 1 and 2.)
In recent years, some pixels other than the imaging purpose (for example, temperature sensor pixels, distance measurement sensor pixels, etc.) are provided in the imaging region of the imaging device, and these pixels are not used for imaging, so there is a defect. It is necessary to perform interpolation processing in the same way as pixels. On the other hand, in addition to defective pixels that occur during the manufacturing process, defective pixels that occur after manufacturing due to cosmic rays, electrostatic breakdown, etc. are configured to detect and correct defects on the image. There is also.

  As described above, in order to achieve both the correction of defects that occurred during the manufacturing process of the image sensor and the correction of defects that occur after manufacturing, the defects that occurred in the manufacturing process that were stored in advance were corrected and then occurred after manufacturing. Some detect and correct defects. (For example, refer to Patent Document 3) In addition, there is a technique in which defective pixel data generated in the manufacturing process and defective pixel data generated after manufacturing are stored in separate memory areas and corrected simultaneously. (For example, see Patent Document 4)

Japanese Patent Laid-Open No. 10-42201 JP 2003-333435 A JP 2002-27323 A JP 2003-259221 A

  However, when correction is performed by the method described in Patent Document 3, correction is preferably performed. However, since a defect detection process is further performed on the image that has been subjected to the correction process (interpolation process), In order to obtain the image, the correction operation must be performed twice. For this reason, there is a problem in the quickness and the like.

  In Patent Documents 3 to 4, there is no detailed mention of a method for detecting defects generated after manufacturing. For example, it is not possible to adequately cope with a method for detecting a defect (separating a subject and a defect) from an image having a different light output for each pixel obtained by photographing a normal subject.

  If the defect detection is to be detected from an image with different light output without interpolating the defective pixel in view of the rapid shooting property, how much the target pixel is from the peripheral pixels (such as adjacent pixels of the same color) of the defect determination target pixel. Guess what is the output. (For example, the estimation is based on the average value of the output of surrounding pixels.) Then, it is determined whether or not it is a defect depending on the degree of deviation from the estimated value. However, if there is a defect in a peripheral pixel (same color adjacent pixel or the like), the defect determination estimation value is shifted, and as a result, there is a high possibility that the defect determination is erroneous.

  In addition, even when a defect (unused) pixel that is stored in advance is subjected to interpolation processing before the defect detection operation, if a defect that occurs later is around the defect interpolation target pixel (such as an adjacent same color pixel), The defective pixel output is also used for calculating the interpolation value. Therefore, accurate interpolation cannot be performed.

An object of the present invention, properly defects of the imaging element that occurred after defects and manufacturing has occurred in the manufacturing process, and a Rukoto issue inspection without degrading the quickness.

In order to solve the above-described problems, an image processing apparatus according to the present invention includes a storage unit that stores in advance unused pixels that do not use an output pixel signal as image data, and determines whether the target pixel is a defective pixel. , if the corresponding pixel that includes the non-use pixel and judgment means for performing, based on the pixel signals output from the different multi few strokes element, before Kifuku few strokes element, output from the unused pixels characterized in that it has control means for the determination means based on the pixel signals output from the double number stroke element excluding the pixel signal is controlled to determine the defect pixel, a.

In order to solve the above problems, an image processing method according to the present invention, the determination target pixel is whether a defective pixel, the pixel signals output from the different multi-number-field containing the said target pixel a determination step of performing on the basis of, if the previous contains the unused pixels Kifuku few strokes element, based on the pixel signals output from the double number stroke element excluding a pixel signal outputted from the unused pixels and having a control step of controlling so as to determine the defect pixels in the determining step Te.

According to the present invention, properly defects of the imaging element that occurred after the manufacturing process and manufacturing, and can Rukoto issue inspection without degrading the quickness.

1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. It is an operation | movement sequence flowchart in 1st Example of this invention. It is an operation | movement sequence flowchart in 1st Example of this invention. It is a figure which shows the example of judgment value calculation of the defect detection in 1st Example of this invention. It is an operation | movement sequence flowchart in 1st Example of this invention. It is an operation | movement sequence flowchart in the 2nd Example of this invention. It is a figure which shows the determination value calculation example of the defect detection in 2nd Example of this invention.

(Example)
(First embodiment)
FIG. 1 is a block diagram showing an overall configuration of an imaging apparatus (image processing apparatus) in an embodiment of the present invention. In FIG. 1, the well-known photographic lens 110 is provided with a motor (not shown), and is provided with a mechanism for driving and focusing the motor in accordance with a processing result of a distance measurement control unit 142 described later. The lens control unit 111 transmits information from the photographing lens 110 to a system control circuit 150 serving as a control unit, and controls the operation of the photographing lens 110.

  The lens control unit 111 includes a control signal generation unit, and motor driving such as a focus adjustment operation of the photographing lens 110 is performed by a pulse signal obtained by the control signal generation unit. The shutter 112 controls the exposure amount of the image sensor 114 which is a CMOS image sensor or the like. The image sensor 114 converts the optical image into an electrical signal (pixel signal). In this embodiment, a CMOS image sensor is used as the image sensor 114.

  A low-pass filter LPF 115 is provided between the photographic lens 110 and the image sensor 114 for cutting an extra wavelength of light that has passed through the photographic lens 110 (an unnecessary wavelength that affects color reproduction). An analog front end circuit (hereinafter referred to as AFE) 116 includes an A / D converter that converts an analog signal output from the image sensor 114 into a digital signal, a clamp circuit (offset adjustment circuit), and a D / A converter. Contains a bowl.

  A timing generation circuit (TG) 118 supplies a clock signal and a control signal to the image sensor 114 and the AFE 116 and is controlled by the memory control circuit 122 and the system control circuit 150. The image processing circuit 120 performs various corrections such as defect detection, defect / unused pixel interpolation processing, predetermined pixel interpolation processing and color conversion processing on the data from the AFE 116 or the data from the memory control circuit 122. I do.

  The image processing circuit 120 includes a detection defect detection block 120-1 having a defective pixel setting function for marking a defect / unused pixel stored in advance and a defect detection function for detecting the position of the defective pixel. ing. Further, a defect (unused pixel) correction block 120-2 for performing pixel interpolation processing (defective pixel correction) on the marking pixel, a development processing block 120-3 for performing color conversion, and the like are also provided. The memory control circuit 122 controls the AFE 116, the timing generation circuit 118, the image processing circuit 120, the image display memory 124, and the memory 130.

  During the EVF operation, images are continuously displayed (moving images are displayed) on the image display unit 128 formed of a TFT LCD, and the movement of the subject can be confirmed. The memory 130 for storing captured still images and moving images has a storage capacity sufficient to store a predetermined number of still images and moving images for a predetermined time. The shutter control unit 140 controls a known shutter 112.

  The distance measurement control unit 142 performs AF (autofocus) processing. The thermometer 144 measures the ambient temperature in the shooting environment and the temperature inside the camera (such as around the image sensor). The photometry control unit 146 performs AE (automatic exposure) processing. The photometry control unit 146 also has a flash photographing function in cooperation with the flash unit 148.

  The flash unit 148 used for shooting in the dark also has a function of projecting AF auxiliary light. The flash unit 148 is directly connected to the accessory shoe 147 that is normally fixedly attached to the image pickup apparatus. However, the flash unit 148 is separated from the image pickup apparatus by a dedicated cable or the like according to the shooting situation. It is also possible to connect to. Whether the flash unit 148 is directly connected to the accessory shoe 147 or connected by a dedicated cable or the like is configured to be able to determine by using a part of the communication line of the accessory shoe 147.

  A system control circuit 150 that controls the entire image processing apparatus incorporates a known CPU or the like. A memory 152 serving as a storage unit stores constants, variables, programs, and the like for operation of the system control circuit 150. Preset pixel defect correction data, shading correction data, and the like are also stored in the memory 152. The display unit 154 displays an operation state, a message, and the like according to the execution of the program in the system control circuit 150.

  The non-volatile memory 156 is configured by an electrically erasable / recordable EEPROM or the like in which a program described later is stored. The operation unit 160 includes a well-known main switch (startup switch) for inputting various operation instructions of the system control circuit 150, a shutter switch, a mode setting dial for switching a photographing mode, and the like.

  The power supply control unit 182 includes a battery detection circuit, a DC-DC converter, and the like. The power supply unit 186 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. The removable recording medium 1200 is a memory card or a hard disk.

  FIG. 2 is a schematic operation sequence during a shooting operation of the imaging apparatus according to the first embodiment of the present invention. In addition, since operations such as photometry / ranging operation that are normally performed in an imaging operation are well known, descriptions in this embodiment are omitted.

  In step S <b> 101, the system control circuit 150 starts signal accumulation by the image sensor 114 via the timing generation circuit 118. In step S102, exposure is performed by opening and closing a mechanical shutter (not shown). In step S103, signal accumulation by the image sensor 114 is terminated.

  In step S104, reading of the charges accumulated in the image sensor 114 from the first row to the subsequent circuit is started. During the read operation, the horizontal OB is clamped by the AFE 116 so that the horizontal OB output becomes a predetermined offset level. The image outputs that are horizontally OB clamped by the AFE 116 are sequentially sent to the image processing circuit 120 at the subsequent stage.

  In step S105, information on unused pixels (defects that do not use the output pixel signal as image data, pixels other than imaging, etc.) stored in advance in the memory 152 in the defect detection block 120-1 in the image processing circuit 120. Based on the above, a marking operation is performed on the correction target pixel.

  The marking operation is not a correction but a process of applying a predetermined mark (for example, a predetermined value [output value 0 etc.] etc.) so that it can be understood that it is a pixel to be interpolated when performing an operation such as a development process. ), And the processing time can be shorter than that when the interpolation processing is performed. Here, as shown in FIG. 4A, the G pixel in e column 6 rows, the B pixel in h column 4 rows, and the B pixel in j column 8 rows are stored in the memory 152 as unused pixels. A marking operation is performed on these three pixels.

  In step S106, it is detected whether or not a defective pixel remains in the image output once marked in step S105, and additional marking is performed on the pixel determined to be a defective pixel. That is, a defective pixel of the image sensor generated after manufacture is detected. The detection method and the like will be described later with reference to FIG.

  In step S107, it is determined whether or not the above steps S104 to S106 have been completed to the last line. If the process has not been completed to the last line, the process returns to step S104 to perform the read operation for the next line, and has been completed to the last line. Then, the process proceeds to step S108.

  In step S108, the defect correction block 120-2 in the image processing circuit 120 serving as an interpolation processing unit performs well-known processing on a pixel that has been marked out of the image output (pixel signal) that has been read out. Perform interpolation processing. Details of this processing will be described with reference to FIG.

  In step S109, in the development processing block 120-3 in the image processing circuit 120, image processing such as well-known development processing / compression processing is performed on the image subjected to the defective pixel interpolation processing in step S108.

  FIG. 3 is a flowchart of the sequence for detecting defective pixels in step S106 of FIG. 2, and the operation is performed in a state where marking of defective pixels stored in advance is completed in step S105 of FIG. Here, as shown in FIG. 4A, the G pixel in e column 6 rows, the B pixel in h column 4 rows, and the B pixel in j column 8 rows are marked as unused pixels.

  In step S201, the output of the same color pixel around the pixel of interest is checked, and it is determined whether marking has been performed (whether there is a defective pixel). If there is marking, the process proceeds to step S202. If there is no marking, the process proceeds to step S203.

  In step S202, a marking position is memorize | stored when there exists marking in the surrounding same color pixel (a several predetermined pixel) in step S201. In step S203, usable pixels other than the marked pixels in the peripheral same color pixels stored in step S202 are confirmed.

  In step S204, the output of the pixel of interest is predicted from the output value (pixel signal) of the surrounding same color pixel stored in step S201 and the output pixel signal of the surrounding pixel that can be used for the determination confirmed in step S203. Set the defect judgment value. In calculating the defect determination value, the pixel output (output pixel signal) at the marking position detected in step S202 is excluded.

For example, the same color pixels in the periphery are assumed to be 8 pixels in the upper left, directly above, upper right, left side, right side, lower left, directly below, and lower right, and all are normal like the R pixel in the c column and 3 rows in FIG. 4B. In the case of a pixel, “total of all outputs ÷ ± n% centered on 8 pixels” is set as the determination value. Further, when the upper right and lower right are marked as defective pixels as in the B pixel in the h column and the 6th row in FIG. 4C, the result of excluding those two pixels from the calculation target is obtained. . That is, the determination value is “± n% centered on ((upper left + left lateral + lower left + upper right + right lateral + below)) / (8 pixels−2 pixels)”. (If the output value is set to 0 by marking the defect, it may be “total of all outputs ÷ (± 8% of 8 pixels−2 pixels)”)
In step S205, the output value of the target pixel is compared with the determination value calculated in step S204 to determine whether the pixel is a defective pixel. If it is determined that the target pixel is a defective pixel, a marking operation is performed in step S206, and the process returns to step S107 in FIG.

  Note that, as an example of the defect determination value calculation, the average value of the peripheral same color pixels is described as a reference. However, the present invention is not limited to this, and linear interpolation is performed on the peripheral pixels in the diagonal of the defect determination target, and linear There is no problem even if the average value of the interpolation values or the calculation with the weighting operation is performed.

  FIG. 5 is a detailed flowchart of the defect correction process in step S108 of FIG. In step S401, it is determined whether or not the pixel is a defective pixel marked in FIG. 3. If it is marked, the process proceeds to step S402, and if not marked, the process proceeds to step S405.

  In step S402, captured image data (pixel signal) of adjacent pixels of the same color is read. In step S403, the correction amount of the defective pixel is calculated from the value of the adjacent pixel obtained in step S402.

  In step S404, the correction amount of the defective pixel calculated in step S403 is written in the nonvolatile memory 156 as output data of the defective pixel. Thus, the correction process for the corresponding pixel is completed.

  In step S405, it is determined whether or not the defect correction process has been completed for all the marking pixels. If not completed, the process proceeds to step S401. If completed, the process proceeds to step S109 in FIG. This completes the defect correction processing sequence.

(Second embodiment)
In the first embodiment of the present invention described above, when there is a defective (unused) pixel stored in advance in the defect determination value calculation area during the defect detection operation, the defect (unused) pixel is calculated as a defect determination value. It is excluded from the target. In such a handling method, the number of pixels used for defect determination value calculation decreases, and the deviation of the defect determination value may increase.

  In the second embodiment of the present invention, in view of the above problems, when there is a defect (unused) pixel stored in advance in the defect determination value calculation area, the defect determination target pixel and the defect (unused) pixel Pixels outside the normal calculation area existing on the line are used as determination value calculation targets. Note that the overall configuration of the image pickup apparatus and the schematic operation sequence during the shooting operation are the same as those in FIGS. 1 and 2 in the first embodiment of the present invention that has already been described. To do.

  FIG. 6 is a flowchart of a sequence for detecting a defective pixel in step S106 in FIG. 2, and the operation is performed in a state where marking of a defective pixel stored in advance is completed in step S105 in FIG.

  In step S501, the output pixel signal of the same color pixel around the pixel of interest is checked, and it is determined whether marking is performed (whether there is a defective pixel). If there is marking, the process proceeds to step S502, and if there is no marking, the process proceeds to step S503. In step S502, a marking position is memorize | stored when there exists marking in the surrounding same color pixel in step S501.

  In step S503, the position of the pixel outside the determination region located on the line of the defect determination target pixel and the marking pixel (determination exclusion pixel) is stored with respect to the marking position stored in step S502. In step S504, usable pixels other than the marked pixels in the peripheral same color pixels stored in step S502 and pixels outside the determination region stored in step S503 are confirmed.

  In step S505, the output of the pixel of interest is predicted from the output value of the surrounding same color pixel stored in step S501 and the surrounding pixel output (output pixel signal) that can be used for the determination confirmed in step S504, thereby determining the defect. Set the value. In calculating the defect determination value, instead of excluding the pixel output (output pixel signal) at the marking position detected in step S502, the output data (pixel signal) of the pixel outside the determination region confirmed in step S504 is used. .

  For example, if the surrounding same color pixels are 8 pixels in the upper left, directly above, upper right, left side, right side, lower left, directly below, and lower right, the upper right ( When e column 6 row) is marked as a defective pixel, the upper right pixel (f column 5 row) is further used as the calculation target. Therefore, the number of determination value calculation pixels does not decrease. However, when there is no pixel outside the determination region because the lower right defective pixel is at the end, such as the B pixel in h column and 6 row in FIG. 7C, the first embodiment of the present invention. Similarly to the above, a determination value is obtained by excluding the marked pixel from the calculation target.

  In step S506, the output value of the target pixel is compared with the determination value calculated in step S505 to determine whether the pixel is a defective pixel. If it is determined that the target pixel is a defective pixel, a marking operation is performed in step S507, and the process returns to step S107 in FIG.

  Note that, as an example of the defect determination value calculation, the average value of the surrounding same color pixels is described as a reference, but the present invention is not limited thereto, and linear interpolation is performed with the surrounding pixels at the diagonal of the defect determination target, There is no problem even if the average value of the linear interpolation value or the calculation performed by weighting is performed.

114 Image sensor 150 System control circuit 152 Memory 156 Non-volatile memory 120 Image processing circuit

Claims (12)

Storage means for previously storing unused pixels that do not use the output pixel signal as image data;
The determination target pixel is whether a defective pixel, and judging means for performing, based on the pixel signals output from the different multi-number-field containing the said target pixel,
If that contains the unused pixels before Kifuku few strokes element, wherein the determination means based on the pixel signals output from the double number stroke element excluding a pixel signal outputted from the non-use pixels from defects Control means for controlling to determine a pixel;
An image processing apparatus comprising: Wherein when that contains the unused pixels in the double number strokes element, wherein the pixel signal outputted from the double number stroke element excluding a pixel signal outputted from the unused pixels, the unused the image processing apparatus according to claim 1, wherein the determining means based on the pixel signals output from the other pixels in place of pixels and controls to determine the defect pixel. 3. The determination unit according to claim 1, wherein the determination unit calculates a determination value based on pixel signals output from the plurality of pixels, and determines whether or not the target pixel is a defective pixel. The image processing apparatus described. The determination unit determines whether the target pixel is a defective pixel based on a result of comparing the determination value and a pixel signal output from the target pixel. An image processing apparatus according to 1. 5. The image processing apparatus according to claim 1, further comprising an interpolation processing unit configured to perform an interpolation process on a pixel signal output from the pixel determined to be the unused pixel and the defective pixel. The image processing apparatus according to claim 1, wherein the plurality of pixels are pixels having a predetermined positional relationship with the target pixel. The determination target pixel is whether a defective pixel, a determination step for, based on the pixel signals output from the different multi-number-field containing the said target pixel,
Before the case that contains the unused pixels Kifuku few strokes element, from defects in said determination step based on the pixel signals output from the double number stroke element excluding a pixel signal outputted from the unused pixels A control step for controlling to determine a pixel;
An image processing method comprising: Wherein in the control step, when it is included the unused pixels in the double number strokes element, wherein the pixel signal outputted from the double number stroke element excluding a pixel signal outputted from the unused pixels, the unused the image processing method according to claim 7, wherein the controller controls to determine the defect pixels in the determining step based on the pixel signals output from the other pixels in place of the pixel. 9. The determination step according to claim 7, wherein the determination step calculates a determination value based on pixel signals output from the plurality of pixels and determines whether or not the target pixel is a defective pixel. The image processing method as described. 10. The determination step includes determining whether or not the target pixel is a defective pixel based on a result of comparing the determination value and a pixel signal output from the target pixel. An image processing method described in 1. 11. The image processing method according to claim 7, further comprising an interpolation process step of performing an interpolation process on a pixel signal output from a pixel determined to be an unused pixel and a defective pixel. The image processing method according to claim 7, wherein the plurality of pixels are pixels having a predetermined positional relationship with the target pixel.