JP2011151627A - Image processor, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To change later a strength of recovery processing for an image for which the recovery processing is performed once. <P>SOLUTION: A photographed image obtained by imaging, and a correction filter to be used for performing processing of correcting blurring resulting from aberration of an optical system of an imaging apparatus used for the imaging to the photographed image are acquired. A corrected image is generated by applying the correction filter to the photographed image. A difference image between the photographed image and the corrected image is generated. The photographed image or the corrected image, and the difference image are stored. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for correcting an image.

  It is known that a subject image formed on an image sensor by an optical system such as a zoom lens is deteriorated (blurred) compared to the original subject. This is due to the aberration of the optical system. In order to obtain an ideal image, a technique for removing the blur caused by such an optical system is called “image restoration processing” or “restoration processing”.

  Examples in which the image restoration technique is applied to a digital camera are disclosed in Patent Document 1 and Patent Document 2. Patent Document 1 discloses a method of performing image processing on a blurred image obtained by an imaging apparatus and generating an image with little deterioration. In the method of Patent Literature 1, image processing is performed in consideration of imaging conditions (aperture, focal length, distance to a subject) and imaging device characteristic information (lens optical characteristics, imaging device identification information, etc.).

  Patent Document 2 discloses a method of determining a degradation function in consideration of shooting conditions for each pixel and performing image restoration using the degradation function. According to the method of Patent Document 2, in order to simplify the arithmetic processing, first, a degradation function is obtained for a representative pixel, and the degradation functions of other pixels are obtained by interpolation.

JP 2000-20691 A JP 2001-197354 A

  However, neither of Patent Document 1 and Patent Document 2 discloses a mechanism for adjusting the degree of recovery processing. It is desirable that the strength of the recovery process can be changed according to the subject and the shooting mode. For example, when a face appears large in a captured image (for example, when it is detected by face detection that the face occupies most of the image), a soft impression is given to the image by performing weak recovery processing and leaving blur. Can be made. Further, it is preferable to perform weak recovery processing when shooting a portrait (for example, shooting in portrait mode) than when shooting a landscape (for example, shooting in landscape mode).

  Furthermore, it is preferable that the strength of the recovery process can be changed arbitrarily (steplessly). In particular, it is preferable to further change the strength of the recovery process for an image obtained by performing the recovery process once. Then, for example, if the image obtained by the imaging device uniformly performing the recovery process is dissatisfied, the user can change the strength of the recovery process and obtain a desired image.

  The present invention makes it possible to later change the strength of the recovery process for an image that has been subjected to the recovery process.

In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement. That is,
Acquisition means for acquiring a captured image obtained by imaging, and a correction filter used for performing processing for correcting the blur caused by the aberration of the optical system of the imaging apparatus used for the imaging on the captured image; ,
Correction means for generating a corrected image by applying the correction filter to the captured image;
Difference calculating means for generating a difference image between the captured image and the corrected image;
A storage unit that stores the photographed image or the corrected image and the difference image is provided.

  It is possible to later change the strength of the recovery process for an image that has been subjected to the recovery process.

1 is a block diagram illustrating an example of an image processing apparatus according to Embodiments 1 to 9. FIG. 4 is a block diagram illustrating an example of a configuration of a signal processing unit 103. FIG. 5 is a flowchart of processing according to the first embodiment. FIG. 10 is a block diagram illustrating an example of an image processing apparatus according to a tenth embodiment. 10 is a flowchart of processing according to the third embodiment. FIG. 10 is a diagram illustrating trimming coordinates according to the fourth embodiment. 10 is a flowchart of processing according to the seventh embodiment. 10 is a flowchart of processing according to Embodiment 10; The figure which shows the change of the point image by correction | amendment of a phase component. The figure which shows MTF about a different color component.

  Embodiments of the present invention will be described below with reference to the drawings. However, the scope of the present invention is not limited to the following examples. Hereinafter, the recovery process is a correction process for an image. In the recovery processing in each of the following embodiments, blur caused by aberration of the optical system of the imaging apparatus, particularly blur caused by lens aberration, is removed based on the point spread function (PSF). However, it is possible to use the method of this embodiment for other image correction processes.

Here, the recovery process based on the PSF will be briefly described. The intensity distribution g of the photographed image is a convolution of the luminance distribution f of the original object and a point spread function (PSF: Point spread function) h representing the imaging performance of the optical system, plus noise n. Represented as:
g = f * h + n (* is a convolution integral)
If g, h, and n are known, the luminance distribution f of the original object can be obtained.

[Example 1]
In the present embodiment, a case where an image captured by a digital camera is output as image data will be mainly described. Another information processing apparatus that does not have information such as a filter necessary for the restoration process can perform the restoration process on the image data by using the information included in the image data, and can display or output the image data. Of course, the processing described in this embodiment can be performed by an information processing apparatus having information necessary for the recovery processing. In this case, another information processing apparatus that does not have the information necessary for the recovery process can perform the recovery process on the image data using the information included in the image data, and display or output it.

  As described below, in this embodiment, when outputting data after demosaic processing, a difference image before and after the recovery processing is added to the captured image. In particular, when compressed, the data amount of the difference image is generally small. According to the method of the present embodiment, the degree of recovery processing can be freely changed later with only a small increase in the amount of data.

  In this embodiment, the lens interchangeable single-lens reflex camera creates image data so that the strength of the recovery process for the image can be changed later. The single-lens reflex camera of the present embodiment can store captured images in either the image quality priority mode or the data size priority mode. In the image quality priority mode, the captured image is stored as RAW data. In the data size priority mode, the captured image is saved as JPEG data. The storage format of the captured image is not limited to these, and may be a storage format such as BMP or TIFF. Further, the image may be stored in a storage format selected from three or more types.

  FIG. 1 is an example of an image processing apparatus according to this embodiment. The imaging unit 101 includes a zoom lens, a focus lens, a diaphragm, a shutter, an optical low-pass filter, an iR cut filter, a color filter, and an imaging device including a CMOS and a CCD. The imaging unit 101 detects the amount of light of the subject. The A / D conversion unit 102 converts the amount of light of the subject detected by the imaging unit 101 into a digital value. The signal processing unit 103 performs various image processing such as demosaic processing, white balance processing, γ processing, recovery processing, and reduction processing for creating thumbnail images on the digital value generated by the A / D conversion unit 102. To generate a digital image. The D / A conversion unit 104 performs analog conversion on the digital image generated by the signal processing unit 103. The encoding unit 105 converts the digital image generated by the signal processing unit 103 into a file format such as JPEG or TIFF.

  The media interface 106 is an interface for connecting a storage medium (for example, a hard disk, a memory card, a CF card, an SD card, or a USB memory) to the image processing apparatus in this embodiment. The CPU 107 controls the operation of each unit included in the image processing apparatus according to this embodiment. The CPU 107 sequentially reads and interprets the instructions stored in the ROM 108 or the RAM 109, and executes processing according to the interpretation result. The ROM 108 and the RAM 109 provide programs, data, and work areas necessary for the processing of the CPU 107. Filter data used for recovery processing corresponding to each of a plurality of imaging conditions is also stored in the ROM 108 (storage means).

  The imaging control unit 110 controls the imaging system according to instructions from the CPU 107. The imaging control unit 110 controls operations such as focusing, opening a shutter, and adjusting an aperture. The operation unit 111 receives the input user instruction. The operation unit 111 includes, for example, a button and a mode dial. The character generator 112 can generate characters and graphics. The display unit 113 displays captured images or characters received from the character generator 112 or the D / A conversion unit 114. As the display unit 113, a general liquid crystal display can be used. The display unit 113 may have a touch screen function. In this case, a user instruction to the display unit 113 can be handled in the same way as an input to the operation unit 111.

  FIG. 2 schematically shows the inside of the signal processing unit 103. The switching unit 201 switches data output from the signal processing unit 103 between RAW data and demosaiced data such as JPEG data. That is, the switching unit 201 sets whether to output RAW data (imaging data) (first mode) or to output demosaic data (second mode). The demosaic processing unit 203 performs demosaic processing on the captured image acquired from the A / D conversion unit 102 (acquisition unit). That is, the demosaic processing unit 203 creates an RGB image from the captured image by interpolating the RGB data in the Bayer array. The demosaic processing unit 203 may perform image processing such as white balance processing or γ processing.

  The recovery processing unit 204 performs recovery processing on the RGB data generated by the demosaic processing unit 203 using the filter data 202 (correction unit). The filter data 202 is data derived from the lens OTF and used for the recovery process. The ROM 108 stores one or more filter data 202. The recovery processing unit 204 selects the filter data 202 from the ROM 108 according to conditions such as the type of lens at the time of shooting, the zoom position, the aperture, the subject distance, and the shooting mode. The difference calculating unit 205 calculates a difference between the image (corrected image) generated by the recovery processing unit 204 and the image generated by the demosaic processing unit 203 (difference calculating unit). The condition acquisition unit 206 acquires imaging conditions at the time of imaging of the image data acquired by the demosaic processing unit 203. The shooting conditions include the lens type, zoom position, aperture value, subject distance, shooting mode, output image type (RAW or JPEG), and the like.

  When the image processing apparatus according to the present exemplary embodiment stores RAW data, the switching unit 201 sends the demosaic RAW data acquired from the A / D conversion unit 102 and the filter data 202 to the encoding unit 105. Then, the encoding unit 105 outputs the RAW data and the filter data 202 in the RAW format. When the image processing apparatus according to the present exemplary embodiment stores JPEG data, the switching unit 201 sends the data output from the demosaic processing unit 203 and the difference data output from the difference calculation unit 205 to the encoding unit 105. Then, the encoding unit 105 compresses the data output from the demosaic processing unit 203 according to the JPEG standard. The encoding unit 105 adds difference data to the compressed data according to the JPEG format, and saves the generated data by outputting it as JPEG data (storing means).

  FIG. 3 is a flowchart illustrating processing of the image processing apparatus according to the present exemplary embodiment. The image processing apparatus according to the present embodiment receives imaging data and outputs it as an image file according to the flowchart of FIG. In step S301, the signal processing unit 103 receives the input imaging data. The imaging data is RGB data in a Bayer array. This data is hereinafter referred to as RAW image data. In step S302, the condition acquisition unit 206 acquires the shooting condition of the shooting data received in step S301. In step S303, the recovery processing unit 204 reads from the ROM 108 a recovery filter that matches the imaging condition acquired in step S302.

  In step S304, the recovery processing unit 204 determines whether or not the image type to be output is RAW data based on the imaging conditions acquired in step S302. If the image type is RAW data, the process proceeds to step S305. If the image type is JPEG data, the process proceeds to step S306. In step S305, the switching unit 201 transmits the RAW data acquired in step S301 and the filter coefficient of the recovery filter acquired in step S303 to the encoding unit 105. The encoding unit 105 stores the RAW data acquired in step S301 and the filter coefficient of the recovery filter acquired in step S303. Then, this process ends.

  On the other hand, in step S306, the demosaic processing unit 203 creates an interpolation image by interpolating the Bayer array image acquired in step S301. In step S307, the restoration processing unit 204 creates a restored image by applying the restoration filter acquired in step S303 to the interpolated image data created in step S306. As a specific example, the recovery processing unit 204 performs a convolution process. In step S308, the difference calculation unit 205 acquires a difference image between the restored image acquired in step S307 and the interpolated image acquired in step S306. Specifically, the difference calculation unit 205 calculates a difference image by calculating a difference between corresponding pixel values of each image. This difference image indicates the correction amount in the recovery process. This difference image is used later when adjusting the strength of the recovery process. A specific method for adjusting the strength of the recovery process will be described in the tenth embodiment.

  In step S309, the encoding unit 105 performs JPEG compression on the interpolated image acquired in step S306. In step S310, the encoding unit 105 stores the JPEG image obtained in step S309 and the difference image acquired in step S308. Then, this process ends.

  In this embodiment, it is possible to select either JPEG output or RAW output. However, the digital camera of this embodiment may be capable of only JPEG output or only RAW output. When JPEG output is performed, the difference image may be output as a file different from the JPEG image. Instead, the difference image may be included in the JPEG image file and output. Specifically, it is conceivable that the encoding unit 105 stores an image in a format that conforms to the Exif file format in step S310. In this case, the difference image may be stored in the Makernotes tag in the Exif IFD in the APP1 marker.

  On the other hand, for RAW data, the filter coefficient may be stored as a separate file from the RAW data file. Alternatively, the filter coefficient may be included in the RAW data file and output. There is no unified RAW data file format. Therefore, the RAW data and the filter coefficient may be stored in any way.

  In this embodiment, the digital camera compresses the captured data using JPEG, and stores the JPEG image and the difference image. However, the captured image stored together with the difference image may be compressed or not compressed as long as it is a demosaiced image. For example, the captured image may be stored as an uncompressed TIFF file conforming to the Exif file format. When the TIFF format is used, the difference image may be stored in the Makernotes tag in the Exif IFD as in the case of JPEG. The captured image may be compressed in the JPEG2000 format. At this time, if the captured image is stored in the JP2 file, the encoding unit 105 may create one BOX into which the difference data is written, and write the difference data in the BOX. When the captured image is stored in the PNG format, the encoding unit 105 prepares a unique chunk and writes the difference data in the chunk. Even when other file formats are adopted, the encoding unit 105 may adopt a method that complies with each format.

  In this embodiment, the encoding unit 105 performs JPEG compression on the interpolated image generated by the demosaic processing unit 203. However, the encoding unit 105 may JPEG compress the recovered image generated by the recovery processing unit 204. In this case, the encoding unit 105 stores the compressed restored image and the difference image generated by the difference calculation unit 205. By using the restored image and the difference image, the strength of the restoration process can be adjusted later. That is, the encoding unit 105 may output a captured image or a corrected image (interpolated image or restored image) that may or may not be compressed, and a difference image.

[Example 2]
In Example 1, a filter coefficient is added to RAW data and stored. In this embodiment, OTF data is added to RAW data and stored. There are two methods by which the processing device changes the strength of the RAW data recovery processing using the added filter coefficient as described in the first embodiment. The first method is to directly change the filter coefficient. For example, the processing device subtracts the sum (usually 1) of the filter coefficients from the central value of the filter. Subsequently, the processing apparatus uniformly multiplies each filter coefficient by x. Finally, the processing device adds the value (i.e., 1) previously subtracted to the central value of the filter. In this way, new filter coefficients can be created. (For simplicity, taking a one-dimensional filter as an example, if the original coefficient is [−2 5 −2] and x is 0.5, the new filter coefficient is [−1 3 −1]. In the second method, the processing device converts the filter coefficient into a frequency space by Fourier transform. Next, the processing device changes the filter gain. Finally, the processor performs an inverse Fourier transform to obtain new filter coefficients.

  When the former method is used, the processing apparatus can change the strength of the filter by using a simple calculation. However, with this method, the frequency characteristics of the changed filter cannot be known. When the latter method is used, the frequency characteristics of the filter can be controlled with higher accuracy. However, since complicated processes such as Fourier transform and inverse Fourier transform are used, the process becomes heavy. In addition, the frequency characteristic of the filter can be obtained from the filter coefficient, but the OTF of the lens cannot be calculated. Therefore, when the calculation method of the recovery filter is changed, for example, when the interpolation method of the RAW image data is changed, it cannot be correctly handled.

  In this embodiment, information for creating a recovery filter, that is, lens OTF data is stored together with the RAW data. The processing in this embodiment is almost the same as that in the first embodiment. Hereinafter, only differences from the first embodiment will be described with reference to FIG. In step S303, the recovery processing unit 204 reads the lens OTF data from the ROM 108 instead of acquiring the recovery filter. Here, the recovery processing unit 204 reads the OTF data corresponding to the imaging condition acquired in step S302. As in the first embodiment, step 305 is performed when RAW data is stored. At this time, the switching unit 201 outputs the RAW image data and the OTF data read in step S303 to the encoding unit 105. The encoding unit 105 stores RAW data OTF data.

  Step S307 is performed when JPEG data is stored, as in the first embodiment. At this time, the recovery processing unit 204 generates a recovery filter from the OTF data read in step S303. Then, using the generated recovery filter, a recovered image is generated from the interpolated image as in the first embodiment. In order to create a recovery filter, an existing method such as a Wiener filter may be used.

  According to the method of the present embodiment, the OTF data of the lens can be stored together with the RAW image data. For this reason, the calculation method of a filter and the frequency characteristic of a filter can be changed freely. Since the filter can be freely changed, the recovery process for the RAW image data can be adjusted more appropriately.

[Example 3]
According to the method of the first embodiment, the strength of the recovery process for the captured image can be changed later. However, it is possible not to change the strength of the recovery process. For example, a photographer who has adjusted the recovery process may not want the image to be adjusted further. In this case, the difference image need not be added to the JPEG image. However, when a JPEG image without a difference image is acquired, it is difficult to determine whether restoration processing has already been adjusted for the image or whether restoration processing has been adjusted. By obtaining a recovery filter from a third party, it is possible to perform recovery processing on an image that has not been adjusted for recovery processing. However, if the recovery process is performed on an image for which the recovery process is adjusted, the photographer's intention is impaired and a problem such as ringing occurs.

  Therefore, it is desirable that a mark indicating that the recovery process is adjusted is attached to the image data whose recovery process has already been adjusted. Therefore, in this embodiment, a mark is attached to a JPEG image in which adjustment of recovery processing is prohibited. Hereinafter, the method of this embodiment will be described with reference to FIG. The method of this example is similar to that of Example 1, and the description of the same steps is omitted. In FIG. 5, steps in which operations similar to those in the first embodiment are performed are assigned the same numbers as in FIG. 3.

  Similar to the first embodiment, the recovery processing unit 204 generates a recovered image in step S307. In this embodiment, the strength of the recovery process may be adjusted by adjusting the coefficient of the recovery filter in step S307. The adjustment of the filter coefficient may be performed according to the shooting conditions, may be performed based on the result of image recognition, or may be performed by the user. In step S501, the difference calculation unit 205 determines whether or not the strength of the recovery process can be adjusted. This determination may be made based on the shooting conditions acquired in step S302, or may be made based on a user instruction.

  When it is possible to adjust the strength of the recovery process, the process proceeds from step S501 to step S308. The processing in step S308 and step S309 is the same as that in the first embodiment. In step S502, the encoding unit 105 sets a value (0 in this embodiment) indicating that recovery processing is possible in the recovery flag Fr. In step S503, the encoding unit 105 stores the pre-recovery JPEG image created in step S309, the recovery flag created in step S502, and the difference image created in step S308.

  When it is impossible to adjust the strength of the recovery process, the process proceeds from step S501 to step S504. In step S504, the encoding unit 105 performs JPEG compression on the recovered image acquired in step S307. In step S505, the encoding unit 105 sets a value (1 in this embodiment) indicating that the recovery process is impossible in the recovery flag Fr. In step S506, the encoding unit 105 stores the JPEG image created in step S504 and the recovery flag created in step S505.

  In this way, when a JPEG image without a recovery differential image is handled, it is possible to distinguish between a JPEG image for which recovery processing has been adjusted and a JPEG image for which recovery processing has not been adjusted. Thereby, it is possible to prevent the image quality from being deteriorated by performing the recovery process twice. In this embodiment, a recovery flag is added when the digital camera stores an image. However, the method of this embodiment can also be used when the information processing apparatus reads a JPEG image generated according to the first embodiment, adjusts the strength of the recovery process, and saves it again.

[Example 4]
In this embodiment, trimming information is added to a JPEG image together with a recovery difference image. JPEG images usually do not contain information indicating that they have been trimmed. When trimming is performed, the image size may differ between the restoration difference image and the original JPEG image. In this case, the restored image cannot be obtained by adding the pixel values of the JPEG image and the difference image. A process for matching the positions of the JPEG image and the difference image is required.

  When the aberration recovery process is performed, the difference image normally indicates edge information. Therefore, it is also possible to acquire edge information from a JPEG image and match the positions of the difference image and the JPEG image using the acquired edge information. However, such pattern matching takes time. Further, for example, when using an image including a large number of fine edges, an image including a similar repetitive pattern, or an image having very little edge information, alignment becomes difficult.

  Hereinafter, the method of this embodiment will be described with reference to FIG. The method of this example is similar to that of Example 1, and the description of the same steps is omitted. In step S310, the encoding unit 105 acquires trimming information. The trimming information includes a flag Ft indicating the presence / absence of trimming, the number of pixels (W, H) of an image before trimming, and a trimmed image position Pt. FIG. 6 shows the trimming information of the trimmed image 602. Reference numeral 601 denotes an image before trimming. Reference numeral 602 denotes an image after trimming. In this embodiment, the number of pixels before trimming is (W, H) = (1600, 1200). Since the trimmed image 602 is trimmed, the flag Ft has a value (1 in this embodiment) indicating that trimming has been performed. The trimming image position Pt may be, for example, the upper left coordinates of the trimming area in the image before trimming. In the present embodiment, Pt (X, Y) of the trimmed image 602 is (300, 300). Thereafter, the encoding unit 105 stores the JPEG image created in step S309, the difference data created in step S308, and the above-described trimming information.

  According to the present embodiment, even when a JPEG image is trimmed, the difference image can be correctly used for the JPEG image by referring to the trimming information. In this embodiment, trimming is performed when the digital camera stores an image. However, the method of this embodiment can also be used when the information processing apparatus reads a JPEG image generated according to the first embodiment, performs trimming, and saves it again.

[Example 5]
In this embodiment, in addition to the difference image, the strength of the recovery process is also stored together with the JPEG image. In the first to fourth embodiments, a JPEG image and a difference image are stored. On the other hand, it is preferable that the adjusted data can be stored after the strength of the recovery process is adjusted. According to the first to fourth embodiments, when the adjusted data is stored, the difference image will be rewritten. However, if the data is further adjusted after adjusting the data, the last stored image is adjusted. In this case, adjustment may be difficult because the adjustment reference changes.

  For example, after the user saves the recovery process with a strength of “0.5”, the user may want to make the recovery process a little stronger. In this case, the user has to set the strength of the recovery process to “1.2” instead of “0.6”, for example. This characteristic makes the user's intuitive operation difficult. Furthermore, there is a possibility that the difference image gradually deteriorates due to the truncation process occurring during adjustment. Therefore, in this embodiment, the strength of the recovery process is stored together with the difference image.

  The method of the present embodiment will be described with reference to FIG. The method of this example is similar to that of Example 1, and the description of the same steps is omitted. In step S310, the encoding unit 105 determines the strength of the recovery process. At this time, the encoding unit 105 considers the imaging conditions acquired in step S302, user instructions, and the like. For example, when the lens type, the shooting distance, the zoom position, the aperture, and the light source are the same, the strength of the recovery process may be weaker when the shooting mode is the portrait mode than when the shooting mode is the landscape mode. As an example, the adjustment value may be set to 1.0 in the landscape mode, and the adjustment value may be set to 0.75 in the portrait mode. As will be described in Example 10, the adjustment value indicates the strength of the recovery process. Thereafter, the encoding unit 105 stores the JPEG image acquired in step S309, the difference image acquired in step S308, and the above adjustment values.

  In this embodiment, the adjustment value is determined in consideration of the shooting mode. However, the digital camera may analyze the captured image, and the encoding unit 105 may determine the adjustment value according to the analysis result. For example, when one face detected by face detection occupies a predetermined ratio or more with respect to the size of the captured image, the adjustment value may be decreased. If it is determined by scene analysis of the captured image that the image is captured indoors, the adjustment value may be decreased in consideration of the possibility of large noise.

  In this embodiment, the adjustment value is determined when the digital camera stores an image. However, the method of this embodiment can also be used when the information processing apparatus reads a JPEG image generated according to this embodiment, performs readjustment of recovery processing, and saves it again. That is, the adjustment value obtained by readjusting the recovery process may be overwritten on the stored adjustment value. According to the present embodiment, when the strength of the recovery process is readjusted, the adjustment reference can be made constant. According to the previous example, when the strength of the recovery process is desired to be stronger than “0.5”, the user may specify “0.6”.

[Example 6]
In the first to fifth embodiments, the correction process using the recovery filter calculated from the OTF data has been described. When the restoration process is performed by the methods of the first to fifth embodiments, both the phase component (PTF component) and the amplitude component (MTF component) of the image are corrected. Therefore, when the recovery process is adjusted using the difference data by the methods of Embodiments 1 to 5, the recovery process of the PTF component and the MTF component has the same strength. The weaker the recovery process, the more asymmetric blur appears, such as color shift due to lateral chromatic aberration and image flow due to coma aberration.

  FIG. 9 shows a point image when only the phase shift (PTF component) is corrected. Hereinafter, the correction of the phase shift is referred to as PTF recovery. The point images MTF and PTF shown in FIG. 9C are shown in FIGS. 9A and 9B, respectively. Also, the MTF and PTF of the point image shown in FIG. 9F are shown in FIGS. 9D and 9E, respectively. The image of FIG. 9F is obtained by correcting the PTF of the image of FIG. Although the image of FIG. 9C and the image of FIG. 9F are blurred, in the image of FIG. 9F, the asymmetry is corrected better than the image of FIG. 9C. The image flow is better suppressed. FIG. 9H is obtained by correcting the PTF of the point image of FIG. Also in this case, the asymmetry is corrected by correcting the PTF.

  Therefore, in the present embodiment, the strength of the recovery correction of the phase shift (PTF) caused by the asymmetry of the aberration is fixed, and only the strength of the MTF recovery processing can be adjusted. The method of this embodiment is the same as that of the first example, and will be described using the flowchart of FIG. The description of the same processing as in the first embodiment is omitted.

In step S303, the recovery processing unit 204 acquires a recovery filter that matches the imaging condition acquired in step S302. In this embodiment, the recovery processing unit 204 acquires both the PTF recovery filter and the MTF recovery filter. There are various methods for dividing the recovery filter into a PTF recovery filter and an MTF recovery filter. For example, when applying an inverse filter, the following equation can be used.
(PTF recovery filter) = (absolute value of OTF) / OTF = MTF / OTF
(MTF recovery filter) = 1 / (absolute value of OTF) = 1 / MTF
Similarly, the recovery filter can be divided by applying a Wiener filter.

  In step S307, the recovery processing unit 204 applies the PTF recovery filter acquired in step S303 to the interpolated image acquired in step S306 to create an image Ip in which only the PTF is recovered (previous). Processing means). Further, the recovery processing unit 204 applies the MTF recovery filter acquired in step S303 to the image Ip to create an image Ipm in which both the MTF and the PTF are recovered. The processing performed by the recovery processing unit 204 corresponds to performing preprocessing for updating an interpolated image by applying a PTF recovery filter and acquiring an image in which both the PTF and the MTF are recovered.

  In step S308, the recovery processing unit 204 creates a difference image by obtaining a difference between the image Ipm and the image Ip for each pixel. The difference image of the present embodiment indicates the amount of MTF correction. In step S309, the encoding unit 105 performs JPEG compression on the image Ip. In step S310, the encoding unit 105 stores the JPEG image created in step S309 and the difference image created in step S308. At this time, the encoding unit 105 displays flag information indicating that the stored difference image is a difference between an image in which both the PTF and the MTF are recovered and an image in which only the PTF is recovered, the JPEG image and the difference Save with image. By using the method of this embodiment, even if the MTF recovery processing is weakened, it is possible to suppress image quality deterioration such as image flow due to aberration asymmetry.

[Example 7]
In the processing of the present embodiment, chromatic aberration is considered. In FIG. 10, (a) and (b) are MTFs of different color components before recovery. By correcting MTFs (a) and (b), MTF (c) is obtained. When the recovery process is weakened, MTF (c) approaches MTF (a) or (b) depending on the color component. That is, when the recovery process is weak, the MTF shifts between different color components. Then, since the blur shape differs for each color component, chromatic aberration occurs. Therefore, in this embodiment, the difference image is generated after correcting the chromatic aberration. Further, if the PTF is deviated between the color components, the shape of the blur is different for each color component, so that chromatic aberration also occurs in this case. Therefore, in this embodiment, after the PTF correction and the chromatic aberration correction are performed, a difference image before and after the MTF correction is created. However, the PTF restoration process and the MTF restoration process do not need to be performed independently as in the sixth embodiment, and a difference image may be generated as in the first embodiment after chromatic aberration correction is performed. .

  The processing in the seventh embodiment is shown in FIG. In the flowchart of FIG. 7, the same reference numerals are assigned to steps performed in the same manner in the first embodiment. In steps S301 and S302, processing is performed as in the first embodiment. In step S <b> 303, the recovery processing unit 204 acquires a PTF recovery filter that corrects phase shift, a filter that corrects chromatic aberration, and an MTF recovery filter. The filter that corrects chromatic aberration corrects an image so that the MTF shift between the color components is small. For example, the filter for correcting the chromatic aberration can bring MTF (a) in FIG. 10 closer to MTF (b).

There are various methods for creating a filter for correcting chromatic aberration. For example, an inverse filter can be applied to create a filter as follows.
Chromatic aberration correction filter = (MTF of (b)) / (MTF of (a))
In addition, a filter can be created by applying a Wiener filter.

  In step S304, the recovery processing unit 204 determines the image type to be output, as in the first embodiment. When RAW data is output, the process proceeds to step S1207, and the RAW image data of the imaging data acquired in step S301 and the PTF recovery filter, chromatic aberration correction filter, and MTF recovery filter acquired in step S303 are stored. The If JPEG data is output, the process proceeds to step S306.

  In step S306, an interpolated image is acquired as in the first embodiment. In step S1201, the recovery processing unit 204 uses the PTF recovery filter acquired in step S303 to create the image Ip from which the PTF has been recovered. In step S1202, the recovery processing unit 204 creates an image Ipc by applying a chromatic aberration correction filter to the image Ip. The image Ipc is an image in which the phase shift and chromatic aberration are corrected. In step S1203, the recovery processing unit 204 creates an image Ipcm by applying an MTF recovery filter to the image Ipc. The image Ipcm is an image in which phase shift, chromatic aberration, and reduction in MTF are corrected.

  In step S1204, the recovery processing unit 204 creates a difference image between the image Ipcm and the image Ipc by calculating a difference between pixel values for each pixel. That is, the difference image indicates a difference between pixel values before and after MTF correction for an image in which phase shift and chromatic aberration are corrected. In step S1205, the encoding unit 105 performs JPEG compression on the image Ipc.

  In step S310, the encoding unit 105 stores the JPEG image generated in step S1205 and the difference image created in step S1204. Here, the encoding unit 105 further displays flag information indicating that the stored difference image is a difference between an image in which PTF, MTF, and chromatic aberration are recovered, and an image in which PTF and chromatic aberration are recovered. Save with JPEG image and difference image. According to the method of the present embodiment, it is possible to adjust the strength of the MTF recovery process for an image with little chromatic aberration. Even if the recovery process is weakened, chromatic aberration does not stand out.

[Example 8]
In many cases, the point image obtained by recovering only the PTF as in the sixth embodiment does not become circular even though it is point-symmetric. For example, a point image as shown in FIG. 9F or FIG. 9H is obtained. The point images are not circular because the MTF shapes in the azimuth direction are different. The two azimuth directions may be, for example, a sagittal direction and a meridional direction. The azimuth direction is a general term for all directions and includes a sagittal direction, a meridional direction, and a direction at an arbitrary angle θ. For example, when the MTFs in the sagittal direction and the meridional direction have the shapes of (a) and (b) in FIG. 10, respectively, a point image similar to FIG. 9 (F) is obtained. If the MTFs in the sagittal direction and the meridional direction match, the point image becomes a circle.

  In this embodiment, the mismatch of MTF in the azimuth direction (MTF aberration in the azimuth direction) is corrected. Then, after correcting the mismatch of the MTF, recovery processing is performed, and difference images before and after the recovery processing are created. The process of the present embodiment is almost the same as the process of the seventh embodiment. In step S303 in FIG. 7, the recovery processing unit 204 acquires a filter (hereinafter referred to as an azimuth correction filter) that corrects the mismatch of the MTFs in the azimuth direction instead of the chromatic aberration correction filter. In step S1202, the recovery processing unit 204 may create the image Ipc by applying an azimuth correction filter to the image Ip.

  As an alternative method, the recovery processing unit 204 can acquire an azimuth correction filter together with the chromatic aberration correction filter in step S303. In step S1202, the recovery processing unit 204 may create the image Ipc by applying a chromatic aberration correction filter and an azimuth correction filter to the image Ip.

  The azimuth correction filter can be created by applying an inverse filter as in the seventh embodiment. For example, it may be obtained as (target MTF) / (MTF of angle to be corrected). An azimuth correction filter can also be created by applying a Wiener filter or the like. According to the method of the present embodiment, the JPEG image becomes an image having a circular blur. Therefore, a beautiful blurred image can be obtained even if the user weakens the recovery process by the subsequent editing.

[Example 9]
In Examples 1 to 8, data could be saved in two types of formats (RAW data and JPEG). However, the demosaiced image may be stored as a compressed image or an uncompressed image. Therefore, in this embodiment, a method for saving an image in three types of formats (RAW data, JPEG, and TIFF storing an uncompressed image) will be described. The method described in the present embodiment is a modification of the method of the first embodiment so that an uncompressed image after demosaic can be stored. However, it will be apparent to those skilled in the art that any of the methods of Examples 2-8 can be similarly modified.

  The method of this embodiment will be described with reference to FIG. Steps S301 to S308 can be performed in the same manner as in the first embodiment. In step S309, the encoding unit 105 determines whether the image is stored in the JPEG format or the TIFF format from the shooting condition acquired in step S302. When the image is stored in the JPEG format, the encoding unit 105 may perform JPEG compression as in the first embodiment. In step S310, the JPEG image and the difference image are saved.

  If it is determined in step S309 that the image is stored in the TIFF format, the encoding unit 105 stores the data in the TIFF format in step S310. Specifically, the encoding unit 105 may store the uncompressed data of the interpolated image obtained in step S306 and the difference data obtained in step S308 in the TIFF file format format. In this embodiment, the TIFF format is adopted as an example of an uncompressed image, but other uncompressed image formats may be adopted. The storage format may be determined from one or more uncompressed image formats, one or more compressed image formats, and a RAW data format.

[Example 10]
In the present embodiment, a method for adjusting the strength of the recovery process using the image data stored in the first embodiment will be described. The processing in this embodiment can be performed by image processing software such as a development processing application, or can be performed by a general image processing apparatus. The method of this embodiment is shown in the flowchart of FIG. In this embodiment, description will be made assuming that the image processing apparatus shown in FIG. 4 performs processing.

  In step S1401, the input unit 402 acquires the image data 401 stored in the first embodiment. Then, the determination unit 403 determines whether the image data 401 is in RAW format or JPEG format. Various methods can be used to determine the data format. For example, the data format can be determined by the extension, or the data format can be determined by reading the format information stored at the beginning of the file. Further, when the image data 401 has a format that is not supported in the processing of this embodiment, a warning can be issued to the user.

  If the image data 401 is in the RAW format, the process proceeds to step S1402. In step S1402, the developing unit 404 generates an interpolated image by performing demosaic processing on the RAW data included in the image data 401. In step S1403, the developing unit 404 acquires a recovery filter included in the image data 401. In step S1404, the recovery unit 406 generates an image obtained by applying a recovery filter to the interpolated image as a recovered image.

  On the other hand, if the image data is in JPEG format, the process proceeds from step S1401 to step S1405. In step S1405, the decoding unit 405 generates a decoded image by decoding the JPEG image included in the image data 401. In the present embodiment, this decoded image corresponds to an interpolated image before performing the recovery process (step S307) in the first embodiment. However, the decoded image is obtained by JPEG compression of the interpolation image and further decoding. Since JPEG compression is lossy compression, the decoded image does not completely match the interpolated image. In step S1406, the decoding unit 405 acquires a difference image included in the image data 401. In step S1407, the recovery unit 406 acquires a composite image obtained by adding the pixel values of the decoded image and the difference image as a recovered image. This restored image corresponds to the restored image generated in step S307 of the first embodiment.

  In step S1408, the recovery unit 1408 determines whether to change the strength of the recovery process. For example, the recovery unit 1408 may display a recovery image via a display unit (not shown) and acquire a user input indicating whether to change the strength of the recovery process. The recovery unit 1408 can also make a determination by performing image analysis on the recovered image. If the strength of the recovery process is not changed, the recovery unit 406 outputs the recovered image as an adjusted image in step S1413. Various image output methods can be employed. For example, the recovery unit 406 can store the recovered image with or without compression. The recovery unit 406 may store the JPEG image (or the interpolated image obtained in step S1402) included in the image data 401 together with the recovery coefficient acquired in step S1409.

  In the case of changing the strength of the recovery process, the process proceeds to step S1409. In step S1409, the recovery unit 406 acquires the strength of the recovery process. The strength of the recovery process can be expressed as a coefficient, for example. When the coefficient is less than 1 greater than 0, the recovery process is weakened. When the coefficient is greater than 1, the recovery process is strengthened. The strength of the recovery processing may be input by the user as in step S1418, or may be determined by image analysis.

  In step S1410, the recovery unit 406 determines whether the image data 401 is RAW data or JPEG data. When the image data 401 is RAW data, the recovery unit 406 corrects the recovery filter acquired in step S1403 in step S1411. The recovery filter can be corrected by the method described in the second embodiment. Then, the process returns to step S1404. In step S1404, a recovered image is generated using the corrected recovery filter and the interpolated image.

  If it is determined in step S1410 that the image data 401 is JPEG data, the recovery unit 406 corrects the difference image acquired in step S1406 in step S1412. Specifically, the coefficient acquired in step S1409 may be applied to each pixel value of the difference image. For example, if the coefficient is 0.5, each pixel value of the difference image may be halved. If the coefficient is 2, each pixel value of the difference image may be halved. Then, the process returns to step S1407. Return processing. In step S1407, a restored image is generated using the corrected difference image and the decoded image.

  According to this embodiment, it is possible to easily adjust the strength of the recovery process for both image data saved as RAW data and JPEG-compressed image data. In this embodiment, the recovery process is performed on the demosaiced image. However, depending on the type of recovery process, a recovery process for an image before demosaicing can be incorporated, or a recovery process can be incorporated in the demosaic process.

  When the trimming information is attached to the JPEG image as in the fourth embodiment, the recovery unit 406 may extract a difference image corresponding to the pixel position indicated by the trimming information. Then, a recovered image may be generated from the extracted difference image and decoded image. Further, as in the fifth embodiment, the strength of the recovery process may be stored together with the difference image or the filter coefficient. In this case, the developing unit 404 reads the strength of the recovery process in step S1403, and corrects the recovery filter coefficient as in step S1411. In step S1406, the decoding unit 405 may read the strength of the recovery process and correct the difference image as in step S1412. In the present embodiment, the method of adjusting the strength of the recovery process using the image data created in the first embodiment has been described. However, as will be apparent to those skilled in the art, even if the image data created by the methods of Embodiments 2 to 9 is used, the image data can be read and the strength of the recovery process can be adjusted by the same processing. .

[Other Examples]
In Examples 1-9, a recovery filter or OTF was saved along with the RAW data. However, information indicating the storage destination of the recovery filter or OTF can also be saved. For example, a URL indicating the storage location of the recovery filter corresponding to the RAW data may be saved. In this case, for example, in step S1403 of the tenth embodiment, the developing unit 404 may read the recovery filter from the position indicated by the URL.

  Also, the file creation methods of the first to ninth embodiments can be combined. For example, when the sixth, seventh, and eighth embodiments are combined, an image stored as a JPEG image is an image in which a phase shift, a difference in MTF between color components, and a difference in MTF in the azimuth direction are corrected. is there. Then, the difference between the image obtained by performing the recovery process (specifically, the process of correcting the decrease in MTF) on the corrected image and the corrected image is stored as a difference image. The

  In Examples 1-9, the image format was selected from RAW, JPEG, and TIFF for ease of explanation. However, RAW format image data and JPEG format (or TIFF format) image data can also be saved simultaneously. In this case, RAW data, JPEG data (or TIFF data), a recovery filter (or OTF data), and difference data may be stored.

  In the first to ninth embodiments, the difference image can be compressed and the compressed difference image can be stored. Since the recovery process is a blur correction process, the information amount of the difference image becomes large at the edge portion of the subject. In other words, the pixel value of the difference image is close to 0 except for the edge portion. Therefore, when compressing a differential image, it is considered that the compression rate increases even if lossless compression is employed. By compressing the difference image, it is possible to suppress an increase in the file size when the recovery process can be adjusted later. It is also possible to store irreversibly compressed difference data. In this case, the file size is further reduced.

  The recovery filter 202 stored in the ROM 108 may also be compressed in some form. In this case, the recovery processing unit 204 may expand the compressed recovery filter. The ROM 108 may store OTF data or PSF data necessary for calculating the recovery filter instead of the recovery filter. In this case, the recovery processing unit 204 may calculate a recovery filter from the OTF data or PSF data. Further, the filter coefficient is not stored together with the RAW data, but OTF data (or MTF and PTF data) or PSF data matching the imaging conditions may be stored.

  In the first to ninth embodiments, the signal processing unit 103 of an imaging apparatus such as a digital camera performs processing. However, a computer having a CPU that controls the entire computer and a memory that stores data may perform the processes of the first to ninth and tenth embodiments. In order to execute the functions of the embodiments in this computer, each function may be expressed by a computer program such as a development processing application and read into the computer. Specifically, a program stored in a storage medium or the like is loaded into the memory, and the CPU executes the loaded program.

  When using the development processing application, a file for storing a photographed image is input. When a file conforming to the Exif file format is used, the shooting conditions may be stored according to the Exif file format. In this way, the development processing application can acquire the shooting conditions. For example, the aperture value can be acquired with reference to an ApproachValue tag or an Fnumber tag in the Exif IFD. The subject distance can be acquired with reference to the Subject Distance tag in the Exif IFD. The focal length of the lens can be acquired with reference to the FocalLength tag. The camera model information can be acquired by referring to the Model tag in the Exif IFD. In the standard Exif IFD, since the lens type information tag is not defined, the lens type information can be acquired with reference to the Makernotes tag or the like. Further, the strength of the default recovery process may be determined according to the shooting scene type acquired by referring to the SceneCaptureType tag. By acquiring a recovery filter using these pieces of information, it is possible to create an image data file similar to those in the first to eighth embodiments.

Claims (9)

Acquisition means for acquiring a captured image obtained by imaging, and a correction filter used for performing processing for correcting the blur caused by the aberration of the optical system of the imaging apparatus used for the imaging on the captured image; ,
Correction means for generating a corrected image by applying the correction filter to the captured image;
Difference calculating means for generating a difference image between the captured image and the corrected image;
An image processing apparatus comprising: a storage unit that stores the captured image or the corrected image and the difference image. Imaging data output from the imaging element by imaging, a captured image obtained by performing demosaic processing on the imaging data, and processing for correcting blur caused by aberration of the optical system of the imaging apparatus used for the imaging An acquisition means for acquiring a correction filter used for performing on a captured image;
Setting means for setting a first mode for storing the imaging data or a second mode for storing the image after the demosaic processing;
Correction means for generating a corrected image by applying the correction filter to the captured image;
Difference calculating means for generating a difference image between the captured image and the corrected image;
In the first mode, the imaging data and the correction filter are stored, and in the second mode, storage means for storing the captured image or the correction image and the difference image is provided. An image processing apparatus.   The image processing apparatus according to claim 1, wherein the storage unit compresses and stores the difference image. Storage means for storing the correction filter corresponding to each of a plurality of imaging conditions;
The acquisition unit acquires an imaging condition at the time of imaging, and selects and acquires the correction filter corresponding to the imaging condition from the storage unit. An image processing apparatus according to 1. The correction filter is a filter that corrects an amplitude component of blur caused by the aberration of the optical system of the imaging device,
Before generating the corrected image, the phase of the blur caused by the aberration of the optical system of the imaging apparatus is corrected for the captured image acquired by the acquisition unit, thereby updating the captured image. The image processing apparatus according to claim 1, further comprising preprocessing means.   Before the corrected image is generated, at least one of the correction of MTF aberration in the azimuth direction and the correction of chromatic aberration is performed on the captured image acquired by the acquisition unit, thereby updating the captured image. The image processing apparatus according to claim 1, further comprising a processing unit. An image processing method performed by an image processing apparatus,
An acquisition step of acquiring a captured image obtained by imaging, and a correction filter used for performing processing for correcting the blur caused by the aberration of the optical system of the imaging apparatus used for the imaging on the captured image; ,
A correction step of generating a corrected image by applying the correction filter to the captured image;
A difference calculating step of generating a difference image between the captured image and the corrected image;
An image processing method comprising: an output step of storing the photographed image or the corrected image and the difference image. An image processing method performed by an image processing apparatus,
Imaging data output from the imaging element by imaging, a captured image obtained by performing demosaic processing on the imaging data, and processing for correcting blur caused by aberration of the optical system of the imaging apparatus used for the imaging An acquisition step of acquiring a correction filter used for performing on a captured image;
A setting step for setting a first mode for storing the imaging data or a second mode for storing the image after the demosaic processing;
A correction step of generating a corrected image by applying the correction filter to the captured image;
A difference calculating step of generating a difference image between the captured image and the corrected image;
In the first mode, the imaging data and the correction filter are output, and in the second mode, an output step of storing the captured image or the corrected image and the difference image is provided. An image processing method characterized by the above.   The computer program for functioning a computer as each means which the image processing apparatus of any one of Claims 1 thru | or 6 has.