JP4983541B2 - Program, image processing apparatus and camera - Google Patents

Description

The present invention relates a program, an image processing apparatus and a camera.

2. Description of the Related Art Conventionally, there has been known an image processing apparatus that corrects an aberration (distortion aberration or lateral chromatic aberration) depending on a distance (image height) from an optical axis center with respect to a digital image captured by an electronic camera or the like. As an example, Patent Document 1 discloses a configuration of an imaging apparatus that corrects distortion of an image.
Japanese Patent No. 3035416

  Incidentally, the amount of distortion and lateral chromatic aberration vary depending on the image height, and the outer edge of the image after correcting the aberration depending on the image height is not a straight line. Therefore, if an appropriate scaling process is not performed on an image in which aberration depending on the image height is corrected, pixel loss (image loss) may occur in the corrected image. Conversely, when it is desired to leave all the images before correction, a part of the image may disappear after correction and the angle of view may decrease. In Patent Document 1, there is no mention of a method of determining a scaling factor for suppressing the above-described image defect and the like, and improvement on these problems has been demanded.

  The present invention is to solve the above-mentioned problems of the prior art. SUMMARY OF THE INVENTION An object of the present invention is to provide means for suppressing image loss and field angle reduction when correcting aberrations depending on image height.

A program according to an embodiment of the present invention includes an input step, a function determination step, an image height ratio determination step, a scaling factor calculation step, and an image correction step for a computer having a data reading unit and a calculation unit. Let it run. In the input step, the data of the image captured through the imaging optical system is read from the data reading unit into the arithmetic unit together with the data of the imaging optical system at the time of capturing the image. In the function determination step, the calculation unit determines a correction function for correcting aberration depending on the image height. In the image height ratio determination step, the calculation unit determines the first reference image height ratio using a smaller value of the image height ratios in the vertical direction or the horizontal direction of the image, and determines the first reference image height ratio based on the predetermined value . Determine the 2 reference image height ratio . In the variable magnification calculation step, the calculation unit obtains an aberration amount of the aberration depending on the image height ratio in the range from the first reference image height ratio to the second reference image height ratio with respect to the correction function, and the aberration in the above range. The magnification of the image corresponding to the minimum or maximum value is obtained . In the image correction step, the calculation unit corrects the aberration depending on the image height on the image. Further, in the image correction step, the calculation unit executes an image scaling process using the above scaling ratio.

An image processing apparatus according to another aspect of the present invention includes an input unit, a function determining unit, an image height ratio determining unit, a scaling factor calculating unit, and an image correcting unit. The input unit reads data of an image captured through the photographing optical system together with data of the photographing optical system at the time of capturing the image. The function determining unit determines a correction function for correcting aberration depending on the image height. The image height ratio determination unit determines the first reference image height ratio using a smaller value of the image height ratios in the vertical direction or the horizontal direction of the image, and based on the predetermined value, the second reference image height ratio Determine the ratio. The magnification calculator calculates an aberration amount of aberration depending on the image height ratio in the range from the first reference image height ratio to the second reference image height ratio with respect to the correction function, and the minimum value of the aberration amount in the above range Alternatively, the magnification of the image corresponding to the maximum value is obtained. The image correction unit corrects the aberration depending on the image height ratio on the image, and executes an image scaling process using the scaling factor.

  In the present invention, the magnification ratio of the image corresponding to the amount of aberration is obtained using the first reference image height and the second reference image height for the correction function, and the image scaling process is executed using the magnification ratio. As a result, it is possible to suppress a lack of image and a decrease in the angle of view after correcting the aberration depending on the image height, and a preferable image can be obtained.

<Description of Configuration of Apparatus in Embodiment>
FIG. 1 shows a configuration example of an image processing system according to an embodiment. The image processing system in FIG. 1 includes a single-lens reflex electronic camera 1 capable of exchanging lenses and an image processing apparatus 2.

  First, the configuration of the electronic camera 1 will be described. The electronic camera 1 has a lens unit 11 and a camera body 12. The lens unit 11 is replaceably mounted on the camera body 12 via mounts 13 and 14 having electrical contacts. Then, when the camera body 12 and the lens unit 11 are connected, an electrical connection by the electrical contacts of the mounts 13 and 14 is established.

  The lens unit 11 houses a lens-side mount 13, a photographing optical system 15, a lens microcomputer 16, and a diaphragm (not shown). For simplicity, in the lens unit 11 of FIG. 1, the photographing optical system 15 is represented by a single lens. Further, the lens microcomputer 16 acquires the focal position (distance to the in-focus subject) and the focal length of the photographing optical system 15 by an encoder (not shown). The lens microcomputer 16 outputs data to the camera body 12 via the electrical contacts of the mount 13.

  On the other hand, the camera body 12 includes a camera-side mount 14, a mirror box unit 17, an image sensor 18, an AFE 19, an image processing unit 20, a memory 21, a recording I / F 22, an external I / F 23, A control unit 24 and a bus 25 are included. Here, the image processing unit 20, the memory 21, the recording I / F 22, the external I / F 23, and the control unit 24 are connected to each other via a bus 25. The electrical contacts of the mount 14 and the mirror box unit 17 are connected to the control unit 24, respectively.

  Inside the mirror box unit 17, a main mirror 17a pivotally supported is disposed. The mirror box unit 17 switches between an observation state in which the light beam is guided to the finder optical system (not shown) and a retracted state in which the light beam is guided to the image sensor 18 by driving the main mirror 17a.

  The image sensor 18 photoelectrically converts the light beam that has passed through the photographic optical system 15 to generate an analog image signal of the subject image. Note that the output of the image sensor 18 is connected to the AFE 19.

  The AFE 19 is an analog front end circuit that performs analog signal processing on the output of the image sensor 18. The AFE 19 performs correlated double sampling, image signal gain adjustment, and image signal A / D conversion. The output of the AFE 19 is connected to the image processing unit 20.

  The image processing unit 20 performs various types of image processing (color interpolation processing, gradation conversion processing, contour enhancement processing, white balance adjustment, etc.) and image compression / decompression processing on the digital image signal. The memory 21 temporarily stores image data in the pre-process and post-process of image processing.

  The recording I / F 22 incorporates a connector to which the storage medium 4 can be detachably attached. The recording I / F 22 executes data writing / reading with respect to the storage medium 4 attached to the connector. The storage medium 4 is composed of a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is illustrated as an example of the storage medium 4.

  The external I / F 23 performs data transmission / reception with the image processing apparatus 2 via the wireless or wired communication line 3.

  The control unit 24 is a processor that performs overall control of the electronic camera 1. For example, the control unit 24 executes various processes related to capturing a recorded image. Note that the control unit 24 can also execute aberration correction processing of a recorded image and image scaling correction by executing a program described later.

  Here, the operation of the electronic camera 1 when capturing a recorded image will be briefly described. In response to pressing of a release button (not shown), the control unit 24 places the main mirror 17a in the retracted state and drives the image sensor 18 to image the subject. The recorded image data is subjected to predetermined processing by the AFE 19 and the image processing unit 20.

  On the other hand, at the time of capturing a recorded image, the control unit 24 receives from the lens microcomputer 16 data on the recording conditions of the recorded image (the type of the lens unit 11 and the focal position and focal length of the imaging optical system 15 at the time of imaging). Then, the control unit 24 associates the recording image data with the above photographing condition data to generate an image file in a predetermined format. The image file also includes information indicating the size of the image sensor 18 and information indicating the size of the recorded image. The image file is recorded on the storage medium 4 by the control unit 24 via the recording I / F 22.

  Next, the configuration of the image processing apparatus 2 will be described. The image processing apparatus 2 is, for example, a personal computer in which an application program for photo retouching is installed, and can perform image processing on recorded image data acquired from the electronic camera 1.

  The image processing apparatus 2 includes a recording I / F 31, a communication unit 32, a storage unit 33, and a CPU 34. The recording I / F 31, the communication unit 32, and the storage unit 33 are each connected to the CPU. In addition, an operation unit 35 and a monitor 36 are connected to the image processing apparatus 2.

  The recording I / F 31 has a connector for connecting the storage medium 4. Then, the recording I / F 31 reads the image file from the storage medium 4 connected to the connector. The communication unit 32 executes data transmission / reception with the electronic camera 1 connected via the communication line 3.

  The storage unit 33 stores the application program and a data group of aberration information generated for each type of the lens unit 11. The storage unit 33 can also record the data of the recorded image after image processing.

  Here, as an example, the aberration information includes function information obtained by approximating the distortion aberration and the lateral chromatic aberration of the lens unit 11 with functions of the focal position and the focal length of the photographing optical system 15. Each aberration information is generated in advance by the manufacturer of the lens unit 11 based on design data and actual measurement data of the photographing optical system 15.

  The CPU 34 executes the above-described application program, and performs various arithmetic processes associated with aberration correction processing and image magnification correction of recorded image data. The operation unit 35 receives various operation inputs related to the application program from the user. Under the control of the CPU 34, an operation screen based on an application program, a display screen showing images before and after image processing, and the like are displayed on the monitor 36.

<Description of Aberration Correction Processing in Embodiment>
Next, an example of aberration correction processing in the present embodiment will be described with reference to FIG. In the following example, a case will be described in which the CPU 34 of the image processing apparatus 2 performs an aberration correction process on a recorded image by executing an application program.

  Step S101: The CPU 34 of the image processing apparatus 2 acquires an image file including data of a recording image to be corrected from the electronic camera 1. In S101, the transfer of the image file between the electronic camera 1 and the image processing apparatus 2 is realized by data communication via the communication line 3 or transmission / reception of the storage medium 4.

  Step S102: The CPU 34 determines a correction function for correcting the aberration depending on the image height, based on the photographing condition data included in the image file (S101). Specifically, first, the CPU 34 reads out aberration information data corresponding to the type of the lens unit 11 from the storage unit 33. Then, the CPU 34 determines a correction function using the above aberration information and the focal position and focal distance at the time of imaging.

Here, assuming that the image height of the ideal image point is Y and the image height of the actual image point is Y ₀ , the distortion amount D applied to the image by the photographing optical system 15 is expressed by the following equation (1).

D = 100 × (Y−Y ₀ ) / Y ₀ [%] (1)
The above-described distortion amount D varies depending on the type of the lens unit 11 (shooting optical system 15), and changes according to the focal position and focal length at the time of image capturing even with the same shooting optical system 15. Further, the distribution of the distortion amount D on the image can generally be sufficiently approximated by a function of only the image height ratio r (r = image height / maximum image height) as shown in the following equation (2).

D (r) ≡ar ⁴ + br ³ + cr ² (2)
The coefficients a, b, and c in the above equation (2) can be determined almost uniquely if the aberration information data corresponding to the type of the lens unit 11 and the focal position and focal length at the time of imaging are determined. Is possible. Therefore, it can be seen that the CPU 34 can determine each coefficient related to the correction function based on the data of the photographing condition of the image file.

However, although only the correction of distortion aberration is mentioned in the above example, it is possible to correct the lateral chromatic aberration at the same time. For example, when the amount of deviation of the red (R) pixel with respect to the green (G) pixel at the image height ratio r obtained in the same manner as the amount of distortion is δD _R (r), the distortion amount D _R ( r) can be defined as the following formula (4). Similarly, when the deviation amount of the blue (B) pixel with respect to the G pixel at the image height ratio r is obtained in the same manner as the distortion aberration amount is δD _B (r), the distortion amount D _B (r) of the B pixel. Can be defined as in equation (5) below.

D _R (r) ≡D (r) + δD _R (r) (4)
D _B (r) ≡D (r) + δD _B (r) (5)
As described above, it can be seen that the lateral chromatic aberration can be corrected by the same processing as in the case of distortion. For the sake of simplicity, the example of the present embodiment will be described centering on the correction of distortion, but unless otherwise specified, the CPU 34 performs correction of distortion and correction of lateral chromatic aberration simultaneously as distortion correction.

  Step S103: The CPU 34 obtains the magnification of the image in accordance with the amount of aberration (distortion aberration, lateral chromatic aberration) depending on the image height.

  FIG. 3 schematically shows a state in which the distortion correction is performed on the recorded image. For the sake of simplicity, FIG. 3 shows only the upper right portion when the recorded image is divided into upper, lower, left and right.

  When distortion correction is performed on a recorded image, the outer edge of the corrected image is not a straight line. Therefore, for example, when no scaling process is performed when distortion correction is performed, a missing pixel occurs due to lack of an image, or conversely, a necessary portion is cut. Therefore, it is necessary to perform image scaling processing simultaneously with distortion correction.

  For example, in the case of normal barrel-type or pincushion-type distortion aberration correction, the scaling factor may be obtained from the distortion amounts obtained at points A, B, and C shown in FIG. Specifically, in correcting barrel distortion, when eliminating a pixel defect in the corrected image, the distortion amount at the vertical or horizontal central portion of the image (distortion at point A or point B in FIG. 3). Amount) may be used.

  Similarly, in correcting a pincushion type distortion aberration, when eliminating a pixel defect in a corrected image, a distortion amount at the corner of the image (a distortion amount at a point C in FIG. 3) may be used.

  However, in the case of general distortion correction when the Jinkasa type is included, the outer edge of the corrected image can have a more complicated shape. Therefore, if the magnification is changed according to the equations (6) and (7), pixel loss may occur. Therefore, it is necessary to determine a scaling factor with higher accuracy.

(Example of algorithm for determining the scaling factor)
Here, an algorithm of a scaling factor determination method by the CPU 34 will be described in the case of eliminating pixel defects in the corrected image. Specifically, when the expansion / contraction amount of the image is represented by the variation L in FIG. 3, the CPU 34 obtains the minimum value of L on the line segment AC and the line segment BC, and the magnification of the image at the portion forming the minimum value. Can be determined. However, in the example of FIG. 3, L takes a positive direction in the x-axis direction and the y-axis direction. If the corrected image size is smaller than the original recorded image size, the value of L is a negative value.

Consider the change in L between line segments AC. When the horizontal size of the image is XSIZE and the vertical size of the image is YSIZE, the value of L (L _AC ) between the line segments AC can be expressed by the following equation (8).

In the above equation, R _max indicates the maximum image height of the image sensor 18 that captured the recorded image.

On the other hand, let us consider the change in L between the line segments BC. At this time, if the restriction that the aspect ratio between the image after distortion correction and the original recorded image is made constant is added, the value of L between the line segments BC (L _BC ) can be expressed by the following equation (9). it can.

In order to obtain the case where the value of L is minimized from the equations (8) and (9), the CPU 34 further performs the following calculation. First, the CPU 34 compares the image height in the horizontal direction and the image height in the vertical direction (values of XSIZE / 2R _max and YSIZE / 2R _max ), selects the smaller value, and selects the image height ratio r _S. And Then, the CPU 34 obtains a value of Dα that is a variable magnification parameter by the following equation (10). Note that Expression (10) is obtained when the maximum image height of the image sensor 18 that captured the recorded image matches the radius of the image circle of the photographing optical system 15 (the image height ratio r at the maximum image height is 1). If applicable).

  Further, when the maximum image height of the image sensor 18 that has captured the recorded image is smaller than the radius of the image circle of the photographing optical system 15, the CPU 34 calculates the value of Dα in the following manner without using the equation (10). .

Here, the radius of the image circle of the imaging optical system 15 is R _lenz , the pixel pitch of the image sensor 18 (the distance between the centers of two adjacent pixels) is l _sen , and the number of pixels in the horizontal direction of the image sensor 18 is XPIC and the number of pixels in the vertical direction of the image sensor 18 are YPIC. At this time, the CPU 34 can obtain the value of Dα by the following equation (11).

  As a case where the maximum image height of the image sensor 18 is smaller than the radius of the image circle of the photographing optical system 15, the following cases are conceivable. The first example is a case where the size of the image sensor 18 is originally smaller than the radius of the image circle of the photographing optical system 15 (see FIG. 4A). In the second example, the maximum image height of the image sensor 18 matches the radius of the image circle of the imaging optical system 15, but the recorded image corresponds to an image of a partial area of the image sensor 18 (in the case of crop shooting). Yes (see FIG. 4B).

(Another example of algorithm for determining the scaling ratio)
Next, as another example of the algorithm using Expressions (8) to (11), a description will be given of a case where the entire range of the original recorded image is not lost after distortion correction (a case where all reductions in the angle of view are prevented). To do. In this case, the CPU 34 may obtain the maximum value of L on the line segment AC and line segment BC, and determine the magnification of the image at the portion where the maximum value is formed. However, in the case of this other example, pixel loss occurs in the corrected image, and the corrected image has a black edge.

  Specifically, when the image height ratio r is 1, the CPU 34 applies the following equation (12) instead of the equation (10) to obtain the value of Dα. When the maximum image height of the image sensor 18 that has captured the recorded image is smaller than the radius of the image circle of the photographing optical system 15, the CPU 34 applies the following equation (13) instead of the equation (11). The value of Dα is obtained.

  In the image processing apparatus 2 of the present embodiment, the CPU 34 determines the value of Dα from the minimum value of L in the default state. Note that the CPU 34 can change an algorithm for determining the value of Dα in accordance with a user instruction.

  Step S104: The CPU 34 executes correction of aberration depending on the image height and image scaling processing at the same time. Specifically, the CPU 34 uses the value of Dα obtained in S103 to execute distortion correction and scaling processing according to the following equation (14).

  Here, when determining the scaling factor so that there is no pixel loss, the CPU 34 may correct the scaling factor in consideration of the ring pixels necessary for the interpolation process. At this time, the CPU 34 corrects the scaling factor as follows according to each interpolation processing algorithm (range of pixels referred to at the time of interpolation).

For example, in the case of cubic interpolation, the CPU 34 defines D _fac as shown in the following equation (15).

In the case of linear interpolation, the CPU 34 defines D _fac as shown in the following equation (16).

And CPU34 correct | amends said Formula (14) like following Formula (17) using _Dfac defined by Formula (15) or Formula (16). Incidentally, the need to maintain a constant aspect ratio of the image when zooming, values of the above D _fac is sufficient to apply only the smaller of the vertical or horizontal direction of the image.

  When the above processing is executed, ring pixels used for interpolation are secured at the outer edge of the image, so that conditional branching is not necessary in the processing of the outer edge portion of the image during the interpolation processing. As a result, the processing speed of the interpolation process is improved, and the time required for distortion correction in this embodiment can be further shortened.

  Step S105: The CPU 34 records the image data (S104) after distortion correction in the storage unit 33 or the like. This is the end of the description of FIG.

  Hereinafter, the operational effects of this embodiment will be described. In the image processing apparatus 2 of the present embodiment, when performing image distortion correction, the maximum or minimum value of the image expansion / contraction amount within a predetermined section of the correction function is obtained, and the portion that forms this maximum value (or minimum value) To determine the magnification of the image.

  Therefore, in the image processing apparatus 2 of the present embodiment, it is possible to obtain an accurate scaling factor according to the specification without depending on the distortion aberration behavior (barrel type, pincushion type, Jinkasa type) of the photographing optical system 15. .

  FIG. 5 shows an example of the scaling process in this embodiment. In the example of FIG. 5, an original image having a Jinkasa type distortion is corrected. Here, when determining the scaling factor so as to eliminate the pixel loss in the corrected image, the CPU 34 can obtain the scaling factor that eliminates the pixel loss and minimizes the angle of view loss. That is, in this case, the image does not need to be trimmed by the user after distortion correction.

  Further, when the entire range of the original recorded image is not lost after the distortion correction, the CPU 34 can minimize the extra black edges attached to the corrected image. In addition, it cannot be overemphasized that the image processing apparatus 2 of this embodiment can acquire the effect similar to the above also in the case of a barrel type or a pincushion type.

<Supplementary items of the embodiment>
(1) In the above embodiment, the control unit 24 of the electronic camera 1 may execute a program so that the aberration correction of the recorded image and the scaling process are realized only by the electronic camera 1. In the above embodiment, an example has been described in which distortion correction and image scaling are realized by software using a program. However, these configurations may of course be realized by hardware using an ASIC or the like.

  (2) In S <b> 102 of the above-described embodiment, the CPU 34 has described the example in which the correction function is determined based on the shooting condition data. However, in the present invention, the CPU 34 may define a correction function in accordance with a user input from the operation unit 35. In this case, it is possible to cope with the case of using an old lens unit not equipped with the lens microcomputer 16 or a lens unit of another manufacturer.

  (3) In the example of the above embodiment, the CPU 34 may perform the correction of distortion aberration and the correction of lateral chromatic aberration independently.

  (4) In the above embodiment, the configuration in which the aberration information data is stored in advance in the storage unit 33 of the image processing apparatus 2 has been described. For example, the lens microcomputer 16 of the lens unit 11 outputs the aberration information data at the time of shooting. Thus, aberration information data may be stored in the image file. In the above embodiment, the case where the electronic camera 1 is a single lens reflex type has been described. However, the electronic camera 1 may have a lens-integrated type (so-called compact type).

  Many features and advantages of the present invention will become apparent from the detailed description. It is intended that the scope of the claims extend to the features and advantages of the embodiments without departing from the spirit and scope of the claims. Further, those having ordinary skill in the art should be able to easily come up with any improvements and modifications, and limit the scope of the inventive embodiments herein to the configurations described and represented herein. There is no intention to do. Therefore, the scope of the present invention can be based on appropriate improvements and equivalents included in the scope disclosed in the embodiments.

1 illustrates a configuration example of an image processing system according to an embodiment. An example of the flow of aberration correction processing in the present embodiment is shown. The state which performed distortion correction to the recorded image is shown typically. (A): The case where the size of the image sensor is smaller than the radius of the image circle is shown. (B): A case in which crop photography is performed when the maximum image height of the image sensor matches the radius of the image circle is schematically shown. An example of the scaling process in this embodiment is shown.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Electronic camera, 2 ... Image processing apparatus, 3 ... Communication line, 4 ... Storage medium, 11 ... Lens unit, 12 ... Camera body, 15 ... Imaging optical system, 16 ... Lens microcomputer, 18 ... Imaging device, 22 ... Recording I / F, 23 ... external I / F, 24 ... control unit, 31 ... recording I / F, 32 ... communication unit, 33 ... storage unit, 34 ... CPU

Claims (8)

For a computer having a data reading unit and a calculation unit,
An input step of reading data of an image captured through the photographing optical system from the data reading unit into the arithmetic unit together with the data of the photographing optical system at the time of capturing the image;
A function determining step for determining a correction function for correcting aberration depending on image height;
Vertically or by using a smaller value of the lateral image height ratio to determine the first reference image height ratio, image height ratio to determine a second reference image height ratio based on a predetermined value of the image A decision step;
An aberration amount of the aberration depending on the image height ratio in the range from the first reference image height ratio to the second reference image height ratio is obtained with respect to the correction function, and the minimum value or the maximum of the aberration amount in the range is determined. A scaling factor calculating step for obtaining a scaling factor of the image corresponding to the value ;
Correcting the aberration depending on the image height on the image, and performing an image scaling process of the image using the scaling factor; and
Is executed by the calculation unit. The program according to claim 1,
A program comprising a step of correcting the first reference image height ratio and the second reference image height ratio using a maximum image height of the photographing optical system and a maximum image height of an image sensor. In the program according to claim 1 or 2,
The aberration depending on the image height includes at least one of distortion and lateral chromatic aberration . In the program according to any one of claims 1 to 3 ,
The aberration depending on the image height is determined in accordance with lens data indicating characteristics of the photographing optical system and data on a focal position and a focal length of the photographing optical system at the time of imaging. program. In the program according to any one of claims 1 to 4 ,
In the image correction step, the image interpolation process is further executed, and the scaling factor is corrected according to the interpolation process algorithm . An input unit that reads data of an image captured through the photographing optical system together with data of the photographing optical system at the time of capturing the image;
A function determining unit for determining a correction function for correcting aberration depending on the image height;
An image height ratio that determines the first reference image height ratio using a smaller value of the image height ratios in the vertical direction or the horizontal direction of the image, and determines the second reference image height ratio based on a predetermined value. A decision unit;
An aberration amount of the aberration depending on the image height ratio in the range from the first reference image height ratio to the second reference image height ratio is obtained with respect to the correction function, and the minimum value or the maximum of the aberration amount in the range is determined. A scaling factor calculation unit for obtaining a scaling factor of the image corresponding to the value;
Correcting the aberration depending on the image height ratio with respect to the image, and performing an image scaling process using the magnification,
An image processing apparatus comprising: The image processing apparatus according to claim 6.
  An image height ratio correction unit that corrects the first reference image height ratio and the second reference image height ratio using the highest image height of the photographing optical system and the highest image height of the image sensor; An image processing apparatus. An imaging unit that captures an image of a subject via an imaging optical system and generates image data;
An image processing apparatus according to claim 6 or 7,
A camera comprising:

Images