JP6566800B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method.

  Conventionally, in a single-lens reflex camera, light that has passed through a lens is reflected by a mirror and guided to an AF sensor to perform autofocus, and then the mirror is operated to guide the light to an image sensor to capture a subject image. Many structures are adopted.

  On the other hand, there is a light field camera (hereinafter referred to as “LF camera”) that generates image data capable of virtually moving the focal plane (refocusing) (see Non-Patent Document 1).

  Further, as a method of measuring the distance to an object, a TOF method (Time−) that calculates the distance by irradiating the object with a light pulse and measuring the time until the reflected light from the object is received by the image sensor. of-Flight method) (see Patent Document 1).

  According to Patent Document 2, it is possible to calculate the focus shift amount (defocus amount) of the subject image by multiplying the image shift amount obtained from the output of the imaging device for video by a variable coefficient.

Ren Ng and 5 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

Japanese Patent Laid-Open No. 7-110381 JP 2013-171257 A

  In the method of Non-Patent Document 1, when the camera has a reflex structure, in order to allow light to enter the image pickup device for video, a mirror-up behavior is required. Will take extra time. Further, it is a large amount of data and high processing cost to determine whether or not the subject exists at a refocusable position based on the output from the LF camera.

  Accordingly, an object of the present invention is to determine whether or not the subject exists at a refocusable position in the data obtained from the video image sensor based on information different from the output from the video image sensor. It is an object to provide an imaging apparatus and an imaging method that make it possible.

According to one aspect of the present invention, imaging means having a plurality of photoelectric conversion units that respectively receive light beams from subjects that have passed through different pupil regions, and distance information relating to the distance of the subject by detecting light from the subject A distance information generating means for generating the information, and a determination means for determining whether the subject is within a distance range that can be refocused in the post-imaging processing based on the distance information and the pupil division configuration of the plurality of photoelectric conversion units. And a control means for controlling the imaging of the subject by the imaging means according to the result of the determination by the determination means , the imaging means being a first imaging element that captures an optical image of the subject, The distance information generation unit includes a second image pickup device that picks up an optical image of the subject, and includes the first image pickup device configured by the plurality of photoelectric conversion units. The means generates an image signal for performing focus detection by the second image sensor, generates an image shift amount in the second image sensor using the image signal, and based on the image shift amount generating the distance information, a feature that you generate information of the maximum refocus amount of the imaging unit based on the information of the pixel pitch of said first pixel pitch and said second imaging element of the imaging device An imaging device is provided.

  According to another aspect of the present invention, the first imaging device having a plurality of photoelectric conversion units that receive light beams from different pupil regions of the common optical system, a plurality of microlenses, and each microlens, A second imaging device having a plurality of photoelectric conversion units that receive light beams from different pupil regions of the common optical system; and a second imaging device based on a signal from the first imaging device. A determination unit that determines whether or not the subject is within a refocusable range when the subject is imaged, and a control unit that controls imaging by the second image sensor according to a result of the determination by the determination unit; An imaging device is provided.

  According to still another aspect of the present invention, infrared light irradiating means for irradiating infrared light, infrared light receiving means for receiving infrared light and outputting received light information, a plurality of microlenses, A second imaging element corresponding to the microlens and having a plurality of photoelectric conversion units that receive light beams from different pupil regions of the optical system; and generating distance information of the subject based on the light reception information; When the subject is imaged by the second image sensor based on the information, a determination unit that determines whether or not the subject is within a refocusable range, and the second determination unit according to a determination result by the determination unit. There is provided an image pickup apparatus comprising a control unit that controls image pickup by the image pickup element.

According to still another aspect of the present invention, there is provided an imaging method using an imaging device having an imaging unit having a plurality of photoelectric conversion units that respectively receive light beams from subjects that have passed through different pupil regions, A distance range in which the subject can be refocused by processing after imaging based on the step of detecting light and generating distance information related to the distance of the subject, and the distance information and a pupil division configuration of the plurality of photoelectric conversion units And a step of controlling imaging of the subject by the imaging unit according to a result of the determination in the determination step , wherein the imaging unit captures an optical image of the subject. A first imaging element configured by the plurality of photoelectric conversion units, and the step of generating the distance information includes a second imaging unit that captures an optical image of the subject. An image signal for performing focus detection is generated by the image element, an image shift amount in the second image sensor is generated using the image signal, and the distance information is generated based on the image shift amount, imaging method is provided which is characterized that you generate information of the maximum refocus amount of the imaging unit based on the information of the pixel pitch of said first pixel pitch and said second imaging element of the imaging device .

1 is a block diagram of an imaging apparatus according to a first embodiment of the present invention. It is a figure which shows the flowchart of the imaging operation of the imaging device which concerns on 1st Embodiment of this invention. It is a figure which shows the example of transition of the positional relationship of the focus of a focus lens and a to-be-photographed object in imaging | photography with the conventional imaging device. It is a figure which shows the example of transition of the focus of a focus lens, and the positional relationship of a to-be-photographed object in imaging | photography with the imaging device which concerns on 1st Embodiment of this invention. It is a figure which shows the example of the unit pixel cell of the image pick-up element which LF camera has. It is a figure which shows the example of the image from which the parallax differs, and the refocus image acquired by LF camera. FIG. 6 is a diagram (No. 1) for describing a defocus amount calculation method in the imaging device according to the first embodiment of the present invention; FIG. 6 is a diagram (No. 2) for describing the defocus amount calculation method in the imaging device according to the first embodiment of the present invention; FIG. 6 is a diagram (No. 3) for describing the defocus amount calculation method in the imaging device according to the first embodiment of the present invention; It is a block diagram of the imaging device concerning a 2nd embodiment of the present invention. It is a figure which shows the flowchart of the imaging operation of the imaging device which concerns on 2nd Embodiment of this invention. It is FIG. (1) for demonstrating distance calculation using the infrared light in the imaging device which concerns on 2nd Embodiment of this invention. It is FIG. (2) for demonstrating the distance calculation using the infrared light in the imaging device which concerns on 2nd Embodiment of this invention. It is a block diagram of the imaging device concerning a 3rd embodiment of the present invention. It is a block diagram of the imaging device concerning a 4th embodiment of the present invention. It is a figure which shows the detail of the image sensor for still images and the image sensor for moving images in the imaging device which concerns on 4th Embodiment of this invention. It is a block diagram which shows the signal processing part in the imaging device which concerns on 4th Embodiment of this invention. It is a figure for demonstrating calculation of the maximum refocus amount in the camera which can be refocused.

  In the LF camera, the determination as to whether or not the subject exists at a refocusable position, that is, the determination regarding the image shift amount in the image sensor for video can be performed based on the calculation result of the maximum refocus amount.

The calculation of the maximum refocus amount will be described with reference to FIG. FIG. 14 is a diagram for explaining the calculation of the maximum refocus amount. FIG. 14 shows a microlens 1401 and a pixel 1402 of the image sensor, and a solid line reaching the pixel 1402 through the microlens 1401 indicates light incident on the image sensor from the subject. Assuming that the angle resolution Δθ, the number of angle divisions Nθ, and the pixel pitch Δx, the maximum refocus amount dmax is calculated by the following equation (1).
dmax = Nθ · Δx / tan (Δθ) (1)

  In other words, an object existing within the distance dmax before and after the optical axis direction around the focused distance (focus position) when shooting is performed to obtain refocusable image data can be refocused. Can be judged. That is, the image shift amount of the imaging signal of the subject obtained from the output of the video image sensor can be calculated.

  Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

[First Embodiment]
A first embodiment of the present invention will be described with reference to FIGS. 1 to 6C.

  FIG. 1 is a block diagram illustrating a configuration of a main part of the imaging apparatus 100 according to the present embodiment, and omits components that are not directly related to the present embodiment. In FIG. 1, light (incident light) 101 incident from the outside is shown. A lens group 102 including a focus lens constitutes a photographing optical system that forms an optical image of a subject. A mirror 103 provided so as to be able to advance and retract on the optical axis of the lens group 102 changes the optical path of light from the subject that has passed through the lens group 102 when positioned on the optical axis.

  The image pickup device 104 for image picks up an optical image of a subject formed by the lens group 102 including a focus lens. As shown in FIG. 14, the video image sensor 104 has an array of microlenses 1401 including a plurality of microlenses 1401 and a pixel array including a plurality of pixels 1402. In the image pickup device 104 for video, a predetermined number of pixels (divided pixels) 1402 share the same microlens 1401. That is, a predetermined number of pixels 1402 are arranged corresponding to each of the plurality of microlenses 1401. Each of the plurality of pixels 1402 includes a photoelectric conversion unit (photoelectric conversion element). The photoelectric conversion units of the plurality of pixels 1402 in the image pickup device 104 are configured to receive a light beam from a subject that has passed through different pupil regions of the common optical system. In this way, the image pickup device 104 for video has a pupil division configuration that enables generation of refocusable image data formed by the microlens 1401 and the predetermined number of pixels 1402.

  The focus detection imaging element 105 has the same pupil division configuration as the video imaging element 104, and each of the plurality of pixels has a photoelectric conversion unit (photoelectric conversion element). The photoelectric conversion units of the plurality of pixels in the focus detection imaging element 105 are configured to receive light beams from different pupil regions of the common optical system. Note that the pixel pitch of the focus detection image sensor 105 and the pixel pitch of the video image sensor 104 may be the same or different from each other, but usually the pixel pitch of the video image sensor 104 is narrower. . The focus detection image sensor 105 is not necessarily the same as the image sensor 104 and may be an image sensor having a pupil division configuration. For example, an AF sensor may not be used mainly for capturing natural images. When the mirror 103, which is the optical path changing means, is lowered to the optical axis, the incident light 101 is guided to the focus detection image sensor 105, and when the mirror 103 is raised, the incident light 101 is directed to the image sensor 104. Led. The focus detection imaging element 105 detects light from the subject, captures a pair of images formed by a pair of focus detection light fluxes passing through the optical system, and generates a pair of image signals. In this embodiment, the main purpose is to use the signal from the focus detection image sensor 105 for focus adjustment. That is, the imaging apparatus according to the present embodiment has a focus detection unit that performs focus detection based on a signal from the focus detection image sensor 105, and based on the detection result of the focus detection unit, the lens group 102 that is a common optical system. Focus control means for controlling the focus position is provided. The focus control unit controls the focus position of the lens group 102 based on the detection result of the focus detection based on the signal from the focus detection image sensor 105. However, the focus detection image sensor 105 is provided to enable focus adjustment, exposure control, and the like as preparation for the main shooting before and after the main shooting for displaying and recording the video by the video image sensor 104. The use of the signal is not particularly limited.

  The distance measurement information calculation unit 106 calculates and generates position information of the focus lens from the image shift amount of the optical image of the subject imaged by the focus detection image sensor 105. The image shift amount here means a shift amount between a pair of image signals generated in the focus detection imaging element 105. In general, an evaluation value of an image shift amount is obtained using SAD (Sum of Absolute Difference), an image shift amount giving the smallest evaluation value is set as a focal point, and a defocus amount based on the image shift amount is determined. Distance information generating means for performing distance measurement is known. The distance measurement information calculation unit 106 performs distance measurement based on the output of the focus detection imaging element 105 by such a method, and generates distance information related to the distance in the depth direction of the subject. The determination unit 107 determines whether or not the current focus lens position is within a refocusable range from an arbitrary focal position based on the result of the distance measurement information calculation unit 106. That is, the determination unit 107 determines whether or not the subject is within a distance range that can be refocused by the processing after imaging. The imaging control unit 108 controls the imaging system according to the result of the determination unit 107. The signal processing unit 109 processes a signal from the video image sensor 104, and generates a refocus image from image data generated by the video image sensor 104, for example. Each unit of the imaging apparatus 100 is controlled by an imaging control unit 108 (including a CPU). The imaging control unit 108 has a memory for storing a control program, and realizes the function of the imaging apparatus 100 by loading and executing the control program stored in the memory. The instructing unit 112 is, for example, a shutter button, and instructs to take an image.

  The display unit 110 and the recording unit 111 display and record the image signals processed by the signal processing unit 109, respectively. At this time, the signal processing unit 109 converts the image signal into a format suitable for the output destination in advance and outputs it to each unit.

  FIG. 2 is a diagram illustrating a flowchart of the imaging operation of the imaging apparatus 100 according to the present embodiment. This operation is performed by the imaging control unit 108 executing a control program to control each unit shown in FIG.

  In step S <b> 202, the focus detection imaging element 105 receives the light 101 incident from the outside through the lens group 102 and the mirror 103 and generates the above-described pair of image signals.

  In step S <b> 203, the distance measurement information calculation unit 106 obtains an image shift amount from the output signal of the focus detection image sensor 105, and obtains a defocus amount based on the image shift amount. Further, the distance measurement information calculation unit 106 calculates the position of the lens group 102 including the focus lens, the position of the subject, and the distance between the position of the lens group 102 and the position of the subject according to the obtained defocus amount. In the present embodiment, the distance measurement information calculation unit 106 obtains the distance in the depth direction of the subject as described above. However, it is only necessary to know information related to the distance of the subject. The distance in the depth direction may be any form.

Next, in step S <b> 204, the distance measurement information calculation unit 106 determines the maximum refocus amount in the image signal output from the video image sensor 104. That is, the distance measurement information calculation unit 106 calculates the ratio between the pixel pitch of the image pickup device 104 (pitch of the photoelectric conversion unit) and the pixel pitch of the focus detection image pickup device 105 (pitch of the photoelectric conversion unit), and Equation (1). Is used to calculate the maximum refocus amount. For example, when the pixel pitch of the image pickup device 104 is D and the pixel pitch of the focus detection image pickup device 105 is d, the maximum refocus amount in the image pickup device 104 is calculated by the following equation (2). Can do. That is, the maximum refocus amount is determined according to the pupil division configuration of the plurality of photoelectric conversion units of the video image sensor 104 and the pupil division configuration of the plurality of photoelectric conversion units of the focus detection image sensor 105.
dmax = (Nθ · Δx / tan (Δθ)) · (D / d) (2)
Nθ, Δx, and Δθ in Expression (2) indicate the number of angle divisions, the pixel pitch, and the angle resolution of the focus detection imaging element 105, respectively. As shown in Expression (2), the maximum refocus amount in the image pickup device 104 is obtained from the ratio of the pixel pitch D of the image pickup device 104 and the pixel pitch d of the focus detection image pickup device 105.

  In step S205, the determination unit 107 determines that the current focus lens position is within a refocusable range from an arbitrary focus position based on the distance between the position of the lens group 102 and the subject and the maximum refocus amount. Determine if it exists. That is, the determination unit 107 determines whether or not the subject is within a refocusable distance range.

  The imaging control unit 108 controls signal processing and imaging of the imaging element 104 for video according to the determination result by the determination unit 107, that is, depending on whether or not the position of the subject is within a refocusable range. The imaging control is, for example, determining whether or not to perform main imaging (instruction) for video recording for imaging a subject and controlling the timing of main imaging. In addition, the imaging control includes controlling the operation of changing the optical path by the mirror 103 so that the incident light 101 guided to the focus detection imaging element 105 is guided to the video imaging element 104. At this time, the imaging control unit 108 informs the user of the result of the determination by the determination unit 107 as to whether or not the subject is within the refocusable distance range by display on the screen or sound on the display unit 110. Also good.

  If the determination unit 107 determines in step S205 that the subject is within the refocusable distance range, in step S206, it is determined whether or not the instruction unit 112 has instructed imaging while the focus lens is being driven. . That is, for example, it is determined whether or not a shutter button (not shown) as the instruction unit 112 has been pressed. If the shutter button is pressed, the subject is imaged by the image sensor 104 in step S207, and the imaging operation is terminated. If the shutter button is not pressed in step S206, the process waits until it is pressed. When it is determined in step S205 that the subject is not within the refocusable distance range, the determination is repeated until the refocusable distance range is reached. As described above, in the present embodiment, control is performed so that the main imaging with video recording cannot be performed until the subject enters the refocusable distance range. The imaging timing is controlled so that the subject is imaged.

  The above-described imaging control operation and effect in the present embodiment will be specifically described with reference to FIGS.

  FIG. 3 is a diagram showing the transition of the positional relationship between the focus of the focus lens and the subject in the case of shooting with a conventional camera. In FIG. 3, the vertical axis represents the distance from the camera, and the horizontal axis represents the elapsed time of the imaging operation.

  As shown in FIG. 3, first, it is assumed that the focus 302 of the focus lens is at a position corresponding to the subject 301 between the subject 301 and the camera with respect to the subject 301 at time T1. As the time on the horizontal axis advances, the focus lens performs a focus adjustment operation, whereby the focus 302 moves toward the in-focus position 303 corresponding to the subject distance. As a result, the focal point 302 gradually approaches the position of the subject 301 and the distance Y between the subject 301 and the focal point 302 becomes shorter.

  At time T2 when time T has elapsed from time T1, the focal point 302 reaches the in-focus position 303, focusing on the subject 301 is achieved, and then the subject 301 is focused by pressing the shutter button. it can. However, when the shutter button is pressed at an earlier stage than time T2, the conventional camera takes an image in a state where the subject 301 is out of focus. Further, even an imaging device capable of refocusing cannot know whether the subject 301 exists within a range where it can be reliably refocused, so there is no guarantee that a refocus image of the subject 301 can be generated after shooting.

  Next, in the imaging apparatus according to the present embodiment, an example in which the imaging operation is controlled based on the determination result of whether or not the subject exists within the refocusable distance range will be described with reference to FIG. . FIG. 4 is a diagram illustrating the transition of the positional relationship between the focus of the focus lens and the subject in photographing by the imaging apparatus according to the present embodiment. 4, elements similar to those in FIG. 3 are denoted by the same reference numerals. In FIG. 4, a refocusable distance range 401 is shown along with the focal point 302. The refocusable distance range 401 is a range within the distance of the maximum refocus amount dmax before and after the focal point 302 in the optical axis direction.

  In FIG. 4, at time T3 when time t (where t <T) has elapsed from time T1, the focal point 302 moves to the position 402, and the subject 301 enters the refocusable distance range 401. When the shutter button is pressed between time T1 and time T3 when the subject 301 does not exist within the refocusable distance range 401, imaging is performed when the focus lens is driven until the focal point 302 moves to the position 402. Is done. Also, if the subject 301 exists within the refocusable distance range 401, imaging is performed if the shutter button is pressed even if the focus lens is not driven so that the focal point 302 moves to the in-focus position 303. To be controlled.

  By performing the above control, it is possible to shorten the time until imaging is possible from the conventional time T to time t. In this embodiment, the light 101 from the subject is incident on the focus detection image sensor 105 instead of the video image sensor 104, and it is determined whether or not the subject is within a refocusable distance range in the post-imaging processing. . Therefore, if this embodiment is applied to a reflex camera that required the behavior of raising the mirror to determine the refocusable range on the image sensor for video, the light from the subject is used for video for the determination. There is no need to enter the image sensor. That is, the behavior of raising the mirror is not necessary. At the same time, it is possible to shorten the time until imaging is possible.

  In addition, when there is no subject within the refocusable distance range, imaging is not performed even when the shutter button is pressed, and the user can refocus to an arbitrary subject according to the result of the determination unit 107. You may make it notify whether it is possible. Further, according to the result of the determination unit 107, a refocusable subject and a impossible subject may be displayed in a live view so as to be distinguishable from each other. In this case, the notification method and the display form are design matters that can be set as appropriate, and are not limited to a specific configuration. In addition, the signal processing of the signal processing unit 109 (for example, generation, display, and recording of a refocus image after imaging and recording) may be controlled according to the result of the determination unit 107.

  Here, the light field data acquired by the imaging apparatus according to the present embodiment will be described with reference to FIGS. 5A and 5B. The light field data acquired by the imaging apparatus according to the present embodiment is data captured by an LF camera (plenoptic camera).

  FIG. 5A is a diagram illustrating an example of an image sensor included in the LF camera, that is, a unit pixel cell of the image sensor 104 for video in the image capturing apparatus 100 according to the present embodiment. FIG. 5B is a diagram illustrating an example of an image with different parallax and a refocus image acquired by the LF camera.

  As shown in FIG. 5A, an image pickup device 104 for an image, which is an image pickup device of an LF camera, includes a microlens array in which a plurality of microlenses 501 are arranged and a pixel array in which a plurality of pixels 502 are arranged. Have. The plurality of microlenses 501 are arranged in a lattice pattern. The plurality of pixels 502 are arranged in a grid pattern.

  The unit pixel cell 500 of the image pickup device 104 for video is configured by a set of a plurality of pixels (divided pixels) 502 per one microlens included in the microlens array. Here, for the sake of simplicity of explanation, a case where the unit pixel cell 500 has 6 × 6 pixels 502 will be described as an example. Further, when specifying the pixel position of the pixel 502 in the unit pixel cell 500, the notation of “pixel 502 (XY)” is used. Here, X is any one of 1 to 6 indicating a column in the 6 × 6 pixel array, and Y is any one of A to F indicating a row in the 6 × 6 pixel array.

  Such unit pixel cells 500 are arranged in the image pickup device 104 for video in a two-dimensional Bayer array. It should be noted that unit pixel cells similar to the unit pixel cells 500 of the image pickup device 104 for video are similarly arranged in the focus detection image pickup device 105.

  A two-dimensional image composed only of pixels 502 existing at the same pixel position of each microlens 501 has a parallax with respect to a two-dimensional image composed only of pixels 502 present at other same pixel positions. That is, for example, a two-dimensional image composed of only the pixel 502 (1A) of each microlens 501 is different from a two-dimensional image composed of only the pixel corresponding to the pixel 502 (2A) of each microlens 501. Has parallax. That is, as shown in FIG. 5B, a total of 36 two-dimensional images with different parallaxes are obtained from the image pickup device 104 having 6 × 6 pixels 502 in the unit pixel cell 500.

  In general, in the LF camera, as shown in FIG. 5B, a refocus image is obtained by synthesizing two-dimensional images having different parallaxes according to the number of pixels of these unit pixel cells.

  The principle for obtaining a refocus image will be described with reference to FIG. 5B. When flowers and leaves are imaged as shown in FIG. 5B, if a two-dimensional image with different parallax is synthesized so that there is no parallax at the flower position, a refocus image focused on the flower position is obtained. In this case, since the images having parallax are added and synthesized at the leaf position, the leaf position is blurred. Further, if the leaves are combined so that there is no parallax, it is possible to obtain a refocus image in which the leaves are in focus and the positions of the flowers are blurred.

  When the image plane distance to be moved is determined in advance, the two-dimensional image of each pixel 502 is shift-added based on a fixed value called a K value defined between the pixels 502. Thereby, it is possible to obtain a refocus image in which the focal plane is virtually moved. The K value is the same as a fixed value for obtaining the defocus amount from the image shift amount when performing the correlation calculation process using the LF camera.

  An example of a defocus amount calculation method used for distance measurement in the present embodiment will be described with reference to FIGS. 6A to 6C.

  FIGS. 6A to 6C each show a state in which light from the subject 601 is incident on the focus detection image sensor 105 via the photographing optical system 602. Note that the photographing optical system 602 includes the lens group 102. 6A to 6C, the mirror 103 in the reflex structure is omitted. FIG. 6A shows a focused state. FIG. 6B shows the rear pin state. FIG. 6C shows the front pin state.

  The focus detection imaging element 105 includes a unit pixel cell 500 similar to the video imaging element 104 shown in FIG. 5A. The focus detection imaging element 105 has a plurality of pixels 502 under each microlens 501 in a microlens array having a plurality of microlenses 501. Two pixels 502 indicated by a and b below the microlens 501 correspond to the pixel 502 (1A) and the pixel 502 (6F) described in FIG. 5A, respectively. 6A to 6C, the positions of the unit pixel cells 500 in the focus detection image sensor 105 are indicated by P1 to P13.

  At the time of distance measurement, the output of the A image pixel group constituted by the pixel 502 (1A) indicated by a and the output of the B image pixel group constituted by the pixel 502 (6F) indicated by b are respectively displayed in the column direction ( Or in the row direction). Then, using these outputs as outputs of the unit pixel cell group of the same color, A image and B image are generated and converted into data, respectively, and the shift of each corresponding point is obtained by SAD calculation. The result of the SAD calculation is obtained as the correlation amount C by the following equation (3).

  Here, YAn and YBn are each a number sequence including the number of n pixels of the horizontal microlens. N in YAn and YBn is the number of microlenses arranged in the pupil division direction for distance measurement (the same as the number of unit pixel cells). However, the value of n should be smaller than the number of arranged microlenses. This is because it depends on the range to be measured. i represents each pixel position, and m is a shift amount for calculating the absolute value of the difference while shifting the pixel. At this time, the position of m taking the smallest correlation value C is the in-focus position, and m is the image shift amount N.

  First, in the in-focus state shown in FIG. 6A, the position where the imaging optical system 602 forms an image is a pixel (photoelectric conversion element) under the P7 microlens 501. For this reason, as shown in the lower part of FIG. 6A, the output of the A image pixel group and the output of the B image pixel group substantially coincide. At this time, the image shift amount N (a) between the subject image by the A image pixel group and the subject image by the B image pixel group obtained by the SAD calculation can be approximated to zero.

  Further, in the rear pin state shown in FIG. 6B, the position where the imaging optical system 602 forms an image is a pixel (photoelectric conversion element) under the micro lens 501 in which the A image pixel is P9 and the B image pixel is P5. . For this reason, as shown in the lower part of FIG. 6B, a deviation occurs between the output of the A image pixel group and the output of the B image pixel group. At this time, an image shift amount N (b) determined by the SAD calculation occurs between the subject image by the A image pixel group and the subject image by the B image pixel group.

  Further, in the front pin state shown in FIG. 6C, the pixel (photoelectric conversion element) under the microlens 501 in which the A image pixel is P5 and the B image pixel is P9 is the position where the imaging optical system 602 forms an image. . For this reason, as shown in the lower part of FIG. 6C, a shift occurs in the opposite direction to the rear pin state between the output of the A image pixel group and the output of the B image pixel group. At this time, between the subject image by the A image pixel group obtained by the SAD calculation and the subject image by the B image pixel group, an image shift amount N (c in the direction opposite to the rear pin state obtained by the SAD calculation is obtained. ) Occurs.

  As described above, in the in-focus state, the A and B image pixel groups are viewing the same subject, but in the rear pin state and the front pin state, the A and B image pixel groups respectively have an image shift amount N ( b) means that the subject is shifted by N (c).

At this time, the defocus amount d can be obtained by a known technique. For example, the defocus amount d can be obtained by the following equation (4) using a coefficient K that is uniquely determined by the image shift amount N and the optical state up to the light receiving element.
d = N × K (4)

  The coefficient K here is a coefficient determined in advance according to the optical state, and is a conversion coefficient for converting the relative image shift amount between the pixel 502 (1A) and the pixel 502 (6F) into a defocus amount. It points to that. By having such a value of K between the pixels, it is possible to calculate the image shift amount at the time of pixel addition in the refocus processing. 6A, 6B, and 6C, the defocus amount d (a) in the focused state, the defocus amount d (b) in the rear pin state, and the defocus amount d (c) in the front pin state, respectively. Is shown as an absolute value.

  In the above description, a case has been described in which distance information is generated based on a signal from the focus detection image sensor 105 and whether or not the subject is within a refocusable range is determined based on the distance information. Is not limited to this.

  For example, in another aspect of determination, first, an image shift amount is calculated based on signals from a plurality of photoelectric conversion units of the focus detection imaging element 105. Next, the image shift amount is determined based on the arrangement interval (pitch) of the plurality of photoelectric conversion units of the focus detection image sensor 105 and the arrangement interval (pitch) of the plurality of photoelectric conversion units of the image sensor 104 for video. This is converted into an image shift amount in the image sensor 104. Next, determination is performed by comparing the converted image shift amount with a refocusable range.

  For example, in still another aspect of the determination, first, a defocus amount is calculated based on signals from a plurality of photoelectric conversion units of the focus detection imaging element 105. Next, a determination is made by comparing the defocus amount with a refocusable range.

[Second Embodiment]
A second embodiment of the present invention will be described with reference to FIGS.

  In the first embodiment, the distance between the position of the focus lens and the position of the subject, that is, the distance information about the subject is detected using the output of the focus detection imaging element 105. In the present embodiment, infrared light reception is performed. It detects using the infrared element which is a means. This embodiment is different from the first embodiment in this respect, that is, an embodiment in which the means for measuring the distance to the subject is another general-purpose measuring means different from the first embodiment. Other configurations of the present embodiment are the same as those of the first embodiment.

  FIG. 7 is a block diagram illustrating a configuration of a main part of the imaging apparatus according to the present embodiment. In FIG. 7, elements similar to those in FIG. 1 are denoted by the same reference numerals and description thereof is omitted here.

  In FIG. 7, an infrared light irradiation unit 701 is an infrared light irradiation unit that irradiates a subject with infrared light. The reflected light 702 is reflected light from the subject of infrared light emitted from the infrared light irradiation unit 701. The infrared element 703 is an infrared light receiving unit that receives the reflected light 702 and outputs light reception information. The distance measurement information calculation unit 706 obtains the distance to the subject as distance information about the subject from the information received by the infrared element 703.

  The imaging operation of the imaging apparatus according to the present embodiment described above will be described with reference to FIG. FIG. 8 is a diagram illustrating a flowchart of the imaging operation of the imaging apparatus according to the present embodiment. In FIG. 8, the same steps as those in FIG. 2 are denoted by the same reference numerals, and description thereof is omitted here.

  In step S801, infrared light is projected from the infrared light irradiation unit 701 onto the subject. Next, in step S802, reflected light from the subject of the infrared light projected in step S801 is received by the infrared element 703. Next, in step S203, the distance measurement information calculation unit 706 calculates the distance to the subject using the output of the infrared element 703, and generates distance information about the subject. The processes after step S204 following step S203 are the same as those in the first embodiment.

  Here, a method of calculating the distance to the subject by the distance measurement information calculation unit 706 will be described with reference to FIGS. 9A and 9B. 9A and 9B, the same elements as those in FIG. 7 are denoted by the same reference numerals.

  FIG. 9A is a diagram for explaining the configuration of distance measurement by the TOF method. In FIG. 9A, an infrared light projecting lens 901 and an infrared light receiving lens 902 are shown. The infrared light (irradiation light) 903 emitted from the infrared light irradiation unit 701 is reflected by the object X that is the subject. The reflected light 904 reflected by the object X is received by the infrared element 703. The distance is measured from the time from the infrared light irradiation to the reception of the reflected light, that is, the time required for the round trip from the camera to the object. Since the speed of light is about 300,000 kilometers per second, the distance can be calculated from the time.

  A time measuring method will be described with reference to FIG. 9B. FIG. 9B is a diagram illustrating the phases of the irradiation light and the reflected light. Since light has a period of a transverse wave, it is possible to measure a time shift from a phase shift. Therefore, the phase difference of light is used for time measurement. In the case of FIG. 9B, since the time Tx is known from the phase shift of the reflected light 904 with respect to the irradiation light 903, the distance to the object X is calculated. A defocus amount is calculated from the calculated distance. According to Patent Document 2, since the defocus amount can be calculated from the image shift amount of the image pickup device 104, the image shift amount of the image pickup device 104 is calculated from the calculated defocus amount by back calculation. Whether or not the current focus lens position is within a refocusable range from an arbitrary focal position is determined from the maximum refocus amount and image shift amount obtained by the equation (1) described in the first embodiment.

  With the above processing method, it is possible to determine a refocusable range on the image pickup device 104 for video without using the focus detection image pickup device 105 used in the first embodiment. Further, when the present invention is applied to a reflex camera that requires the behavior of raising the mirror to determine the refocusable range on the image pickup device 104 for video, the mirror is raised as in the first embodiment. No behavior is required. Therefore, it is possible to shorten the time until imaging is possible.

  The method for controlling the imaging apparatus according to the result of the determination unit 107 is the same as that described in the first embodiment.

  In the present embodiment, an infrared element is used as a distance measuring method that does not use a focus detection imaging element. However, the present invention is not limited to this as long as distance information about a subject can be obtained.

  According to the embodiment of the present invention described in detail above, it is possible to determine whether or not the subject exists at a refocusable position without causing light to enter the image pickup device for video. An image sensor that can be shortened can be provided.

[Third Embodiment]
In the first embodiment, a case has been described in which it is determined whether or not refocusing is possible based on the output of the focus detection image sensor 105 in a situation where incident light to the image sensor 104 for video is blocked by the mirror 103. . According to the first embodiment, the response of shooting can be improved.

  In contrast to the first embodiment, for the purpose of displaying a live view image to the user, it is possible to secure incident light to the image pickup device 104 by configuring the mirror 103 with a semi-transmissive mirror. Even in such a configuration, it is possible to realize power saving and highly accurate determination by determining whether or not refocusing is possible using the focus detection imaging element 105 as in the first embodiment. .

  For example, in order to determine whether or not a subject exists within a distance range that can be refocused by the image pickup device 104 for video, it is necessary to read out a pixel 1402 (see FIG. 14) having parallax and perform distance measurement. Therefore, if two different parallax images are an A image and a B image, it is necessary to read twice as much as the original image for imaging. In general, since the image pickup device 104 for video has a larger number of pixels than the image pickup device 105 for focus detection, there is a problem that requires a lot of power.

  Furthermore, it is desirable that the readout frame rate of the video image sensor 104 be synchronized with the display rate of the display unit 110. For this reason, there are cases where it is difficult to obtain an exposure amount that is most suitable for distance measurement at low illuminance when distance measurement is performed using the image pickup device 104 for video.

  In the third embodiment, a solution for such a problem will be described with reference to FIG. FIG. 10 is a block diagram illustrating the imaging apparatus according to the present embodiment. The imaging apparatus according to the third embodiment is realized by a configuration in which the mirror 103 in the configuration of the first embodiment shown in FIG.

  As shown in FIG. 10, instead of the mirror 103, a semi-transmissive mirror 803 that functions as a light distribution unit that distributes the incident light 101 is provided. The semi-transmissive mirror 803 distributes the amount of incident light 101 to the image pickup device 104 for video and the image pickup device 105 for focus detection based on the transmittance. For example, when the semi-transmissive mirror 803 is configured by a pellicle mirror having a transmittance of 50%, incident light 101 having a light amount of 50% is incident on the image pickup device 104 for image formation. Further, the remaining 50% of incident light 101 is incident on the focus detection image sensor 105 to form an image. Note that the transmittance of the semi-transmissive mirror 803 is not limited to 50%, and can be appropriately selected.

  The image pickup device 104 for video performs pixel addition with a predetermined number of pixels 1402 sharing the same microlens 1401 (see FIG. 14) as a unit. Further, during the live view operation, the same color pixels are added or thinned out from the image pickup device 104 in accordance with the number of display pixels of the display unit 110 that performs live view display. By performing the processing in this manner, it is not necessary to read out all the pixels of the image pickup device 104 during the live view operation, so that power consumption can be reduced.

  The electric charges read from the image pickup device 104 are output to the signal processing unit 109 via an A / D conversion unit (not shown). The signal processing unit 109 generates a display image used in the live view and outputs it to the display unit 110. The display unit 110 displays the input display image on a panel represented by, for example, an LCD, and notifies the user of the current subject state.

  On the other hand, the focus detection imaging element 105 receives the incident light 101 whose light amount is reduced by the semi-transmissive mirror 803 as in the first embodiment. The distance measurement information calculation unit 106 obtains the defocus amount based on the output signal of the focus detection image sensor 105 as in the first embodiment.

  The focus detection image sensor 105 can perform exposure control independently of the video image sensor 104.

  For example, when the display unit 110 displays a display image at 50 fps, the video image sensor 104 generally captures images at substantially the same frame rate as the display unit 110. For this reason, it is difficult to obtain an exposure time of 1/50 second or longer. Furthermore, a good signal with less noise is desirable in order to perform distance measurement correctly. However, the incident light 101 that has passed through the semi-transmissive mirror 803 has a smaller amount of light than usual, and thus noise can increase. For these reasons, there are cases where it is desired to extend the exposure time and secure the exposure amount. Therefore, it is desirable that an exposure time suitable for distance measurement can be set independently of the frame rate of the display unit 110.

  Further, the exposure amount suitable for distance measurement does not necessarily match the exposure amount set for obtaining the display image. Therefore, it is desirable that the exposure amount used for distance measurement can be set independently of the display image.

  In the present embodiment, as described above, the focus detection imaging device 105 can be controlled by the imaging control unit 108 independently of the video imaging device 104 based on an exposure photometric sensor (not shown). ing. Thereby, it is possible to set an exposure amount suitable for distance measurement. The configuration for performing the exposure control is not particularly limited, and various configurations can be used.

  Next, as in the first embodiment, the determination unit 107 determines whether or not the subject exists within a refocusable distance range. If it is determined that the subject exists within the refocusable distance range, the semi-transparent mirror 803 is moved up so that almost all the light quantity of the incident light 101 is incident on the image pickup device 104 for image capturing. Do.

  By performing the processing in this manner, even when the mirror 103 is changed to the semi-transmissive mirror 803, it is possible to appropriately determine whether or not refocusing can be performed with high power and high accuracy, and an imaging operation can be performed.

[Fourth Embodiment]
In the third embodiment, the configuration when the mirror 103 is changed to the semi-transmissive mirror 803 is described, and the processing in which both the image pickup device 104 for video and the image pickup device 105 for focus detection perform an imaging operation in the configuration is described. .

  In the fourth embodiment, an operation of determining a refocusable range when using each image pickup device for different purposes in an image pickup apparatus including a plurality of image pickup devices will be described.

  This will be specifically described with reference to FIG. FIG. 11 is a block diagram illustrating the imaging apparatus according to the present embodiment. In addition, the same code | symbol is attached | subjected to the element which does not change with 1st and 3rd embodiment, and description is abbreviate | omitted.

  In the imaging apparatus 1100 according to the present embodiment, a still image sensor 1101 for acquiring a still image and a moving image sensor 1102 for acquiring a moving image are provided as a video image sensor. The still image pickup device 1101 and the moving image pickup device 1102 each have a pupil division configuration as will be described later.

  Similar to the third embodiment, a semi-transmissive mirror 803 is provided instead of the mirror 103. The semi-transmissive mirror 803 functions as a light distribution unit as in the third embodiment. For example, half of the amount of incident light 101 is distributed to the still image sensor 1101 and the other half to the moving image sensor 1102. .

  A signal processing unit 109 similar to that of the first embodiment is provided for the still image sensor 1101. On the other hand, a signal processing unit 1103 is provided for the moving image sensor 1102.

  The still image sensor 1101 and the moving image sensor 1102 will be described with reference to FIG. FIG. 12 is a diagram showing details of the still image sensor 1101 and the moving image sensor 1102.

  In the still image pickup element 1101, unit pixel cells 1201s are two-dimensionally arranged in a Bayer array. The unit pixel cells 1201s are arranged at a cell pitch sd. In the unit pixel cell 1201s, as described in FIG. 14, for example, 6 × 6 pixels 1202s are arranged so as to share a microlens.

  Similar to the still image pickup device 1101, the moving image pickup device 1102 has unit pixel cells 1201m arranged two-dimensionally in a Bayer arrangement. The unit pixel cells 1201m are arranged at a cell pitch md. In the unit pixel cell 1201m, similarly to the still image pickup device 1101, for example, 6 × 6 pixels 1202m are arranged so as to share a microlens.

  Compared with the moving image pickup device 1102, the still image pickup device 1101 has a feature that the pixel pitch of the unit pixel cell and the pixel pitch of the pixel included in the unit pixel cell are narrower in order to pick up a fine image. doing. Accordingly, the number of pixels of the still image sensor 1101 is larger than that of the moving image sensor 1102. Further, the number of pixels 1202s included in the unit pixel cell 1201s of the still image sensor 1101 and the number of pixels 1202m included in the unit pixel cell 1201m of the moving image sensor 1102 are the same. For this reason, the ratio between the pixel pitch of the pixel 1202s and the pixel pitch of the pixel 1202m is substantially the same as the ratio between the cell pitch sd and the cell pitch md.

  The signal processing unit 109 performs the same processing as that of the first embodiment on the output of the still image sensor 1101.

  On the other hand, the signal processing unit 1103 performs image processing on the output of the moving image sensor 1102 and outputs the compressed moving image data to the recording unit 111 and also outputs the image data being recorded to the display unit 110. . Further, using the pixel 1202m arranged as the pixel 1402 (see FIG. 14), the same processing as the distance measurement information calculation unit 106 of the first embodiment such as calculating the defocus amount is performed, and the processing result is obtained. The data is output to the determination unit 107.

  A specific configuration of the signal processing unit 1103 will be described with reference to FIG.

  The signal processing unit 1103 includes a pixel correction unit 1300, a demosaic processing unit 1301, a pixel addition unit 1302, a moving image compression unit 1303, and a ranging calculation unit 1304. The output of the moving image sensor 1102 is input to the pixel correction unit 1300.

  The pixel correction unit 1300 performs correction processing for pixel defects typified by defective pixels included in the output of the moving image pickup device 1102, and outputs the output subjected to the correction processing to the demosaic processing unit 1301.

  The demosaic processing unit 1301 performs color interpolation of the pixels 1202m arranged in a Bayer shape and converts the color space of the output image to a YUV space, and then adds a pixel addition unit 1302, a moving image compression unit 1303, and a ranging calculation unit 1304. Respectively. At this time, as a unit of color interpolation, a plurality of pixels 1202m sharing one microlens are set as one unit, and color interpolation is performed using pixels 1202m having the same relative position in each unit. That is, in the case of having 6 × 6 pixels 1202m, the demosaic processing unit 1031 generates 36 images and outputs them to each of the above-described units.

  The pixel addition unit 1302 performs image addition processing in units of pixel groups including a plurality of pixels 1202m sharing one microlens to generate an image signal with a shallow depth of field, and the generated image signal is compressed into a moving image. Output to the unit 1303.

  The moving image compression unit 1303 performs irreversible moving image compression processing on the output of the pixel addition unit 1302 based on the moving image compression standard represented by H264, and outputs the compressed image signal to the recording unit 111. An image signal obtained by the irreversible compression process is called a proxy image, and is lightweight moving image data used for the editing process.

  In addition, the moving image compression unit 1303 performs lossless compression processing on 36 pieces of image data that is an image group output from the demosaic processing unit 1301, and outputs the compressed image data to the recording unit 111. This losslessly compressed image data is used to generate a virtual viewpoint moving image that can obtain an arbitrary viewpoint and a refocus moving image that performs refocus processing.

  The ranging calculation unit 1304 obtains a defocus amount by SAD calculation between the pixels 1202 m and outputs the defocus amount to the determination unit 107. As a pixel used for the SAD calculation, a pair of pixels may be selected from the pixels 1202m. For the purpose of improving the S / N, a plurality of pixels 1202m are added to the left eye pixel and the right eye pixel. And may be generated.

  Based on the calculated defocus amount and the ratio of md and sd, the determination unit 107 sets the distance range within which the refocus can be performed on the still image sensor 1101 at the focus lens position of the lens group 102 of the current frame. It is determined whether or not a subject exists. Also in the present embodiment, determination by the determination unit 107 can be performed as in the first embodiment.

  By processing as described above, even when a plurality of image sensors for video are used for different purposes, the subject within the distance range that can be refocused on the other image sensor by distance measurement using one image sensor. It is possible to determine whether or not exists.

  In the present embodiment, it is determined whether or not the refocusable range is based on the ratio of md and sd. However, the present invention is not limited to this. For example, the refocusable defocus amount obtained based on md and sd may be obtained in advance, and the determination process may be performed by comparing the defocus amounts.

  In the present embodiment, in addition to the above-described determination, the determination may be performed based on a comparison of a fixed coefficient K value for converting an image shift amount by a correlation calculation between phase difference pixels into a defocus amount. Further, it may be determined by conversion of the image shift amount whether or not the subject exists in a refocusable range by comparing a fixed coefficient K value for converting the image shift amount obtained by the SAD calculation into the defocus amount. .

  In this embodiment, the moving image sensor 1102 has been described as having a pixel (sub-pixel) in all unit pixel cells. However, the moving image sensor 1102 is not limited thereto. As the moving image pickup element 1102, an image pickup element having a focus detection pixel in part may be used.

  In the present embodiment, it is determined whether or not the subject exists in the refocusable range of the still image pickup device 1101 based on the output of the moving image pickup device 1102. However, the present invention is not limited to this. Conversely, the refocusable range of the moving image sensor 1102 may be determined by the still image sensor 1101.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  According to the embodiment of the present invention, based on information different from the output from the image sensor for video, whether or not the subject exists at a refocusable position in the data obtained from the image sensor for video is determined. Can be determined. Therefore, according to the embodiment of the present invention, it is possible to provide an imaging apparatus capable of reducing the photographing time or the processing load.

DESCRIPTION OF SYMBOLS 100 Imaging device 104 Image pick-up element 105 Image sensor 106 for focus detection Distance information calculation part 107 Determination part 108 Imaging control part 701 Infrared light irradiation part 703 Infrared element 706 Distance information calculation part

Claims (12)

An imaging unit having a plurality of photoelectric conversion units that respectively receive light beams from subjects that have passed through different pupil regions;
Distance information generating means for detecting light from the subject and generating distance information related to the distance of the subject;
A determination unit configured to determine whether the subject is within a distance range that can be refocused in a process after imaging based on the distance information and a pupil division configuration of the plurality of photoelectric conversion units;
Control means for controlling the imaging of the subject by the imaging means according to the result of the determination by the determination means ,
The imaging means is a first imaging element that captures an optical image of the subject, and includes a first imaging element configured by the plurality of photoelectric conversion units,
The distance information generating unit includes a second imaging element that captures an optical image of the subject,
The distance information generation means generates an image signal for focus detection by the second image sensor, generates an image shift amount in the second image sensor using the image signal, and generates the image shift. based on the amount and generating the distance information, that generates the information of the maximum refocus amount of the imaging unit based on the information of the pixel pitch of said first pixel pitch and said second imaging element of the imaging device An imaging apparatus characterized by that.   The control unit controls at least one of determination of whether or not the subject can be imaged, timing of imaging of the subject, and notification of the determination result according to a result of determination by the determination unit. Item 2. The imaging device according to Item 1.   When it is determined that the subject is not within the refocusable distance range, the control unit controls the imaging unit so that the subject is not imaged, and the subject is within the refocusable distance range. 3. The imaging apparatus according to claim 2, wherein when it is determined that the imaging unit is, the imaging unit is controlled to perform imaging of the subject. Instructing means for instructing the imaging means to image the subject,
When the determination unit determines that the subject is not within the refocusable distance range and the instruction unit instructs to capture the subject, the control unit determines that the subject is to be refocused. The imaging apparatus according to claim 3, wherein the imaging unit causes the imaging unit to perform imaging when it is determined that the distance is within a possible range. A display unit for displaying a result of determination by the determination unit;
The said control means displays the result of determination by the said determination means with respect to the said several subject on the said display means so that identification can be carried out for every to-be-photographed object, The Claim 1 thru | or 4 characterized by the above-mentioned. Imaging device. The light distribution unit according to any one of claims 1 to 5, further comprising a light distribution unit configured to distribute and input the light from the subject to the first image sensor and the second image sensor. The imaging device described. The imaging apparatus according to claim 6 , wherein one of the first imaging element and the second imaging element is an image sensor for a still image and the other is an image sensor for a moving image. Infrared light irradiation means for irradiating infrared light;
Infrared light receiving means for receiving infrared light and outputting received light information is further provided,
The distance information generating means receives the distance information based on the received light information that is received and output by the infrared light receiving means by receiving the reflected light of the infrared light irradiated on the subject by the infrared light irradiating means. 6. The imaging apparatus according to claim 1, wherein the information is generated and information on a maximum refocus amount of the imaging unit is generated based on the pupil division configuration. The imaging unit includes a plurality of microlenses and a plurality of pixels, and a predetermined number of the pixels are arranged for each of the plurality of microlenses, and the microlens and the predetermined number of pixels the imaging apparatus according to any one of claims 1 to 8, characterized in that the pupil division structure is formed. An imaging method using an imaging device having an imaging means having a plurality of photoelectric conversion units that respectively receive light beams from subjects that have passed through different pupil regions,
Detecting light from the subject to generate distance information regarding the distance of the subject;
Determining whether or not the subject is within a distance range that can be refocused in a post-imaging process based on the distance information and a pupil division configuration of the plurality of photoelectric conversion units;
Controlling the imaging of the subject by the imaging means according to the result of the determination ,
The imaging means is a first imaging element that captures an optical image of the subject, and includes a first imaging element configured by the plurality of photoelectric conversion units,
In the step of generating the distance information, an image signal for focus detection is generated by a second image sensor that captures an optical image of the subject, and an image on the second image sensor is used by using the image signal. A shift amount is generated, the distance information is generated based on the image shift amount, and the maximum of the imaging unit is determined based on the pixel pitch of the first image sensor and the pixel pitch information of the second image sensor. imaging method characterized that you generate information refocus amount. A program for controlling an imaging apparatus having an imaging means having a plurality of photoelectric conversion units that respectively receive light beams from subjects that have passed through different pupil regions,
Computer
Distance information generating means for detecting light from the subject and generating distance information relating to the distance of the subject;
Based on the distance information and the pupil division configuration of the plurality of photoelectric conversion units, a determination unit that determines whether or not the subject is within a distance range that can be refocused in a process after imaging, and a result of determination by the determination unit To function as a control means for controlling the imaging of the subject by the imaging means ,
The imaging means is a first imaging element that captures an optical image of the subject, and includes a first imaging element configured by the plurality of photoelectric conversion units,
The distance information generating unit includes a second imaging element that captures an optical image of the subject,
The distance information generation means generates an image signal for focus detection by the second image sensor, generates an image shift amount in the second image sensor using the image signal, and generates the image shift. based on the amount and generating the distance information, that generates the information of the maximum refocus amount of the imaging unit based on the information of the pixel pitch of said first pixel pitch and said second imaging element of the imaging device program.   A computer-readable storage medium storing the program according to claim 11.