JP6606838B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to image processing based on a plurality of images.

  Video cameras, television systems, etc., perform image processing based on multiple images. For example, high-resolution images that exceed the resolution (number of pixels) of the image sensor and high-quality images with excellent reproducibility such as hues. (So-called “super-resolution processing”). There, a plurality of images in which the relative position between the subject image and the pixel position is different by moving the image sensor relative to the subject image while the positional relationship between the subject and the camera or the state of the subject is fixed. These images are taken and combined by sampling point interpolation or color information interpolation. Thereby, a high-quality image can be obtained for the number of pixels of the image sensor.

  When super-resolution processing is performed, if there is a change (motion) in the positional relationship between the subject and the camera or the state of the subject during shooting of multiple images, the motion is estimated and corrected, and then The image must be synthesized. Therefore, in a television system or the like, if there is an erroneous motion estimation, the generated composite image is disturbed. In order to prevent this, for example, when an abnormal motion vector is detected in a part of a video between frames before and after detecting a motion vector, super-resolution processing is not executed for that part (see Patent Document 1).

JP 2009-140393 A

  However, for example, in a situation where super-resolution processing is performed at the time of shooting, the photographer cannot determine whether or not the image processing has been successful unless he / she looks at the generated image in detail. It has been processed. In addition, a situation occurs in which the photographer notices that the photographer has failed in the editing work without noticing the failure on the spot. Therefore, it is necessary to improve such inconvenience.

  An imaging apparatus of the present invention includes a control unit that controls acquisition of a plurality of images while changing a relative position between a subject image and an imaging element, an image processing unit that performs image processing based on the acquired plurality of images, A detection unit that detects a difference in the subject image in at least a part between at least two acquired images, and the control unit acquires the image again or notifies the difference in the subject image according to the difference in the subject image. I do. For shooting to acquire a plurality of images, the image sensor can be shifted with respect to the subject image, or vice versa.

  Such an imaging apparatus can be applied to, for example, a camera capable of performing a shooting operation for generating a high-quality image such as super-resolution shooting, and can improve resolution (number of pixels) and accurately. For a plurality of image acquisitions related to image quality improvement such as color interpolation, it is possible to detect a difference in the subject image, acquire an image again, and notify the difference. That is, if there is no difference in subject images, the present invention can be applied to shooting that can generate one image as it is from a plurality of images.

  For example, the control unit can notify the difference between the subject images by displaying an image in which a portion where the difference between the subject images occurs is displayed. In addition, a light emitting unit may be provided, and the control unit may cause the light emitting unit to emit light so as to notify the difference between the subject images.

  Here, the “subject image difference” is caused by a change including subject movement, brightness, hue, and the like. If there is a difference in at least part of the subject image between a plurality of images, the subject image has a difference. Suppose it has occurred. For example, the detection unit detects at least one of the movement of the subject image, the change in the brightness of the subject image, and the change in the color of the subject image as a difference between the subject images. “Re-acquisition of image” is based on image processing, such as re-shooting (exposure to image sensor and acquisition of image (data)) so as to obtain a plurality of images with no difference in subject images. The image acquisition operation is controlled to complete the acquisition of a plurality of images.

  The detection unit can detect a difference between subject images based on a difference between at least two images. For example, the detection unit obtains a difference in luminance or color information between corresponding pixels between at least two images, and detects a difference in subject images based on the obtained difference. The detection unit can obtain a difference between luminance histograms or color information histograms of at least two images, and can detect a difference between subject images based on the obtained difference. Further, the detection unit can obtain a difference between an average value of luminance or an average value of color information of at least two images and detect a difference between subject images based on the obtained difference.

  The detection unit can detect at least one of a change in the position of the subject image between at least two images, a change in the brightness of the subject image, and a change in the color of the subject image as a difference between the subject images. It is. For example, the detection unit detects the difference between the subject images before completion of all the acquisition of the plurality of images based on the image processing, and when the difference between the subject images is detected, the control unit detects the plurality of images from the beginning. Re-acquisition may be executed.

  Alternatively, when a difference between subject images is detected, the control unit can execute re-acquisition from the last acquired image among a plurality of already acquired images. On the other hand, the detection unit detects the difference between the subject image of the reference image and the subject image of the other image after completing the acquisition of the plurality of images based on the image processing, and detects the difference between the subject images. It is also possible to execute re-acquisition for the image generated at that time.

  The detection unit detects a difference between the subject images before completion of all acquisition of the plurality of images based on the image processing or after completion of all acquisition of the plurality of images based on the image processing, and the control unit detects the difference between the subject images. When detected, it is also possible to execute storage of at least one of the already acquired images and display of at least one of the already acquired images. In addition, the detection unit can detect a difference in the subject image after image processing based on a plurality of images, and the control unit can notify the difference in the subject image when the difference in the subject image is detected.

  On the other hand, it is also possible to detect a difference in the subject image even before shooting (before the release button operation) such as a through image display period. For example, before starting the acquisition of a plurality of images, the difference between the subject images is detected with a pre-shooting detection unit that detects a difference in the subject images between the plurality of images obtained by the image sensor while changing the relative positions. And a pre-shooting control unit for notifying the difference between the subject images. The pre-shooting detection unit and the detection unit, and the pre-shooting control unit and the control unit may be separate, or substantially the same, or may be included in the detection unit and the control unit. Such a configuration related to photographing can be configured regardless of the above-described acquisition of a plurality of images, difference detection, image re-acquisition, or notification.

  An imaging method according to another aspect of the present invention executes acquisition of a plurality of images while changing a relative position between a subject image and an imaging device, and executes image processing based on the acquired plurality of images to perform another processing. An image is generated, a difference in at least a part of the subject image between at least two acquired images is detected, and an image acquisition or notification of the difference in the subject image is performed again according to the difference in the subject image.

In another aspect of the present invention, a program recorded on a computer-readable medium is
The imaging apparatus controls the acquisition of a plurality of images while changing the relative position between the subject image and the image sensor, and executes image processing based on the acquired plurality of images to generate another image. The image processing means functions as detection means for detecting at least a part of the subject image difference between the acquired at least two images, and according to the difference in the subject image, the image acquisition or the subject image The control means is caused to function so as to notify the difference.

  An image processing apparatus according to another aspect of the present invention performs image processing based on a plurality of images obtained by acquiring a plurality of images while changing a relative position between a subject image and an imaging element, An image processing unit that generates a difference, a detection unit that detects a difference in at least a portion of the subject image between at least two images, and a notification unit that notifies the difference in the subject image when a difference in the subject image is detected With.

  An imaging apparatus according to another aspect of the present invention includes a control unit that controls acquisition of a plurality of images while changing a relative position between a subject image and an imaging element, and image processing that performs image processing based on the acquired plurality of images. And a detection unit that detects a difference in at least a part of the subject image between at least two images, and the control unit changes the number of acquisition times of the plurality of images according to the difference in the subject image.

  For example, when a difference between subject images is detected, the number of acquisitions is increased by acquiring images again so that a plurality of images without differences are obtained. Alternatively, when it is impossible to finally generate one image due to the difference in the subject image, it is possible to end the image acquisition halfway and reduce the number of acquisitions.

  According to the present invention, it is possible to prevent image generation failure in image processing based on a plurality of images.

It is a block diagram of the digital camera which is 1st Embodiment. It is a schematic perspective view of an image blur correction mechanism. It is the figure which showed the flow of the super-resolution process. It is the figure which showed the shift position of the image sensor at the time of super-resolution imaging | photography. It is the figure which showed the high quality image partially. It is the figure which showed the difference of a pixel value It is the figure which showed the difference of the average value of the whole image. It is the figure which showed the histogram difference of the whole image. 5 is a flowchart of super-resolution imaging and super-resolution processing in the present embodiment. 10 is a flowchart of super-resolution imaging and super-resolution processing in the second embodiment. 10 is a flowchart of super-resolution imaging and super-resolution processing in the third embodiment. 14 is a flowchart of super-resolution imaging and processing in the fourth embodiment. 10 is a flowchart of super-resolution imaging and processing in the fifth embodiment. It is the figure which showed the display screen which notifies a to-be-photographed image change. 14 is a flowchart of super-resolution imaging and processing in the sixth embodiment.

  Below, the digital camera which is this embodiment is demonstrated with reference to drawings.

  FIG. 1 is a block diagram of a digital camera according to the first embodiment. FIG. 2 is a schematic perspective view of the image blur correction mechanism.

  The digital camera 10 includes a camera body 20 and a photographic lens 30 (lens barrel) that can be attached to and detached from the camera body 20, and an LCD 24 is provided as a display device on the back of the camera body 20. Further, in accordance with an input operation to the release button 15 or a mode dial, a cross button, an execution button (none of which are shown), a shooting operation, a mode setting, and the like are performed.

  A controller 40 constituted by a CPU, a DSP, etc., outputs a control signal to the aperture / shutter drive circuit 23, the image sensor drive circuit 60, etc., and a signal sent from the photographing operation switch 26 etc. that detects a release button operation or the like. Accordingly, operation control of the entire camera such as exposure control, shooting / recording operation, playback operation, function setting processing is performed. A camera operation control program is stored in advance in a recording medium such as a ROM (not shown).

  The communication memory 33 stores data such as lens resolving power, and is read out by the controller 40 when the photographing lens 30 is attached. The light emitting unit 90 can emit light in a predetermined shooting operation mode, and notifies the user and the like of shooting conditions.

  The imaging optical system 31 including a focusing lens or the like causes light to be imaged on the image sensor 22 when light from the subject is incident, thereby forming a subject image on the image sensor 22. The image sensor 22 can be constituted by a CCD, a CMOS, or the like, and a color filter array (not shown) is disposed oppositely on the light receiving surface of the image sensor 22. Here, the color filter array has a configuration in which R, G, and B color filter elements are arranged in, for example, a Bayer array.

  While the through image (preceding image) is displayed on the LCD 24 in the photographing mode, pixel signals for one field or one frame are sequentially read from the image sensor 22 at a predetermined time interval (such as 1/60 second). The read R, G, and B pixel signals are sent to the controller 40 via an AFE circuit (not shown).

  The controller 40 performs gain processing, color conversion processing, white balance adjustment, and the like on the input series of pixel signals to generate R, G, and B color image signals. Then, the controller 40 drives and controls the LCD 24 based on the generated R, G, B color image signals, whereby a real-time moving image is displayed on the LCD 24.

  When the release button 15 is half-pressed, the AF drive circuit 62 drives the focusing lens based on the pixel signal read from the image sensor 22 according to the contrast method. Further, the controller 40 detects the brightness of the subject image based on the pixel signal read from the image sensor 22 and calculates the exposure value.

  When the release button 15 is fully pressed, the drive control unit 41 of the controller 40 outputs a control signal to the aperture / shutter drive circuit 23 to control exposure. Thereby, a pixel signal for one frame is read from the image sensor 22. The controller 40 generates still image data based on the read pixel signal for one frame, acquires the image data in a compressed or uncompressed state, and records it in the image memory 25.

  The image sensor 22 is mounted on the image blur correction mechanism 50 and is relatively movable along the x and y axis directions orthogonal to the optical axis (z axis) of the photographing optical system 31. As shown in FIG. 2, the image blur correction mechanism 50 includes a plate-like support portion 51 fixed to the camera body 20 and a movable stage 52 that can move relative to the support portion 51. The magnets M1, M2, and M3 are arranged to face the movable stage 52. On the movable stage 52, the yokes Y1, Y2, and Y3 are disposed so as to face the magnets M1, M2, and M3, respectively.

  The movable stage 52 is provided with driving coils C1, C2, and C3. By passing an AC drive signal through the driving coils C1, C2, and C3 in a magnetic field, the movable stage 52 is along an optical axis orthogonal plane. It moves relative to the support portion 51. The drive control unit 41 of the controller 40 controls the drive signal generation unit 61 of the image sensor drive circuit 60, and thereby an AC drive signal is output from the drive signal generation unit 61.

  Hall sensors H1, H2, and H3 are disposed in the vicinity of the driving coils C1, C2, and C3 of the support 51, and the hall sensors H1, H2, and H3 are movable by detecting the magnetic force of the magnets M1, M2, and M3. The position of the stage 52 (image sensor 22) in the plane orthogonal to the optical axis is detected. Further, the gyro sensor 28 provided in the camera body 20 detects blurring of the camera body 20.

  According to the shake (tilt angle or angular velocity) of the camera body 20 detected by the gyro sensor 28 and the position of the image sensor 22 by the Hall sensors H1, H2, and H3, the controller 40 moves the image shake correction device 50 along the plane orthogonal to the optical axis. Drive control is performed to shift the imaging area of the subject image so as to cancel the image displacement caused by camera shake. By this camera shake correction operation, it is possible to obtain a subject image without image blurring.

  In the present embodiment, it is possible to acquire an image with higher image quality (hereinafter referred to as a high quality image) using the image blur correction mechanism 50. Specifically, a plurality of shootings (hereinafter referred to as super-resolution shooting) are performed while shifting the shooting target area. At this time, the user shoots with the camera 10 fixed, such as by setting the camera 10 on a tripod. Then, image processing (synthesizing) is performed based on the obtained plurality of images (hereinafter referred to as captured images), and the obtained images (hereinafter referred to as high-quality images) are recorded / displayed.

  When super-resolution imaging is performed, four images are sequentially acquired here. The super-resolution operation switch 27 detects ON / OFF of the super-resolution processing and the user's setting operation regarding the super-resolution processing.

  Hereinafter, super-resolution imaging will be described with reference to FIGS.

  FIG. 3 is a diagram showing a flow of super-resolution processing. FIG. 4 is a diagram illustrating the shift position of the image sensor during super-resolution imaging. FIG. 5 is a diagram partially showing a high-quality image. However, the flow of FIG. 3 is a flow for explaining the super-resolution processing, and the operation of the camera 10 is actually controlled by the flow described later.

  When the release button 15 is pressed and super-resolution shooting is started, the image sensor 22 moves to the first shooting position and performs shooting (exposure to the image sensor 22 and acquisition of an image) (S101, S102). . Then, the position of the image sensor 22 at the time of shooting the first image is used as a reference position, and the shooting position of the image sensor 22 is shifted to a different position at the time of shooting the second, third, and fourth images. (S103-S107). The first to fourth images are automatically taken after the first release button 15 is pressed. At the time of super-resolution imaging, the camera shake correction function may be temporarily set to OFF regardless of the setting so far, and the original setting may be restored after super-resolution imaging is completed.

  FIG. 4 shows the positions of the image sensor 22 at the time of shooting, and the horizontal and vertical directions of the image are represented by X and Y coordinates, respectively. For the sake of explanation, it is assumed that the image region IR is configured by a 6 × 6 pixel array. The image sensor 22 shifts at every photographing, that is, the image sensor 22 is formed in the image region IR by moving the position (pixel position) of the image sensor 22 relative to the subject image formed by the photographing optical system 31. The image varies with each shot.

  Here, the image sensor 22 moves to a position shifted by +1 pixel in the X direction when the first image is taken as a reference position. In the third shooting, the image sensor 22 moves to a position shifted by +1 pixel and −1 pixel in the X and Y directions with respect to the reference position, and in the fourth shooting in the Y direction with respect to the reference position. The image sensor 22 moves to a position shifted by −1 pixel. As a result, captured images I1 to I4 having different positions of the image region IR with respect to the subject image are generated.

  As described above, the Bayer array color filter array is disposed on the light receiving surface of the image sensor 22, and any one of R, G, and B color elements is disposed to face each pixel. Therefore, if attention is paid to one piece of image data before the color interpolation process, the pixel signal output from each pixel includes only R, G, or B color information. In FIG. 4, the symbols of the respective pixels are represented as “11R, 12G, 13R,.

  On the other hand, when super-resolution imaging is performed, the light receiving area IR of the image sensor 22 is sequentially sequentially moved by one pixel, so that the same portion is imaged by adjacent pixels having other color filter elements. Thereby, all the R, G, and B color information are included in the pixel signal of each pixel.

  For example, in the first shooting, the image PA formed by the pixel 33R is shifted by one pixel of the image sensor 22, and in the second, third, and fourth shooting, the pixels 32G, 22B, 23G is formed. Since the pixel signals output from the pixels 32G, 22B, and 23G include G or B color information, it is possible to obtain a pixel signal that includes all the R, G, and B color information for the image PA. it can. For the G component, the average value of the pixel signals output from 32G and 23G is calculated here as the G component pixel value.

  When four captured images I1 to I4 are obtained, image processing (super-resolution processing) for combining the captured images I1 to I4 is executed, and a high-quality image is generated separately from the four captured images I1 to I4. The That is, photographed image data based on the R, G, and B pixel values of each pixel is formed. The generated high-quality image is displayed on the LCD 24 and recorded in the image memory 25 (S108 to S110).

  FIG. 5 is a diagram illustrating a high-quality image. The image area IR ′ of the high-quality image IH is composed of a 5 × 5 pixel array excluding the right end column that does not include all the R, G, and B information in the 6 × 6 pixel array shown in FIG. Yes. Each pixel has R, G, and B pixel values. For example, the image PA 'of the pixel 22 corresponding to the image PA in FIG. 4 includes R, G, and B color information.

  The generation of a high-quality image realized by super-resolution imaging by performing super-resolution imaging a plurality of times by such movement of the image sensor 22 accompanied by the imaging region overlap does not change the subject (image). It is assumed that there is no difference in the subject image in the image). That is, on the premise that the image sensor 22 is moved and photographed in a state where there is no movement / blur, brightness, hue change or the like in the subject image, the high-quality image has the same subject portion, the same brightness, and color. Must be made up of pixels with a based image.

  However, during super-resolution shooting, if the subject moves or the subject's brightness, hue, etc. change to cause a difference in the subject image, the pixels of the high-quality image are images based on the same shooting conditions. do not become. As a result, an image in which an inappropriate image appears partially or entirely is generated.

  Therefore, in the present embodiment, when a change in the subject image occurs, complementary shooting (acquisition of an image again) is performed. That is, a photographed image in which no change (difference) in the subject image occurs. Specifically, if the subject image has moved, the brightness of the subject image has changed, or the color of the subject image has changed, the photographing is performed again. A high-quality image is generated based on a plurality of photographed images with no change in the subject image.

  Here, three detection methods are prepared as methods for detecting the change in the subject image, and detection is possible based on the difference between the pixel values, the difference between the histograms, and the difference between the average values. Hereinafter, each detection method will be described with reference to FIGS.

  FIG. 6 is a diagram illustrating a difference between pixel values. For the two captured images I1 and I2 obtained by the first and second images, a difference in luminance value is calculated between corresponding pixels, and pixels whose difference is greater than or equal to the threshold T1 are counted. Then, when the count number of the pixels that are equal to or greater than the threshold T1 among the total number of pixels is equal to or greater than the threshold T2, it is determined that a change in the subject image has occurred. The threshold values T1 and T2 are determined in consideration of shooting conditions and the like, and are stored in advance in a ROM or the like.

  For example, when the subject is moving during super-resolution imaging, pixels that do not become the image of the same subject are generated between the obtained captured images. In FIG. 6, the images of the pixels PX are not based on the same subject in the captured images I <b> 1 and I <b> 2 obtained at the time of the first image capturing and the second image capturing.

  FIG. 7 is a diagram showing the difference between the average values of the entire image. An average luminance value is calculated for each of the two captured images I1 and I2 obtained by the first and second images. Then, a difference between the luminance average values is obtained, and when the difference amount is equal to or greater than the threshold value T3, it is determined that the subject image has changed.

  For example, if a change in illumination caused by flickering of a fluorescent lamp, etc., or a change in the brightness of the subject image caused by the operation of the shutter or aperture occurs during super-resolution shooting, the subject does not move. Regarding the brightness of the subject image, there is a difference between the captured images I1 and I2. Such a change in the subject image can be detected by calculating a difference between the average values.

  FIG. 8 is a diagram showing a histogram difference of the entire image. Histogram calculations are performed on the two captured images I1 and I2 obtained by the first and second images. Then, the difference between the histograms HG1 and HG2 obtained by the histogram calculation (see the hatched portion in FIG. 8) is obtained. If the difference amount is equal to or greater than the threshold value T4, it is determined that the subject image has changed.

  For example, if something unexpected (such as a bird) temporarily enters the shooting range during super-resolution shooting, the movement of the entire subject image changes relatively little, but the brightness, color, etc. Differences occur between I1 and I2. Even when the sun hidden behind the clouds leaks sunlight due to slight movement of the clouds, a difference occurs between the captured images. Such a change in the subject image can be detected by calculating a histogram difference.

  In this embodiment, the user can set any one detection method in mode setting or the like, or two or more detection methods can be combined and detected. As for the pixel difference value, the histogram difference value, and the average difference value, each difference based on the color information may be obtained instead of the luminance value and used, or the luminance value and the color information may be used in combination.

  FIG. 9 is a flowchart of super-resolution imaging and super-resolution processing in the present embodiment.

  The image sensor 22 is moved to a predetermined position at the time of photographing the first image and the second image, and photographing is performed (S201 to S204). Then, it is detected whether or not a subject image change has occurred between the first photographed image and the second photographed image (S205). If the subject image has changed, the process returns to step S201, the image sensor 22 is moved to the first position, and the first image is taken again (that is, the first image is taken for the second time).

  On the other hand, if there is no change in the subject image between the first and second photographed images, the process shifts to the third photograph (S206, S207) and the first and third photographed images. Whether or not the subject image has changed is determined (S208). If there is a change in the subject image, the process returns to step S201 and starts again from the first image.

  If it is determined in step S208 that the subject image has not changed, the process proceeds to the fourth shooting (S209, S210), and the subject image changes between the first and fourth shot images. It is determined whether or not it has occurred (S211). If there is a change in the subject image, the process returns to step S201 and starts again from the first image.

  If it is determined in step S211 that the subject image has not changed, super-resolution processing is executed based on the acquired four captured images, and a high-quality image is stored and displayed (S212 to S214).

  As described above, according to the present embodiment, in super-resolution imaging, is there a change in the subject image between the captured image obtained by the first imaging and the captured image obtained by the second and subsequent imaging? Is detected every time shooting is performed, and if the subject image changes, super-resolution shooting is restarted from the beginning.

  In this way, a high-quality image is obtained only from a photographic image in which the subject image has moved (difference has occurred in the subject image) and the subject image does not move. It is possible to efficiently shoot, display and record a high quality image without bothering to process, display and record the image.

  In addition, since a change in the subject image is detected with reference to the first photographed image, if a subject image has moved due to shooting of a moving subject or the like, it can be detected appropriately. For super-resolution shooting, in order to prevent long-time shooting, a limit is set on the number of repeated shooting (re-acquisition of images). If the limit is exceeded, the super-resolution shooting process is forcibly terminated once. If it is done, it is more efficient.

  Next, the digital camera which is 2nd Embodiment is demonstrated using FIG. In the second embodiment, when a change in the subject image occurs, the shooting performed immediately before is performed again. About another structure, it is substantially the same as 1st Embodiment.

  FIG. 10 is a flowchart of super-resolution imaging and super-resolution processing in the second embodiment.

  After the first and second images are taken, it is detected whether or not a subject image change has occurred between the first and second shot images (S301 to S305). If there is a change in the subject image, the second shot, the detection of the change in the subject image are performed for the second time, the third time, and so on until a captured image with substantially no change in the subject image is obtained.

  If no change in the subject image has occurred in step S305, the third image is shot, and it is detected whether or not the subject image has changed between the second and third shot images. (S306 to 308). If there is a change in the subject image, the third shot and the change detection of the subject image are performed for the second time, the third time, and so on until a captured image with substantially no change in the subject image is obtained.

  If it is determined in step S308 that there is no change in the subject image, the fourth image is taken, and whether there is a change in the subject image between the third image and the fourth image. Whether or not is detected (S309 to S311). If it is determined that a change in the subject image has occurred, the fourth shot and the detection of the change in the subject image are detected for the second time, the third time, and so on until a captured image with substantially no change in the subject image is obtained. And do.

  If it is determined in step S311 that the subject image has not changed, super-resolution processing is executed based on the obtained four captured images, and a high-quality image is stored and displayed (S312 to S314).

  As described above, according to the second embodiment, in super-resolution imaging, a captured image obtained by imaging of the (N−1) th sheet (N is an integer of 2, 3, or 4 here), Whether or not there is a change in the subject image with the shot image obtained in the Nth shooting is detected every time shooting is performed, and if there is a change in the subject image, shooting without change in the subject image The Nth image is taken until an image is obtained.

  Since a change in the subject image is detected between two captured images that are adjacent in time series, it is possible to appropriately detect changes in brightness and hue caused by illumination flicker or shutter / aperture variation. . In addition, by not performing shooting from the beginning, depending on the subject, super-resolution shooting may be able to be completed in a short time.

  Note that a limit may be provided for the number of repetitions of imaging. Further, the present invention is not limited to shooting four images, and super-resolution shooting over a plurality of images may be performed.

  Next, a third embodiment will be described with reference to FIG. In the third embodiment, “N (N is an integer of any one of 1, 2, 3, and 4) sheets” in which the shooting of four images is performed first and then the subject image changes after that. Retry only the "eye". About another structure, it is substantially the same as 1st Embodiment.

  FIG. 11 is a flowchart of super-resolution imaging and super-resolution processing in the third embodiment.

  The four images with the movement of the image sensor 22 are continuously taken to acquire four shot images and temporarily stored in a memory such as a RAM (S401 to S408). Then, after the super-resolution shooting is finished, for example, it is determined whether or not the subject image has changed in the second, third, and fourth shot images with reference to the first shot image (see FIG. S409). When the change in the subject image has occurred, the photographing for obtaining the photographed image is retried (S413 to S421). When the subject image has not changed, a high-quality image is stored and displayed by super-resolution processing based on the four images (S410 to S412).

  As described above, according to the third embodiment, super-resolution imaging (multiple imaging) is terminated first, and then a change in the subject image is detected and imaging is performed again. Thereby, depending on the subject, it may be possible to finish super-resolution shooting in a short time. In addition, it is possible to efficiently process even in the case of a brightness change without movement.

  In the first to third embodiments, changes in the subject image are detected by extracting two consecutively photographed images or the second and subsequent photographed images based on the first photographed image. However, the present invention is not limited to this, and a change in the subject image may be detected based on a photographed image by another combination. Alternatively, a change in the subject image may be detected based on three or more captured images.

  Next, a digital camera according to the fourth embodiment will be described with reference to FIG. In the fourth embodiment, the complementary shooting as in the first to third embodiments is not performed, the user is notified that the subject image has changed, and the super-resolution processing is not performed.

  FIG. 12 is a flowchart of super-resolution imaging and processing in the fourth embodiment.

  As in the first embodiment, each time the second, third, and fourth shots are taken, whether or not the subject image has changed is determined based on the first shot image ( S501 to S511). If the subject image has not changed, a high-quality image is generated by super-resolution processing and displayed / recorded (S512 to S514).

  On the other hand, when a change in the subject image occurs, the controller 40 does not execute the super-resolution process, stores each of the four captured images, and displays the first or other captured images on the LCD 24. (S514, S515). Then, the user is notified that the subject image has changed (S516). Here, a warning text (for example, “the subject image has changed!” Etc.) notifying the LCD 24 of the subject image change is displayed together with the first photographed image or only the warning text.

  As described above, according to the fourth embodiment, when a change in the subject image is detected, the super-resolution processing is not executed, the change in the subject image is notified to the user, and each captured image is stored. The captured image of is displayed. This prevents unnecessary super-resolution processing that fails as a result from being executed, displayed, and recorded. In addition, by detecting a change in the subject image for each shooting, it is possible to immediately stop super-resolution shooting and prompt the user to review the shooting environment. Since super-resolution processing is not performed even when the subject image changes that cannot be recognized by the user during shooting, unnecessary image processing, image recording, and image display are avoided. Moreover, since each captured image is stored individually, even if it is not suitable for super-resolution processing, it can be diverted to another purpose.

  Note that the notification of the subject image change to the user by the screen display can be configured other than the display, and the light emitting unit 90 (see FIG. 1) provided in the camera body 20 can emit light. Alternatively, a predetermined sound or message such as a buzzer sound may be emitted.

  Next, a digital camera according to the fifth embodiment will be described with reference to FIGS. In the fifth embodiment, super-resolution processing is performed regardless of the change in the subject image. On the other hand, if the subject image has changed, the user is notified of the change.

  FIG. 13 is a flowchart of super-resolution imaging and processing in the fifth embodiment. FIG. 14 is a view showing a display screen for informing a subject image change.

  The super-resolution imaging of four images is continued, and the first to fourth images are temporarily stored in the memory, while the images are generated by super-resolution processing, displayed, and stored (S601 to S601). S611). On the other hand, it is determined whether or not a subject image change has occurred in the second, third, and fourth shot images with reference to the first shot image (S612). If a change in the subject image has occurred, the user is notified of this by using the generated high-quality image (S613).

  FIG. 14 shows a high-quality image obtained by super-resolution imaging of two subjects SI and MI. Since movement or the like has occurred in the subject MI, image editing processing is performed on the image area of the subject MI so that the subject MI can be identified from the subject SI. Here, the subject image change is notified to the user by painting the subject MI with the same color.

  As described above, according to the fifth embodiment, when a change in the subject image is detected, the user can know the changed portion from the image obtained by the super-resolution processing. Even in an image in which the subject image change is difficult to recognize, the user can know the part of the change. An arbitrary captured image may be displayed and displayed so that a portion where the subject image has changed in the image can be identified. Further, the outline of the image area where the change has occurred may be highlighted. According to this, although a high-quality image including an error due to a change in the subject image is strictly generated, the position and range of the error can be clearly recognized by the user. If it is a narrow part of the outermost peripheral part of the screen, it may be possible to cope by trimming that part. It is possible to prevent the worst situation where the user is not aware of the error and releases or delivers it.

  Next, a sixth embodiment will be described with reference to FIG. In the sixth embodiment, a subject image change is detected before super-resolution imaging.

  FIG. 15 is a flowchart of super-resolution imaging and processing in the sixth embodiment. If the super-resolution processing mode is set under the through image display mode, the processing is started. Alternatively, the processing may be started after an autofocus operation or the like is performed by half-pressing the release button 15 under the through image display mode or the super-resolution processing mode setting.

  In step S701, it is determined whether or not the subject image has changed based on the pixel signals for one field / frame read at a predetermined time interval during the through image (preceding image) display. Here, since the position of the image sensor 22 is fixed, a change in the subject image is detected between pixel signals (pre-photographing images) for two consecutive fields / frames. Further, the threshold value used when detecting the subject image change described above is set to a value smaller than the threshold value described above.

  If there is a change in the subject image, a warning message is displayed to notify that change, or the LED 90 emits light (S702). On the other hand, if the subject image change is not detected and the release button 15 is fully pressed, super-resolution shooting and super-resolution processing are executed, and a high-quality image is displayed and stored (S703-S715).

  As described above, according to the sixth embodiment, when a change in the subject image is detected before the super-resolution imaging is started, the user is notified of the change. As a result, it is possible to prevent unnecessary super-resolution imaging (which fails even when image processing is performed). It should be noted that the change in the subject image may be detected from the pre-photographing images with four or other numbers, or the change in the subject image may be detected by motion vector detection or the like. Further, the present embodiment may be configured not to perform subject image change detection and complementary shooting or user notification in super-resolution shooting. Note that as a modification of the sixth embodiment, even when the release button 15 is fully pressed, the exposure operation is not performed immediately, and a change in the subject image is detected for a certain period. Super-resolution processing may be started and a high-quality image may be displayed and stored.

  In the first to sixth embodiments, super-resolution imaging by moving the image sensor by one pixel is performed to realize a high-quality image with improved color reproducibility. However, the amount of movement of the image sensor is 1. The super-resolution processing may be performed by moving the image sensor with an amount of movement larger than that or a sub-pixel (1 pixel or less) accuracy. That is, it may be applied to super-resolution processing based on a plurality of captured images.

  For super-resolution imaging, instead of moving the image sensor relative to the subject image, the subject image may be moved relative to the image sensor by driving the optical element eccentrically, and super-resolution imaging may be performed. An image processing unit may be provided separately from the controller. Digital cameras can be applied to both single-lens reflex cameras with removable lenses (including so-called “mirrorless single-lens cameras”) and compact cameras with integrated lenses. The present invention can also be applied to an imaging device.

  On the other hand, super-resolution processing can also be executed by image editing software or an image processing apparatus that reads a captured image recorded by a camera or the like and performs image editing processing. In this case, in association with each captured image data, a plurality of sets of captured image information and the presence / absence of changes in the subject image are recorded, and a plurality of sets of captured images captured by the image editing software or the image processing apparatus are recorded. When there is a change in the subject image, the user can recognize that the subject image has changed by notifying the user of that fact.

  As described above, in order to generate an image by super-resolution processing based on a plurality of images using a camera or the like, a plurality of images (exposure and exposure) are shifted while shifting the pixel position (image sensor position) at the time of shooting. When performing image acquisition), the user has conventionally moved when shooting multiple images, or when there is a change in brightness or color during shooting due to illumination, shutter, or aperture operation. Therefore, there is a possibility that the image processing is useless and the failed image is recorded and displayed as it is. If the high-resolution image generation by the super-resolution processing fails, the photographer must take another image while confirming the composition, the exposure condition, and the like. The inconvenience is solved and the photographing work can be performed efficiently.

10 Digital camera (imaging device)
22 Image sensor (image sensor)
40 controller (image processing unit, control unit, detection unit, pre-shooting detection unit, pre-shooting control unit)

Claims (16)

A control unit that controls the acquisition of a plurality of images while changing the relative position between the subject image and the image sensor;
An image processing unit that performs image processing based on the plurality of acquired images;
A detection unit that detects a difference in the subject image in at least a part between the acquired at least two images,
Wherein the control unit is, depending on the difference of the object image, a line cormorants imaging device a notification of differences in acquisition or the object image again image,
An image pickup apparatus , wherein the control unit acquires an image again for an image in which a difference between subject images is detected in the acquired image.   The imaging apparatus according to claim 1, wherein the detection unit detects a difference between the subject images based on a difference between at least two images.   The said detection part calculates | requires the difference of the brightness | luminance or color information between the corresponding pixels between at least two images, and detects the difference of the said subject image based on the calculated | required difference. 3. The imaging device according to any one of 2.   The said detection part calculates | requires the difference of the brightness | luminance histogram or color information histogram of at least 2 image, and detects the difference of the said to-be-photographed image based on the calculated | required difference. The imaging device described in 1.   The detection unit obtains a difference between an average value of luminance of at least two images or an average value of color information, and detects a difference between the subject images based on the obtained difference. 5. The imaging device according to any one of 4.   The detection unit detects at least one of a change in position of a subject image between at least two images, a change in brightness of the subject image, and a change in color of the subject image as a difference between the subject images. The imaging apparatus according to claim 1, wherein: The imaging apparatus according to claim 1 , wherein the control unit acquires an image again until an image in which a difference between subject images does not occur is obtained . The detection unit detects a difference between the subject images before completion of all acquisition of a plurality of images based on the image processing;
The said control part performs reacquisition from the image acquired last among the some images already acquired, if the difference of the said to-be-photographed image is detected, Any one of Claim 1 thru | or 6 characterized by the above-mentioned. An imaging apparatus according to claim 1. The detection unit detects a difference between a subject image of a reference image and a subject image of another image after completion of acquisition of a plurality of images based on the image processing,
The imaging apparatus according to claim 1, wherein when the difference in the subject image is detected, the control unit executes re-acquisition of the image in which the difference has occurred. The detection unit detects a difference between the subject images before completion of all acquisition of a plurality of images based on the image processing or after completion of acquisition of a plurality of images based on the image processing;
When the difference between the subject images is detected, the control unit executes at least one of storing at least one of the already acquired images and displaying at least one of the already acquired images. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. The detection unit detects a difference between the subject images after image processing based on the plurality of images;
The imaging apparatus according to claim 1, wherein when the difference between the subject images is detected, the control unit notifies the difference between the subject images. A pre-shooting detection unit that detects a difference in a subject image between a plurality of images obtained by an image sensor while changing the relative position before starting the acquisition of a plurality of images;
The imaging apparatus according to claim 1, further comprising: a pre-shooting control unit that notifies the difference between the subject images when the difference between the subject images is detected.   The imaging according to any one of claims 1 to 12, wherein the control unit notifies the difference between the subject images by displaying an image in which a portion where the difference between the subject images occurs is displayed. apparatus. The imaging apparatus according to claim 1, further comprising: a light emitting unit, wherein the control unit notifies the difference between the subject images by causing the light emitting unit to emit light. Execute acquisition of multiple images while changing the relative position of the subject image and the image sensor,
Perform image processing based on the acquired multiple images to generate another image,
Detecting at least a portion of the subject image difference between the acquired at least two images,
An imaging method, wherein an image is acquired again for an image in which a difference between subject images is detected in the acquired image. The imaging device
Control means for controlling acquisition of a plurality of images while changing the relative position between the subject image and the image sensor;
Image processing means for executing image processing based on the plurality of acquired images and generating another image;
Function as detection means for detecting a difference in at least a portion of the subject image between at least two acquired images;
The image differences of the subject image is detected in the acquired image, to perform acquisition of the image again, the program for causing to function as the control means.

Images