JP2020129753A - Imaging apparatus and imaging method - Google Patents

Abstract

To provide an imaging apparatus in which an image intended by a user is acquired by comparison lightness combination even in a case that there is sudden unnecessary light, and an imaging method.SOLUTION: A subjected image is captured by an imaging element and image data are generated and outputted. An image data combination unit performs comparison lightness combination on image data which are exposed and outputted for a predetermined exposure time by the imaging element. Image data which are outputted from the imaging element and of which the exposure amount is appropriate are stored in a basic image data storage unit as basic image data. Image data which are outputted from the imaging element and of which the exposure amount is excessive are stored in a subtraction image data storage unit as subtraction image data. The subtraction image data are made into correction image data by filtering processing (S31), second combination image data are generated by subtracting the correction image data from combination image data which are outputted by the image data combination unit, and comparison lightness combination is performed on the second combination image data and the basic image data stored in the basic image data storage unit (S33).SELECTED DRAWING: Figure 2B

Description

The present invention performs so-called comparative bright combining, in which image data is repeatedly read from an image sensor, pixel values of pixels at corresponding positions in each image data are compared, and the image values are replaced by the brighter pixel value to perform image combining. The present invention relates to an image pickup apparatus and an image pickup method capable of performing the above.

Conventionally, a comparative bright combination process has been known. This comparative bright combination process is an image process that compares pixel values at corresponding positions in image data that are repeatedly output, selects a pixel value having a large luminance value, and generates a combined image. By performing this comparative light synthesis process, for example, the light trails of the stars are extended due to the diurnal motion of the earth, and the background scenery coexists without becoming white, and it seems as if exposure was performed for a long time. Images can be obtained.

During shooting for the comparative bright combination processing, for example, a subject having extremely bright brightness may be captured due to sudden light emission. In the comparative bright combination process, when a pixel value with a high brightness is selected, this brightness is maintained, so even if the subject returns to the original brightness thereafter, the influence of the subject when it is extremely bright is affected. However, it will not be the intended composite image. Therefore, it has been proposed that the comparative bright combination processing is not executed when the average luminance value of the image data for combination captured repeatedly is larger than a predetermined value (see Patent Document 1).

JP, 2013-172372, A

In the comparative bright combination process disclosed in Patent Document 1, when the average luminance value of the image data for combination is larger than a predetermined value, the comparative bright combination process is not executed, so that there is a suddenly bright subject ( For example, when the subject in the foreground is illuminated by the headlights of the automobile), an image in which the light trails of stars are interrupted (for example, refer to the image 52 in FIG. 4A). In this case, the user discards the image taken for a long time and redoes the shooting again, which is not convenient.

The present invention has been made in view of such circumstances, and provides an image pickup apparatus and an image pickup method that can obtain an image intended by a user by comparative bright combination even when there is a sudden unnecessary light. ..

In order to achieve the above object, an image pickup apparatus according to a first aspect of the present invention includes an image pickup device that picks up a subject image, generates image data, and outputs the image data; and image data that is exposed and output by the image pickup device at a predetermined exposure time. An image data composition unit that repeatedly and repeatedly performs bright composition, a basic image data storage unit that stores image data that is output from the image sensor and has an appropriate exposure amount as basic image data, and an output amount that is output from the image sensor and has an excessive exposure amount. Image data storage unit that stores such image data as subtraction image data, and the subtraction image data is filtered to obtain correction image data, and the correction is performed based on the combined image data output from the image data combining unit. And a control unit for generating the second combined image data by subtracting the use image data and comparing and combining the second combined image data with the basic image data stored in the basic image data storage unit. ..

In the image pickup apparatus according to a second aspect of the present invention, in the first aspect of the present invention, the basic image data storage unit sets the image data that is first exposed and output among the image data that is repeatedly exposed and output as the basic image. Store as data.
In the image pickup apparatus according to a third aspect of the present invention, in the first aspect of the present invention, the control unit filters the subtraction image data with a smoothing filter to obtain the correction image data.
In the imaging device according to a fourth aspect of the present invention, in the third aspect of the present invention, the smoothing filter process is a process of interpolating pixel data of a target pixel with pixel data of pixels located around a target pixel to be filtered. Is.

In the image pickup device according to a fifth aspect of the present invention, in the first aspect of the invention, the control unit performs, as the filtering process, user operation of the subtraction image data to replace a designated pixel or a designated range of pixels with another level. Then, the corrected image data is generated.
In the image pickup apparatus according to a sixth aspect of the present invention, in the first aspect of the present invention, when the captured image has distance information, the control unit sets the pixel of the subject according to the distance information to another level when the captured image has distance information. The replacement is performed to generate the corrected image data.

According to a seventh aspect of the present invention, in the image pickup apparatus according to the first aspect, the subtraction image data storage section can store the first subtraction image data and the second subtraction image data. The control unit subtracts the first correction image data corresponding to the first subtraction image data from the synthesis image data to generate the second synthesis image data, and the second synthesis image The data and the basic image data are comparatively light-combined, and the second correction image data corresponding to the second subtraction image data is subtracted from the comparatively-light-combined image data to obtain the third combined image data. It is generated, and the third combined image data and the basic image data are subjected to comparative light combination.

According to an eighth aspect of the present invention, in the image pickup apparatus according to the first aspect, the subtraction image data storage section can store the first subtraction image data and the second subtraction image data. The control unit compares the first subtraction image data and the second subtraction image data with each other, and makes the combined image data the subtraction image data.
According to a ninth aspect of the present invention, in the image pickup apparatus according to the first aspect of the present invention, a plurality of pieces of subtraction image data are recorded, the recorded pieces of subtraction image data are compared and brightly combined, and used as correction subtraction image data. It is possible.
In the image pickup apparatus according to a tenth invention, in the first invention, the basic image data storage unit and the subtraction image data storage unit are arranged in the same memory.

An image pickup method according to an eleventh aspect of the present invention is configured such that a subject image is picked up by an image pickup device to generate and output image data, and image data exposed by the image pickup device at a predetermined exposure time is output by an image data synthesizing unit. Image data with appropriate exposure amount is stored as basic image data in the basic image data storage section after repeated comparison and light composition, and image data with excessive exposure amount is output from the image sensor and subtracted. Image data for subtraction stored in the subtraction image data storage unit, the subtraction image data is filtered to obtain correction image data, and the correction image data is subtracted from the combined image data output from the image data combining unit. Then, the second combined image data is generated, and the second combined image data and the basic image data stored in the basic image data storage section are compared and brightly combined.

According to the present invention, it is possible to provide an image pickup apparatus and an image pickup method that can obtain an image intended by a user by comparative bright combination even when there is sudden unnecessary light.

FIG. 1 is a block diagram mainly showing an electrical configuration of a camera of one embodiment of the present invention. 6 is a flowchart showing an operation of the camera of one embodiment of the present invention. 6 is a flowchart showing an operation of the camera of one embodiment of the present invention. 7 is a flowchart showing a modified example of the operation of the camera of one embodiment of the present invention. 7 is a flowchart showing a modified example of the operation of the camera of one embodiment of the present invention. It is a figure explaining the comparative bright synthetic|combination process in the camera which concerns on one Embodiment of this invention. It is a figure explaining the comparative bright synthetic|combination process in the camera which concerns on one Embodiment of this invention. It is a figure explaining the comparative bright synthetic|combination process in the camera which concerns on one Embodiment of this invention. It is a figure explaining the comparative bright synthetic|combination process in the camera which concerns on one Embodiment of this invention. It is a figure explaining the comparative bright synthetic|combination process in the camera which concerns on one Embodiment of this invention.

An example applied to a camera as one embodiment of the present invention will be described below. This camera has an imaging unit, converts the subject image into image data by the imaging unit, and performs live view display of the subject image on a display unit arranged on the back surface of the main body based on the converted image data. The photographer determines the composition and shutter timing by observing the live view display. During the release operation, the image data is recorded on the recording medium. The image data recorded on the recording medium can be reproduced and displayed on the display unit when the reproduction mode is selected.

Further, this camera can set a comparative bright combination mode in which a combined image is generated by the comparative bright combination processing. When the comparative bright combination mode is set, each time the image data is read from the image sensor, the pixel data at the corresponding pixel positions of the captured image data and the stored image data are compared, and the brighter pixel data An image is generated by combining the pixel data replaced with. When the starry sky is photographed for a long time in this comparative bright combination mode, an image in which the light traces of stars are reflected can be obtained.

Further, when the comparatively bright combination mode is set and sudden unnecessary light such as a headlight of an automobile is emitted, the image at this time is stored (for example, S15 in FIG. 2A, The image 55 in FIG. 4B) is generated based on this image for correction, which is overexposed by sudden unnecessary light (for example, S31 in FIG. 2B, image 56 in FIG. 4C). Then, a subtraction image in which the overexposed portion is erased is generated using the correction image (for example, S33 in FIG. 2B, image 57 in FIG. 4D). A comparative bright combination is performed using this subtracted image and a base image (for example, S11 in FIG. 2A, image 53 in FIGS. 4B and 4E) that is not affected by sudden unnecessary light. By this synthesizing process, it is possible to obtain a comparatively bright synthetic image in which light traces are not interrupted and the influence of sudden unnecessary light is removed.

FIG. 1 is a block diagram mainly showing the electrical configuration of a camera according to an embodiment of the present invention. The camera according to the present embodiment has an imaging unit 1, an image processing unit 10, a system control unit 20, a bus 31, and each unit connected to the bus 31.

The image pickup unit 1 includes a lens 2, a mechanical shutter 3, and an image sensor 4. In addition, in the present embodiment, description will be made assuming that the lens is integrally formed with the camera body. However, it goes without saying that the lens may be detachably attached to the camera body as an interchangeable lens. In this case, the image sensor is arranged on the camera body side, and a communication circuit for communication of data, control commands, etc. may be provided between the interchangeable lens and the camera body.

The lens 2 is an optical system that forms a subject light image (optical image of the subject) on the image sensor 4. The focus lens in the lens 2 moves in the optical axis direction of the lens 2 based on a control signal from the system control unit 20 to perform focus adjustment. The lens 2 has an optical diaphragm inside. The size of the aperture of the optical aperture changes according to the aperture value, and the amount of light on the image sensor 4 is adjusted. The aperture value is a control parameter for adjusting the exposure amount.

The mechanical shutter 3 is arranged on the optical axis of the lens 2 and in front of the image sensor 4. The mechanical shutter 3 controls the exposure time (shutter speed) for guiding the photographing light flux from the lens 2 onto the image sensor 4. If, for example, a focal plane shutter is adopted as the mechanical shutter 3, the photographing light flux from the lens 2 can reach the image sensor 4 by opening and closing the shutter front curtain and the rear curtain, or the photographing light flux can reach the image sensor 4. It is possible to block light so that it does not reach. At this time, the shutter speed is the time from the start of the shutter front curtain to the start of the rear shutter curtain. In the mechanical shutter 3, the opening time of the shutter is controlled based on the shutter speed calculated by the system control unit 20. The mechanical shutter 3 is not limited to the focal plane shutter, and may be, for example, a lens shutter, or the mechanical shutter may be eliminated and only the electronic shutter may be used.

The image sensor 4 is an image sensor such as a CMOS image sensor or a CCD image sensor. The image sensor has a plurality of pixels arranged two-dimensionally and converts each pixel into an electric signal. The subject light image formed by the lens 2 is converted into an electric signal by each pixel of the image pickup element to generate image data. The generated image data is output to the image processing unit 10 and the bus 31. The image sensor 4 has a plurality of pixels and functions as an image sensor that photoelectrically converts incident light and generates an image signal. The image sensor 4 also functions as an image sensor that captures a subject image, generates image data, and outputs the image data.

The bus 31 is a signal line for transmitting/receiving a signal between each unit in the camera. In the example shown in FIG. 1, the bus 31, the image pickup unit 1, the image processing unit 10, the distance information determination unit 15, the exposure determination unit 16, the system control unit 20, the internal memory 33, the external memory 36, the display unit 37, An input IF (Interface) 38, a condition setting unit 39, and a touch panel 40 are connected. Although the external memory 36 is illustrated as being arranged inside the camera in FIG. 1, the external memory 31 may be configured by a memory card or the like that can be attached to and detached from the camera. The configuration does not have to be unique.

The image processing unit 10 has an image processing circuit and performs various image processes on the image data output from the image sensor 4. The image processing unit 10 has an image data synthesis unit 11 and a development processing unit 14.

The image data composition unit 11 has an image composition circuit and composes a plurality of pieces of image data to generate at least one piece of composite image data. The image data synthesizing unit 11 includes a comparative bright synthesizing unit 12 and a filter processing unit 13.

The comparative bright synthesizing unit 12 has a comparative bright synthesizing circuit and performs a comparative bright synthesizing process. The comparative bright combination process is a process of roughly comparing the pixel data of the corresponding pixels of the two images and replacing the pixel data with the brighter pixel data to generate a comparative bright composite image. Specifically, the following processing is performed. The comparative light composition unit 12 compares the pixel data of the pixels arranged at the same position of the image sensor 4 with respect to the image data read from the image sensor 4 and the plurality of pieces of image data stored in the internal memory 33. .. Then, the comparative bright combination unit 12 generates a comparative bright combined image using the larger one of the results of comparing the pixel data, that is, the brighter pixel data. By performing such processing for all the pixel positions, each pixel data of the comparative bright composite image is composed of only the brightest pixel data. The comparative bright combination unit 12 functions as an image data combining unit that repeatedly performs comparative bright combination of image data that is exposed and output by the image sensor for a predetermined exposure time (see, for example, S17 in FIGS. 2A and 3A).

The filter processing unit 13 has a filter processing circuit and performs a filter process on the pixel data output from the image sensor 4. Examples of filter processing performed by the filter processing unit 13 include a median filter that calculates a median value of pixel values around a target pixel, an averaging filter that calculates an average value of pixel values in the vicinity of the target pixel, and the like.

The development processing unit 14 has a development processing circuit and performs various development processes on the generated RAW image data. The development processing includes demosaicing processing, white balance adjustment processing, noise reduction processing, YC signal generation processing, gamma correction processing, resizing processing, and image compression processing. The resizing process is a process of converting the number of pixels of the image data read out from the image sensor 4 into another number of pixels, and by this process, for example, the number of display pixels of the display unit 37 can be matched.

The distance information determination unit 15 has a distance measuring circuit or an AF (Auto focus) circuit and determines the distance to the subject. The distance information is acquired by contrast AF, phase difference AF, or the like. The contrast AF is a method of adjusting the focus from the peak position of the high frequency component of the image data from the image sensor 4, and the distance information can be obtained based on the focus lens position at the focus position. Further, in the phase difference AF, a phase difference detecting pixel that receives the light beam from the focus lens by dividing the pupil is arranged on the image pickup surface of the image sensor 4, and the phase difference is calculated from the output of the phase difference detecting pixel. Distance information can be obtained from this phase difference. Further, as the phase difference AF, in addition to arranging the phase difference detecting pixels on the image pickup surface of the image sensor 4, a dedicated phase difference detecting sensor is arranged and the calculation is performed based on the output from this sensor. You can Further, the distance information may be acquired by using a TOF (Time of Flight) that calculates the distance to the subject by measuring the time until the emitted light is reflected by the object and received.

The exposure determination unit 16 has an exposure determination circuit and uses image data from the image sensor 4 to determine whether overexposure, proper exposure, or underexposure. The exposure determination unit 16 can determine whether the image data in the exposure target range set by the touch panel 40 or the like is overexposed (for example, see S11 in FIG. 2A).

The strobe 17 has a light-emitting element for illumination such as a xenon tube or an LED built in the camera body, and can illuminate a subject in response to a light-emission command from the system control unit 20 during main photographing. it can. The external strobe 18 is a strobe that can be attached to the camera body. The external strobe 18 has a light emitting element for illumination, such as a xenon tube or an LED, inside the external strobe body, and can illuminate a subject in response to a light emitting command from the system control unit 20 during main photographing. it can.

The display unit 37 has a display panel such as a TFT (Thin Film Transistor) liquid crystal or an organic EL (Electro Luminescence). The display unit 37 is arranged as a rear display unit provided on the rear surface of the camera body, or an EVF (Electronic View Finder) for observing the display through the eyepiece. The display unit 37 displays the image subjected to the image processing by the image processing unit 10.

The input IF 38 is an operation unit for receiving various operations from the photographer and performing various settings and instructions for the camera. The input IF 38 includes, for example, an operation member such as a power button for turning on/off the power of the camera, a release button for operating the start and end of photographing, and is provided on the front surface of the rear display unit or the like. The touch panel 40 etc. are provided. The touch panel 40 detects the position touched by the user and outputs the detected position to the bus 31. When the touch panel 40 is provided, settings and instructions can be input by touch operation.

The condition setting unit 39 sets, on the menu screen displayed on the display unit 37, shooting conditions of the camera and conditions relating to processing of the shot image.

The internal memory 33 is an electrically rewritable volatile memory. The internal memory 33 stores the image data read from the image sensor 4 or the image data during the image processing by the image processing unit 10 (a basic image data recording unit 33a, a subtraction image data recording described later). See section 33b). The internal memory 33 is a storage unit that also stores various processing programs and setting values necessary for the operation of the camera. The CPU in the system control unit 20 controls each unit in the camera according to the processing program stored in the internal memory 33. The internal memory 3 is composed of, for example, a nonvolatile memory such as a flash memory or an SDRAM, a volatile memory, or a combination thereof.

The internal memory 33 has a basic image data recording unit 33a and a subtraction image data recording unit 33b. The basic image data recording unit 33a is an area for recording the image data of the first image captured when performing repeated image capture in order to perform the comparative bright combination processing (see, for example, S11 in FIGS. 2A and 3A). .. The basic image data recording unit 33a functions as a basic image data storage unit that stores image data output from the image sensor and having an appropriate exposure amount as basic image data (see, for example, S11 in FIGS. 2A and 3A). The basic image data storage unit stores, as the basic image data, the image data that is first exposed and output among the image data that is repeatedly exposed and output (for example, see S9 in FIGS. 2A and 3A).

The subtraction image data recording unit 33b is an area for recording the image data at that time when the exposure target range is overexposed (for example, see S13 in FIGS. 2A and 3A). The subtraction image data recording unit 33b functions as a subtraction image data storage unit that stores the image data that is output from the image sensor and has an excessive exposure amount as subtraction image data (see, for example, S15 in FIGS. 2A and 3A). ). The basic image data storage unit and the subtraction image data storage unit are arranged in the same memory. Of course, the basic image data storage unit and the subtraction image data storage unit may be arranged in different memories.

The external memory 36 is an electrically rewritable nonvolatile memory, such as a memory card that can be loaded into the camera body, or a nonvolatile storage medium fixed inside the camera body. The external memory 36 records the image data that has been image-processed for recording by the image processing unit 10, and the recorded image data is read during reproduction. The image data read from the external memory 36 can be output to the outside of the camera.

The system control unit 20 is a controller and is a processor having a CPU (Central Processing Unit), peripheral circuits, and the like. The system control unit 20 controls each unit in the camera according to the processing program stored in the internal memory 33.

The system control unit 20 filters the subtraction image data into correction image data (see, for example, S31 in FIG. 2B and S16 in FIG. 3A), and corrects the correction image from the combined image data output from the image data combining unit. The second synthetic image data is generated by subtracting the data (see, for example, S33 in FIG. 2B and S21 in FIG. 3B), and the second synthetic image data and the basic image data stored in the basic image data storage unit are combined. It functions as a control unit for performing comparative light synthesis (see, for example, S33 in FIG. 2B and S21 in FIG. 3B).

The control unit causes the filter processing unit to filter the subtraction image data by the smoothing filter to obtain the correction image data (see, for example, S31 in FIG. 2B and S16 in FIG. 3A). The above-described processing by the smoothing filter is processing of interpolating the pixel data of the target pixel with the pixel data of the pixels located around the target pixel to be filtered. As a filtering process, the control unit replaces the designated pixel or the designated range of pixels with another level by the user operation of the subtraction image data to generate corrected image data. As a filtering process, when the captured image has distance information, the control unit replaces the pixel of the subject according to the distance information with another level to generate the corrected image data.

Next, the comparative bright combination processing according to the present embodiment will be described using the flowcharts shown in FIGS. 2A and 2B. This flow is realized by the CPU in the system control unit 20 controlling each unit in the camera according to the program stored in the internal memory 33. In this flow (the same applies to FIGS. 3A and 3B described later), the processing required for the comparative bright combination is described, and the processing for realizing other functions is omitted.

When the flow shown in FIG. 2A starts, first, shooting conditions are set (S1). In this step, the condition setting unit 39 sets shooting conditions such as an aperture value, a shutter speed value, and ISO sensitivity. This setting may be manually set by the user through the input IF 38, and the system control unit 20 detects the subject brightness value based on the image data acquired by the image sensor 4, and based on this brightness value, The shooting conditions may be set automatically.

After setting the shooting conditions, the "exposure target range" is then designated (S3). The exposure target range is the range of the subject that is desired to be kept at proper exposure in the image even when unnecessary light such as sudden light emission occurs. This designation may be made by the user using the touch panel 40 or the like, or may be automatically set by the system control unit 20. For example, the range in which the subject on the near side is located may be set based on the distance information detected by the distance information determination unit 15. Further, the system control unit 20 may set the range in which the subject on the near side is located by analyzing the image data acquired by the image sensor. Further, in this step, the system control unit 20 also sets a threshold value used when detecting as overexposure. The threshold value may be preset as a design value, or may be manually set by the user.

When the exposure target range is set, next, an operation for starting shooting is performed (S5). When the user observes the live view image displayed on the display unit 37 and determines that the shutter timing is reached, the user operates the release button in the input IF 38 or touches the screen (touch panel) of the display unit 37. In this step, it is determined whether or not the release button has been operated or the touch operation has been performed. If the result of this determination is that the release button has been operated or the touch operation has been performed, shooting is started. When shooting is started, the image processing unit 10 performs image processing on the image data from the image sensor 4.

Then, it is determined whether or not it is the first shot (S9). In the case of performing the comparative bright combination processing, when shooting is started, the system control unit 20 repeatedly reads image data from the image sensor 4 at a predetermined frame rate. Then, the read image data is subjected to image processing and then temporarily stored in the internal memory 33. In this step, it is determined whether or not the image data is based on the first shot.

If the result of determination in step S9 is the first shot, it is saved as a base image (S11). Here, the system control unit 20 records the image data acquired by the first shooting as a “base image” in the basic image data recording unit 33a in the internal memory 33.

On the other hand, as a result of the determination in step S9, if it is not the first shot, that is, if the second and subsequent shots are taken, it is determined whether the exposure target range is overexposed (S13). Here, the exposure determination unit 16 determines whether or not the brightness of the image data in the exposure target range designated in step S3 exceeds the threshold value. An image 55 in FIG. 4B, which will be described later, is an example in which the brightness of the image data is over, and the subject on the near side is so-called whiteout.

If the result of determination in step S13 is that the exposure target range is overexposed, it is saved as a subtraction image (S15). In this step, the system control unit 20 records the image data determined to be overexposed in the subtraction image data recording unit 33b in the internal memory 33 as the “subtraction image”.

If the result of determination in step S13 is that the exposure range has not been overexposed, or if it has been recorded as a subtraction image in step S15, then comparative bright combination is performed and this is saved as a combined image (S17). In this step, the comparative light composition unit 12 performs the comparative light composition processing. In this way, the first photographed image data is temporarily stored in the internal memory 33 in a state where the comparative bright combination processing is not performed. When the second captured image data is acquired, the pixel data at the corresponding pixel positions of the first captured image data and the second captured image data are compared, and the brighter pixel data is combined to make the comparison clear. It is stored as image data. For all pixels, the brighter pixel data are compared and combined to generate a first combined image, and the combined image data is temporarily stored in the internal memory 33. After that, every time the Nth captured image data is acquired, the temporarily stored composite image data (the (N-1)th composite image) and the Nth captured image data are compared for each pixel. , Generates a composite image by replacing it with the brighter pixel data and saves it.

When the composite image is saved in step S17 or the base image is saved in step S11, the image is then displayed on the monitor (display unit 37) (S19). Here, the system control unit 20 causes the display unit 37 to display the composite image generated in step 17 as an intermediate process. In the case of the first image capturing, the base image stored in step S11 is displayed. Further, since the combined image generated in step S17 is subjected to the combining process in the overexposed range while being overexposed, the image affected by the sudden light emission is displayed on the monitor.

Next, it is determined whether or not it is a shooting end operation (S23). The user operates the release button again or performs a touch operation on the screen of the display unit 37 to end the shooting for the comparative bright combination processing. In this step, the system control unit 20 determines whether or not these ending operations have been performed. If the result of this determination is that the ending operation has not been performed, processing returns to step S7, and shooting for comparative bright combination processing continues.

If the result of determination in step S23 is that a shooting end operation has been performed, shooting is ended (S25). In this step, the system control unit 20 stops the image capturing process by the image sensor 4.

When the shooting end operation is performed, it is next determined whether or not an overexposed image is included (S27). As described above, during shooting, it is determined in step S13 whether or not the exposure target range is overexposed. In this step, after the shooting operation is completed, the system control unit 20 determines whether or not there is an overexposed image in the shot image.

If the result of determination in step S27 is that an overexposed image is not included, the composite image is saved in the external memory as the final image (S29). In this case, since the overexposed image is not included, it is not necessary to correct the whiteout portion. Therefore, in step S17, the system control unit 20 records the stored comparative bright composite image as the final image in the external memory 36.

On the other hand, if the result of determination in step 27 is that an overexposed image is included, in step S31 and subsequent steps, a comparative bright combined image from which sudden unnecessary light has been removed is generated. First, the subtraction image is filtered and saved as a “correction image” (S31). Here, the filter processing unit 13 performs the filter processing on the subtraction image stored in step S15. In other words, if the captured image includes an overexposed image, that is, if the “subtraction image” is stored in the internal memory, the “subtraction image” is filtered and the overexposed portion is processed. Other than the above are corrected, and the image after the filter processing is stored in the internal memory 33 as a “correction image”. For example, as will be described later with reference to FIG. 4B, when the overexposed image 55 (subtraction image) is filtered, the star image disappears, and a part of the subject that is overexposed due to sudden light emission remains. The image 56 (correction image) is obtained.

In step S31 (similarly to step S16 in FIG. 3A described later), the filter processing performed by the filter processing unit 13 includes, for example, the following processing.
(1) The bright spots of stars are deleted by the median filter.
(2) The bright spots of stars are deleted by the averaging filter.
(3) The object specified in the exposure target range is detected by contour detection, and the other levels are replaced with black levels.
(4) A range is specified by a user operation, and pixel data within this range is replaced with 0LSB. LSB is an abbreviation for least significant bit and is a minimum value bit in a binary number. As the range specification by the user operation, the area may be specified by the touch panel 40, or the subject may be selected and the other part may be replaced.
(5) When the distance information determination unit 15 obtains the distance information of the target object, the pixels corresponding to the target object having a different distance from the target object specified in the exposure target range may be replaced with the black level.

The filter processing unit 13 in the present embodiment is not limited to a filter process in a narrow sense such as a median filter or an average filter. That is, there is no problem if all the pixel data other than the specified exposure target range (subject necessary for the subtraction image) is set to the black level or 0LSB. Further, the correction may include a correction based on the surrounding pixel information, and may be replaced with another level.

After the correction image is stored in step S31, a comparative bright combination process is performed (S33). Here, first, the image processing unit 10 or the system control unit 20 subtracts the image data of the “correction image” from the image data of the “composite image”. That is, the correction image data stored in step S31 is subtracted from the combined image data generated and stored in step 17. This subtraction is performed for each pixel data at the corresponding pixel position. That is, in this process, the image (correction image) in which the image other than the overexposed portion is erased is subtracted from the composite image that has been subjected to the comparative bright combination using the overexposed image. In the image obtained by this subtraction processing (subtracted image), the overexposed subject due to unnecessary light is erased, while the light trace of the star remains as it is.

If the overexposed part (subject) is deleted by the subtraction process (“composite image”-“correction image”), then the comparative bright combination process is performed using the subtracted image (subtracted image) and the base image. To do. The base image is the first captured image stored in step S11. At the start of shooting, the shooting conditions are set so that the proper exposure is obtained, so the base image is an image shot with proper exposure. As described above, in the image subjected to the subtraction processing (subtraction image), the overexposed portion (subject) is erased while leaving the star trail. The base image is an image taken with proper exposure. By performing the comparative light combination processing using these two images, it is possible to combine the images in which the light trace of the star remains as it is but the spurious unnecessary light is removed.

When the comparative bright combination is performed in step S33, the comparative bright combined image is stored in the external memory (S35). Here, the system control unit 20 records the comparative bright composite image generated in step S33 in the external memory 36. When the image is stored in the external memory in step S29 or step S35, this flow ends.

As described above, in this flow, the first captured image is stored as the base image (S11), and when the second and subsequent captured images are overexposed, the image is used as the subtraction image. It is saved (S15). Then, each time image data is output from the image sensor 4, comparative bright combination is performed and the combined image is stored (S17). After shooting, if there is an overexposed image, a correction image obtained by filtering the subtraction image is saved (S31), and a subtraction image is generated by subtracting the correction image from the composite image, Comparative bright combination processing is performed using the subtracted image and the base image to generate a final comparative bright combined image (S33). Therefore, it is possible to obtain an image in which the overexposed portion is erased and replaced with proper exposure.

In addition, in the present embodiment, the first captured image is used as the base image. However, the base image may be the second and subsequent images, as long as the specified exposure target range is an image captured with proper exposure. That is, as long as the exposure of the exposure target range is proper, it is not limited to the first sheet.

In addition, in the present embodiment, it may be determined that the exposure target range is overexposed a plurality of times in step S13. In this case, the subtraction image data storage unit 33b is capable of storing at least the first subtraction image data and the second subtraction image data. Then, the first correction image data corresponding to the first subtraction image data is subtracted from the composite image data to generate the second composite image data, and the second composite image data and the basic image data (base Image) and subtractively subtracts the second correction image data corresponding to the second subtraction image data from the comparatively bright combined image data to generate third combined image data. The composite image data and the basic image data may be compared and lightly combined.

Further, the first subtraction image data and the second subtraction image data are compared and brightly combined, and the combined image data is used as the subtraction image data. If this subtraction image data can be created, in step S31, the subtraction image data can be filtered to create correction image data. Further, as another method, a plurality of image data for subtraction may be recorded, and the recorded image data for subtraction may be compared and brightly combined to create the image data for subtraction for correction. Also in this case, if the subtraction image data can be created, the correction image data can be created in step 31.

Next, a modified example of the comparative bright combination processing according to the present embodiment will be described using the flowcharts shown in FIGS. 3A and 3B. This flow is also realized by the CPU in the system control unit 20 controlling each unit in the camera according to the program stored in the internal memory 33.

In the comparative bright combination processing according to the embodiment of the present invention described above, after the shooting operation is completed, the overexposed subject (portion) is erased, and the comparative bright combination processing is performed so that this portion is properly exposed. Was there. On the other hand, in this modified example, the subject (portion) that is overexposed is erased during the shooting for the comparative bright combination, and the comparative bright combination processing is performed to obtain the proper exposure. There is. In the flowcharts shown in FIGS. 3A and 3B, steps for performing the same processes as those in FIGS. 2A and 2B are designated by the same step numbers, and detailed description thereof will be omitted.

When the flow of the comparative bright combination processing shown in FIG. 3A starts, first, the shooting conditions are set (S1), the exposure target range is designated (S3), and when the shooting start operation is performed (S5), shooting is performed ( S7). It is determined whether or not the first shot was taken (S9), and if the first shot was taken, the image is saved as a base image (S11). If the image is saved as the base image in step S11, or if the first image is not captured, it is next determined whether the exposure target range is overexposed (S13).

If the result of determination in step S13 is overexposure in the exposure target range, it is saved as a subtraction image (S15). Then, the subtraction image is filtered and stored as a correction image (S16). Here, the same filtering process as in step S31 is performed on the subtraction image stored in step S15. In the flow shown in FIGS. 2A and 2B, the filtering process is performed after the shooting operation is completed, but in the present modification, the subtraction image is subjected to the filtering process during shooting.

If the result of determination in step S13 is overexposure, or if in step S16 filter processing has been carried out and the image has been saved as a correction image, then comparative bright combining processing is carried out and the image is saved as a combined image (S17).

Next, it is determined whether the exposure target range is overexposed (S20). Here, the exposure determination unit 16 determines whether the exposure target range is overexposed as in step S13. If the result of this determination is overexposure, then comparative bright combination is carried out (S21). Here, first, the image processing unit 10 or the system control unit 20 subtracts the image data of the “correction image” from the image data of the “composite image”. That is, the correction image data stored in step S16 is subtracted from the composite image data generated and stored in step S17. This subtraction process is the same as step S33, so detailed description thereof will be omitted.

In step S21, when the image of the portion that is overexposed by the subtraction process (“composite image”-“correction image”) is erased, the comparative light combination unit 12 then performs the subtraction process, as in step S33. The comparative bright combination process is performed using the image (subtraction image) and the base image. Since this comparative bright combination processing is also the same as step S33, detailed description thereof will be omitted. In the flow shown in FIGS. 2A and 2B, the comparative bright combination processing is performed using the subtraction image and the base image after the shooting operation is completed, but in the present modification, the comparative bright combination processing is performed during shooting. I am trying to give it. The image data of the composite image by the comparative bright composition generated in step S21 is temporarily stored in the internal memory 33.

If the result of determination in step S20 is that the exposure target range has not been overexposed, or if comparative bright combination has been carried out in step S21, then a combined image is displayed on the monitor (S22). In step S21, the portion that is overexposed by the subtraction process (“composite image”-“correction image”) is erased, and the comparative bright combination process is performed using the subtracted image (subtracted image) and the base image. Since this is performed, the synthetic image is an image from which the unnecessary unnecessary light is removed. That is, the spurious unnecessary light is removed from the composite image displayed in step S22.

Next, it is determined whether or not it is a shooting end operation (S23). If the result of this determination is that the shooting operation has not ended, processing returns to step S7 and shooting for comparative bright combination is performed. On the other hand, if the result of determination in step S23 is that the shooting operation has ended, shooting ends (S25).

Upon completion of shooting, the composite image is then stored in the external memory as the final image (S29). Here, the system control unit 20 records the combined image stored in step S17 or S21 in the external memory 36. As described above, in the composite image stored in step S21, the unnecessary light that has suddenly been removed during shooting is removed. Therefore, unlike the case of the flow of FIGS. 2A and 2B, it is not necessary to remove sudden unnecessary light after the shooting operation is completed. When the composite image is stored in the external memory 29, this flow ends.

As described above, in this flow, when the first captured image is stored as the base image (S11), and the exposure target range of the second and subsequent captured images other than the first is overexposed (S13: Yes). ), the image is stored as a subtraction image (S15), and the subtraction image is filtered and stored as a correction image (S16). If the exposure target range is overexposed (S20: Yes), the correction image is subtracted from the composite image, and the comparative bright combination is performed using the subtracted image and the base image (S21). .. As described above, in the present modification, a filtering process and a subtracting process are performed during shooting to generate a combined image from which sudden unnecessary light is removed (see S21), and this combined image is displayed ( (See S22). Therefore, an image including unnecessary light is not displayed, and the user can concentrate on shooting without being aware of unnecessary light even during shooting.

Next, image processing in the embodiment and the modification of the present invention will be described with reference to FIGS. 4A to 4E. The image 51 shown in FIG. 4A is the final image intended by the user. In this image 51, a plurality of star light traces MLT are shown in the sky, and a subject Ob (the subject is a tree in FIG. 4A) designated as a shooting target range is shown on the closest side. The image 52 is a composite image generated by the comparative bright compositing process when an image affected by sudden unnecessary light is discriminated and the comparative bright compositing is not performed on the image (see Patent Document 1). In this case, each locus of the light trace MLT is interrupted at the location MLTb. The embodiment of the present invention and the modification thereof prevent the discontinuity of the locus.

The image 53 shown in FIG. 4B is a base image acquired in the first shooting (see S11 in FIGS. 2A and 3A). In this image 53, the subject Ob is photographed on the near side, and a plurality of stars ST are photographed in the sky in the shape of dots, which are photographed with an appropriate exposure. The image 54 is photographed a plurality of times and is subjected to comparative bright combination processing when the object Ob in the photographing target range is overexposed due to sudden unnecessary light in a state where comparative bright combination of image data is performed. It is an image after it. That is, the star ST is not affected by sudden unnecessary light, but the subject Ob is overexposed due to the effect of sudden unnecessary light. Since the image in which such overexposure has occurred is different from the image intended by the user, overexposure is corrected. In order to correct this overexposed subject Ob, first, the image 55 of the overexposed frame is saved as a subtraction image (see S15 in FIGS. 2A and 3A). This image 55 shows a plurality of stars ST and a subject Ob on the short-distance side that is overexposed.

Subsequently, as shown in FIG. 4C, filter processing is performed on the overexposed image 55 (subtraction image) (see S31 of FIG. 2B and S16 of FIG. 3A), and an image 56 (correction image) is obtained. To generate. The filtering process is a process of replacing the pixel value of the pixel of interest (in this case, a star) with the median value or average value of the pixels around the pixel of interest. Since the pixel value around the star is small, when the filter processing unit 13 filters the image 55, the pixel value at the position of the star becomes small. As a result, the image 56 in which the star ST disappears is obtained. However, since the object Ob has a large area and a plurality of overexposed pixel data are adjacent to each other, the object Ob remains overexposed and remains unerased even after the filtering process.

Next, as shown in FIG. 4D, the image 57 (subtraction image) is generated by subtracting the image 56 (correction image) from the image 54 (composite image) (see S33 in FIG. 2B and S21 in FIG. 3B). ). That is, the image 54 (composite image) is an image after comparative bright combination when the object Ob in the shooting target range is overexposed. Further, the image 56 (correction image) is an image in which only the object Ob that is overexposed by performing the filter processing as described above, removing the star ST, is captured. By subtracting the image 56 from the image 54, it is possible to obtain an image 57 in which the subject Ob that has been overexposed by unnecessary light disappears. The light trace MFT is reflected in this image 57.

When the image 57 (subtracted image) is generated, next, as shown in FIG. 4E, the comparative bright combination process using the image 57 (subtracted image) and the image 53 (base image) or the addition process of the image 54 and the image 53. (See S33 of FIG. 2B and S21 of FIG. 3B). The light trace MLT of the image 57 is not affected by the sudden unnecessary light, and the subject Ob affected by the sudden unnecessary light is removed. Further, the subject Ob of the image 53 (base image) is photographed with proper exposure as described above. Although the star ST is shown, it overlaps with the light trace MLT when performing the comparative light combination process or the addition process. Therefore, in the image 58 subjected to the comparative bright combination process or the addition process using the image 57 and the image 53, the subject Ob is properly exposed and the light trace MLT is not interrupted. That is, the image 58 becomes the same as the image 51 intended by the user.

As described above, in the embodiment and the modified example of the present invention, a subject image is picked up by an image pickup device to generate and output image data (for example, S5 in FIG. 2A), and a predetermined exposure time is set by the image pickup device. The image data that is exposed and output in (1) is repeatedly subjected to comparative bright combination by the image data combining section (for example, S17 in FIG. 2A). In addition, image data output from the image sensor and having an appropriate exposure amount is stored in the basic image data storage unit as basic image data (for example, S11 in FIG. 2A), and image data output from the image sensor and having an excessive exposure amount is stored. Is stored in the subtraction image data storage unit as subtraction image data (for example, S15 in FIG. 2A). Then, the subtraction image data is filtered to be the correction image data (for example, S31 in FIG. 2B), and the correction image data is subtracted from the composite image data output from the image data composition unit to obtain the second composite image. Data is generated, and the second combined image data and the basic image data stored in the basic image data storage unit are comparatively combined (for example, S33 in FIG. 2B). Therefore, even when there is a sudden unnecessary light, the image intended by the user can be acquired by the comparative bright combination.

It should be noted that in the embodiment and the modification of the present invention, the first image is stored in the basic image data storage unit as an image captured with proper exposure. However, the second and subsequent images may be used as long as they are captured with proper exposure, instead of the first image. Further, although the filter process is performed when the correction image is created from the subtraction image, the correction process is not limited to the narrow sense filter process, and may be performed by performing interpolation calculation on the surrounding pixel data, and the specified range Various arithmetic processing or image processing may be performed, such as replacing the level of the pixel data of (3) with another level (see (1) to (5) described above as the filter processing).

In addition, in the embodiment and the modification of the present invention, the distance information determination unit 15, the exposure determination unit 14, and the like are configured separately from the system control unit 20. However, of course, all or part of each unit may be configured by software and executed by the CPU in the system control unit 20. Further, each unit in the image processing unit 10 may be configured by a hardware circuit, or may be a hardware configuration such as a gate circuit generated based on a programming language described by Verilog, and a DSP ( A hardware configuration using software such as Digital Signal Processor) may be used. Further, each circuit unit in a processor configured by an integrated circuit such as FPGA (Field Programmable Gate Array) may be used. Alternatively, a processor including one or more CPUs may execute the functions of each unit by reading and executing a computer program recorded in a recording medium. Of course, these may be combined appropriately.

Further, in the embodiment and the modified example of the present invention, a digital camera is used as a device for photographing, but the camera may be a digital single lens reflex camera, a mirrorless camera, a compact digital camera, or a video camera. It may be a video camera such as a camera or a movie camera, and may be a mobile phone, a smartphone, a personal digital assistant (PC), a tablet computer, a camera built in a game machine, a medical camera, a microscope, or the like. It can be a camera for scientific equipment, a car-mounted camera, or a surveillance camera. In any case, the present invention can be applied to any photographing device capable of synthesizing a plurality of image data.

Further, among the techniques described in the present specification, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording medium and the recording unit may be recorded at the time of product shipment, may be used as a distributed recording medium, or may be downloaded via the Internet.

Further, in the embodiment of the present invention, the operation in the present embodiment is described using the flowchart, but the processing procedure may be changed in order, or any step may be omitted. Steps may be added, and the specific processing content in each step may be changed.

Further, with respect to the operation flow in the claims, the description, and the drawings, for convenience, even if description is made using words that express the order of “first”, “next”, etc. It does not mean that implementation in this order is essential.

The present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements within a range not departing from the gist of the invention in an implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some of all the constituent elements shown in the embodiment may be deleted. Furthermore, the constituent elements of different embodiments may be combined appropriately.

1... Image pickup unit, 2... Lens (aperture), 3... Mechanical shutter, 4... Image sensor, 10... Image processing unit, 11... Image combining unit, 12... Comparison Light combining unit, 13... Filter processing unit, 14... Development processing unit, 15... Distance information determination unit, 16... Exposure determination unit, 17... Strobe, 18... External strobe, 20... System control unit, 31... Bus, 33... Internal memory, 33a... Basic image data recording unit, 33b... Basic image data recording unit, 36... External memory, 37. ..Display unit, 38... Input IF, 39... Condition setting unit, 40... Touch panel

Claims (11)

An image sensor that captures a subject image, generates image data, and outputs the image data;
An image data synthesizing unit for repeatedly comparing and brightly synthesizing image data that is exposed by the image sensor for a predetermined exposure time and is output,
A basic image data storage unit that stores image data output from the image sensor and having an appropriate exposure amount as basic image data,
A subtraction image data storage unit that stores the image data output from the image pickup device and having an excessive exposure amount as subtraction image data;
The subtraction image data is filtered to be correction image data, and the correction image data is subtracted from the composite image data output from the image data composition unit to generate second composite image data, A control unit for comparing and combining the composite image data of No. 2 and the basic image data stored in the basic image data storage unit;
An image pickup apparatus comprising: The image pickup apparatus according to claim 1, wherein the basic image data storage unit stores, as the basic image data, image data that is first exposed and output among the image data that is repeatedly exposed and output. .. The image pickup apparatus according to claim 1, wherein the control unit filters the subtraction image data with a smoothing filter to obtain the correction image data. The image pickup apparatus according to claim 3, wherein the process by the smoothing filter is a process of interpolating pixel data of a target pixel with pixel data of pixels located around a target pixel to be filtered. The control unit generates the corrected image data by performing a user operation on the subtraction image data to replace a designated pixel or a designated range of pixels with another level as the filtering process. The imaging device according to. The control unit, as the filter processing, when the captured image has distance information, replaces the pixel of the subject according to the distance information with another level to generate the corrected image data. The imaging device according to 1. The subtraction image data storage unit is capable of storing first subtraction image data and second subtraction image data,
The control unit subtracts the first correction image data corresponding to the first subtraction image data from the synthesis image data to generate the second synthesis image data, and the second synthesis image data. And the basic image data are subjected to comparative bright combination, and the second correction image data corresponding to the second subtraction image data is subtracted from the comparatively bright combined image data to generate third combined image data. The image pickup apparatus according to claim 1, wherein the third combined image data and the basic image data are subjected to comparative bright combination. The subtraction image data storage unit is capable of storing first subtraction image data and second subtraction image data,
2. The control unit according to claim 1, wherein the first subtraction image data and the second subtraction image data are compared and brightly combined, and the combined image data is used as the subtraction image data. Imaging device. The image pickup apparatus according to claim 1, wherein a plurality of pieces of subtraction image data are recorded, and the recorded pieces of subtraction image data are compared and brightly combined to obtain subtraction image data for correction. The imaging apparatus according to claim 1, wherein the basic image data storage unit and the subtraction image data storage unit are arranged in the same memory. The subject image is captured by the image sensor to generate and output image data,
Image data that is exposed and output at a predetermined exposure time by the image pickup device is repeatedly compared and brightly combined by an image data combining unit,
The image data output from the image pickup device and having an appropriate exposure amount is stored in the basic image data storage unit as basic image data,
The image data output from the image sensor and having an excessive exposure amount is stored in the subtraction image data storage unit as subtraction image data,
The subtraction image data is filtered to be correction image data, and the correction image data is subtracted from the composite image data output from the image data composition unit to generate second composite image data, The comparative image composition of the composite image data of No. 2 and the basic image data stored in the basic image data storage unit is performed.
An imaging method characterized by the above.