JP5184325B2 - Image capturing apparatus, image capturing apparatus control method, and program - Google Patents

Description

  The present invention relates to a photographing apparatus and a photographing apparatus control method particularly suitable for multi-frame photographing, and a program for causing a computer to execute the photographing apparatus control method.

  In scenes where the shutter speed is slow and the effects of camera shake are strong in an imaging device such as a digital camera, multi-frame shooting is performed multiple times with an exposure period shorter than the appropriate exposure period, and is acquired by multi-frame shooting. In addition, by aligning and synthesizing a plurality of images, an image free from the influence of camera shake can be obtained.

In such multi-frame shooting, a technique has been proposed in which a plurality of images are acquired by performing a plurality of shootings by controlling the exposure time in accordance with the amount of blur of the shooting apparatus (see Patent Document 1). There has also been proposed a method of controlling the gain when simultaneously reading and combining a plurality of images according to the motion of a moving object included in the plurality of images (see Patent Document 2). In addition, when an image pickup device such as a CMOS that can partially read out charges is used in the photographing apparatus, and when reading out charges from the image pickup device, when reading at a high screen rate, the center of the image pickup device is used. In the case where charges are read from only a partial area and are read at a normal screen rate, a method of reading charges from the entire area of the image sensor has been proposed (see Patent Document 3).
Japanese Patent Laid-Open No. 2003-32540 Japanese Patent Laid-Open No. 5-37851 Japanese Patent Application Laid-Open No. 11-10312

  The methods described in Patent Documents 1 and 2 read out charges from the entire area of the image sensor used in the imaging apparatus, and therefore the frame rate is slow. When the frame rate is thus reduced, the tracking of the motion with respect to the moving object becomes worse, and the afterimage of the moving object becomes longer. As a result, the image quality of the composite image decreases. In addition, since the charge is read from the image sensor a plurality of times, especially when the image sensor has a noise persistence such as CMOS, there is residual noise in the charge every time the charge is read. Therefore, when combining a plurality of images by superimposing, noise is superimposed, and therefore, there is a problem that the noise becomes conspicuous. On the other hand, the method described in Patent Document 3 can increase the frame rate when reading is performed at a high screen rate, but is acquired because the charge is read only from the central portion of the image sensor. The angle of view of the image to be narrowed.

  The present invention has been made in view of the above circumstances, and an object thereof is to improve the image quality of an image acquired by multiframe shooting.

An imaging device according to the present invention includes an image sensor that photoelectrically converts a subject image to obtain an image signal;
By repeatedly reading out charges from only the moving object region corresponding to the moving object included in the subject image in the image sensor in response to the movement of the moving object region until the exposure time reaches a predetermined time from the start of exposure. The plurality of first image signals are acquired, and when the predetermined time is reached, the image sensor is controlled to acquire the second image signal by reading out charges from the entire area of the image sensor. The image pickup control means is provided.

  The “moving object region” means a region where a subject moving relatively with respect to the image pickup device at the time of image pickup is picked up among the image pickup regions of the image pickup device. Note that the moving object region may be a region only of the moving object, or may be a region in a predetermined range surrounding the moving object.

In the photographing apparatus according to the present invention, a moving object detection unit that detects the position of the moving object and predicts the moving position based on an image represented by an image signal acquired by the imaging element before the start of the exposure. We shall prepare further,
The imaging control means may be means for controlling the imaging element so as to read out the electric charge only from the moving body region based on the position of the moving body and the moving position.

  “Before the start of exposure” means before the first and second images for generating the composite image are acquired by pressing the release button. Specifically, it means a state in which a through image is displayed on the monitor before shooting.

  In the imaging apparatus according to the present invention, the imaging control unit reads the charge from an area including the next moving body area where the charge is read next time, when the charge is read from the moving body area once. The image sensor may be controlled so as to obtain the first image signal by discarding the charges read from the area including the moving object area.

  In the imaging apparatus according to the present invention, the imaging control unit reads the charge from an area including the next moving body area where the charge is read next time, when the charge is read from the moving body area once. The image sensor may be controlled to acquire the first image signal together with the electric charge read from the region including the moving body region.

  The “region including the next moving body region where the charge is read next time” may be a region including only the next moving body region, or may be a region including the next moving body region and other regions. For example, in the case of a region including only the next moving body region, the reading of charges is performed from the moving body region and the next moving body region that are targets of charge reading. In the case of a region including a region other than the next moving body region, the region from which the charge is read can be a region that surrounds both the moving body region to be read and the next moving body region. In particular, it is possible to easily read out charges from the image sensor by making the area from which charges are read out a rectangular area.

  In the photographing apparatus according to the present invention, a plurality of first images represented by the plurality of first image signals and a second image represented by the second image signals are synthesized to form a composite image. It is also possible to further include a synthesis means for generating

In the photographing apparatus according to the present invention, the brightness of the first image and the second image is corrected according to the exposure time of each pixel in the plurality of first images and the second image. It further includes brightness correction means,
The synthesizing unit may be a unit that generates the synthesized image by synthesizing the plurality of first images and the second image after the brightness correction is performed.

A method for controlling a photographing apparatus according to the present invention is a method for controlling a photographing apparatus including an imaging element that photoelectrically converts a subject image to obtain an image signal,
By repeatedly reading out charges from only the moving object region corresponding to the moving object included in the subject image in the image sensor in response to the movement of the moving object region until the exposure time reaches a predetermined time from the start of exposure. Obtaining a plurality of first image signals;
When the predetermined time is reached, the second image signal is obtained by reading out charges from the entire area of the image sensor.

  In addition, you may provide as a program for making a computer perform the control method of the imaging device by this invention.

  According to the present invention, until the exposure time reaches a predetermined time from the start of exposure, the reading of charges from only the moving object region corresponding to the moving object included in the subject image in the image sensor is repeated corresponding to the movement of the moving object region. A plurality of first image signals are acquired, and when a predetermined time is reached, the charge is read from the entire area of the image sensor and the second image signal is acquired. As described above, according to the present invention, the charge is repeatedly read from only the moving object region corresponding to the moving object in the image sensor, and therefore, when the first image signal is acquired as compared with the case of reading the charge from the entire region of the image sensor. The frame rate can be increased. For this reason, the afterimage of the moving object is less likely to occur in the first image represented by the first image signal, and as a result, the moving object included in the composite image has a high image quality without an afterimage. it can. On the other hand, since the second image signal is acquired only once when the exposure time reaches a predetermined time, the frame rate is not affected. Further, since the second image signal is acquired only once, the influence of residual noise of the image sensor is small. Therefore, it is possible to improve the image quality of a composite image obtained by combining a plurality of first images and second images. Further, since the second image signal uses charges read from the entire area of the image sensor, the angle of view of the composite image is not reduced.

  In addition, by predicting the position of the moving object and its moving position, and reading out charges from the moving object region based on the position of the moving object and its moving position, the first image represented by the first image signal is obtained. The moving object can be surely included.

  Further, at the time of reading the charge once from the moving object region, the charge is also read from the region including the next moving object region where the charge is read next time, the charge read from the next moving object region is discarded, and the first image signal is obtained. By acquiring, the exposure time of a moving body area | region can be made the same. For this reason, it is not necessary to correct the brightness of the first image at the time of generating the composite image, or the brightness correction process can be simplified, so that the brightness correction process is performed at high speed. Can do.

  Further, at the time of reading the charge once from the moving object region, the charge is read also from the region including the next moving object region where the charge is read next time, and the first image signal is acquired together with the charge read from the next moving object region. Thus, the exposure time of the moving object region can be made the same. For this reason, it is not necessary to correct the brightness of the first image at the time of generating the composite image, or the brightness correction process can be simplified, so that the brightness correction process is performed at high speed. Can do. Further, since the read charges can be used effectively, the degree of brightness correction at the time of generating the composite image can be reduced, and as a result, the image quality of the composite image can be further improved.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. 1 and 2 are views showing the appearance of a digital camera 1 to which a photographing apparatus according to an embodiment of the present invention is applied. As shown in FIGS. 1 and 2, a release button 2, a power button 3, and a zoom lever 4 are provided on the top of the digital camera 1.

  The release button 2 has a structure in which two types of operations can be instructed by pressing in two steps. For example, in shooting using an automatic exposure adjustment function (AE: Auto Exposure) and an automatic focus adjustment function (AF: Auto Focus), the digital camera 1 performs a first pressing operation (half-pressing) in which the release button 2 is lightly pressed. (Also called exposure), preparation for shooting such as exposure adjustment and focusing is performed. In this state, when a second pressing operation (also referred to as full pressing) in which the release button 2 is strongly pressed is performed, the digital camera 1 starts exposure and records image data for one screen obtained by the exposure. Record on media.

  On the back of the digital camera 1, a monitor 5 such as a liquid crystal, a mode dial 6 used for setting a shooting mode, and various operation buttons 8 are provided. In the present embodiment, it is possible to set a shooting mode for shooting, a playback mode for playing back an image recorded on a recording medium on the monitor 5, and the like. In addition to the normal shooting mode for performing normal shooting, the multi-frame mode for performing multi-frame shooting can be set as the shooting mode.

  Next, the internal configuration of the digital camera 1 will be described. FIG. 3 is a schematic block diagram showing an internal configuration of a digital camera to which the photographing apparatus according to the embodiment of the present invention is applied. As shown in FIG. 3, the digital camera 1 to which the photographing apparatus according to the present embodiment is applied has an imaging system 9.

  The imaging system 9 includes a photographing lens 10 including a focus lens 10a for focusing on a subject and a zoom lens 10b for realizing a zoom function. Each lens can be moved in the optical axis direction by a focus lens driving unit 11 and a zoom lens driving unit 12 each including a motor and a motor driver. The focus lens driving unit 11 controls the movement of each lens based on an instruction from the AF processing unit 28 described later, and the zoom lens driving unit 12 based on an instruction from the CPU 40 according to the operation of the zoom lever 4. .

  The diaphragm 14 includes a plurality of diaphragm blades. The aperture drive unit 15 is a small motor such as a stepping motor, and adjusts the position of the aperture blade so that the aperture size of the aperture becomes a size suitable for the purpose according to the aperture value data output from the AE processing unit 29. .

  The shutter 16 is a mechanical shutter and is driven by a shutter driving unit 17. The shutter drive unit 17 controls the opening and closing of the shutter 16 according to a signal generated by pressing the release button and the shutter speed data output from the AE processing unit 29.

  An image sensor 18 is provided behind the shutter 16. In the present embodiment, it is assumed that a CMOS type image sensor 18 is used. The imaging element 18 has a photoelectric surface in which a large number of light receiving elements are two-dimensionally arranged, and subject light that has passed through an optical system such as the photographing lens 10 forms an image on the photoelectric surface and is subjected to photoelectric conversion. In front of the photocathode, a microlens array for condensing light on each pixel and a color filter array in which filters of R, G, and B colors are regularly arranged are arranged. The image sensor 18 outputs the electric charge accumulated for each pixel as an analog photographing signal one pixel at a time in synchronization with the readout signal supplied from the image sensor controller 19. The time from the start of charge accumulation in each pixel until the charge is read out, that is, the shutter speed of the electronic shutter, is determined by the electronic shutter drive signal supplied from the image sensor control unit 19. This exposure time is set by an AE processing unit 29 described later. The gain of the image sensor 18 is adjusted by the image sensor control unit 19 so that an analog image signal having a predetermined size can be obtained.

  The analog photographing signal read from the image sensor 18 is input to an analog front end (AFE) 20. The AFE 20 includes a correlated double sampling circuit (CDS) that removes noise from the analog signal, an auto gain controller (AGC) that adjusts the gain of the analog signal, and an A / D converter (ADC) that converts the analog signal into a digital signal. It consists of. The image data converted into the digital signal is RAW data having R, G, and B density values for each pixel.

  The timing generator 21 generates a synchronization signal. By supplying this timing signal to the shutter drive unit 17, the image sensor control unit 19, and the AFE 20, the release button is operated, the shutter 16 is opened and closed, and the image sensor 18. The readout of the charges from the AFE 20 and the processing of the AFE 20 are synchronized.

  In addition, the digital camera 1 has a flash 24 that emits light when necessary during photographing. The light emission of the flash 24 is controlled by the light emission control unit 23.

  The digital camera 1 also has an image input controller 25 that transfers the image data output from the AFE 20 to another processing unit via the data bus 41, and a frame memory 26 that temporarily stores the image data transferred from the image input controller 25. Is provided.

  The frame memory 26 is a working memory used when performing various processes described later on image data. For example, an SDRAM (Synchronous Dynamic Random Access Memory) that performs data transfer in synchronization with a bus clock signal having a constant period. Is used.

  The display control unit 27 displays the image data stored in the frame memory 26 on the monitor 5 as a through image, or displays the image stored on the recording medium 34 on the monitor 5 in the reproduction mode. is there. The through image is captured by the imaging system 9 at predetermined time intervals in synchronization with the synchronization signal generated by the timing generator 21 while the imaging mode is selected.

  The AF processing unit 28 and the AE processing unit 29 determine shooting conditions based on the pre-image. The pre-image is represented by image data stored in the frame memory 26 as a result of the CPU 40 having detected a half-press signal generated by half-pressing the release button causing the image sensor 18 to perform pre-photographing. It is an image.

  The AF processing unit 28 detects the in-focus position of the focus lens 10a based on the pre-image. As a method for detecting the in-focus position, for example, a TTL method for detecting the in-focus position using the feature that the contrast of the image is high when the desired subject is in focus is used.

  The AE processing unit 29 measures the subject brightness based on the pre-image, and sets and outputs the aperture value and the shutter speed as the exposure set value based on the measured subject brightness (AE process). Specifically, when the metering method is divided metering, the pre-image is divided into 64 × 8 metering areas, for example, 8 × 8, and the brightness of each area and the program lines determined in advance and stored in the internal memory 35 described later. Based on the figure, the shutter speed and aperture value are set. When the metering method is center-weighted metering or spot metering, the shutter speed and the aperture value are set based on the brightness of the central area or the predetermined area.

  Here, in the digital camera 1 according to the present embodiment, when the normal shooting mode is set, one exposure is performed once from the image sensor 18 in response to a full pressing operation of the release button. Shooting to read out the charge. On the other hand, when the multi-frame mode is set, the charge reading is usually performed once in a plurality of times during the exposure period, and the charge is read at each time. Acquire multiple images. Note that a plurality of images are combined as described later to generate a combined image. In the multi-frame mode, the AE processing unit 29 sets the exposure time after division, that is, the charge readout interval (T) based on the shutter speed set based on the pre-image, that is, the exposure time (T0). And the number of times of reading n is set.

  During shooting in the normal shooting mode, the shutter 16 is opened at the shutter speed set by the AE processing unit 29, and the image sensor 18 is exposed for an exposure time T0 corresponding to the shutter speed, and one charge is read out. Is called. On the other hand, at the time of shooting in the multi-frame mode, the shutter 16 is opened after the release button is pressed until the charge is read for the set number of times of reading n, and the interval of the set exposure time T during that time. As a result, the charge is repeatedly read from the image sensor 18 to obtain a plurality of images.

  In the present embodiment, in the multi-frame mode, charges are read out only from the moving object region corresponding to the detected moving object position and the predicted moving object movement position, as will be described later. In addition, when reading the last charge, the charge is read from the entire area of the image sensor 18. Specific charge readout will be described later.

  The AWB processing unit 30 automatically adjusts the white balance at the time of shooting (AWB processing).

  The image processing unit 31 performs image quality correction processing such as brightness correction, gradation correction, sharpness correction, and color correction on the image data of the main image, and Y data that is RAW data as luminance signals and a blue color difference signal. YC processing is performed for conversion into YC data consisting of Cb data that is and Cr data that is a red color difference signal. The main image is an image represented by image data that is captured from the image sensor 18 by main shooting executed when the release button is fully pressed and is stored in the frame memory 26 via the AFE 20 and the image input controller 25. It is. In addition, the image processing unit 31 performs brightness correction processing according to the exposure time on an image acquired by shooting in a multi-frame mode to be described later.

  The compression / decompression processing unit 32 performs a compression process on the image data of the main image processed by the image processing unit 31 in a compression format such as JPEG, and generates an image file. A tag storing incidental information such as shooting date and time is added to the image file based on the Exif format or the like.

  The media control unit 33 accesses a recording medium 34 that is detachably set on a media throttle (not shown), and controls writing and reading of an image file.

  The internal memory 35 stores various constants set in the digital camera 1, a lookup table, a program executed by the CPU 40, and the like.

  The digital camera 1 also includes a moving object detection unit 36 and a synthesis unit 37.

  The moving object detection unit 36 detects a moving object included in the image. Specifically, when shooting in the multi-frame mode, a pixel having a large movement is obtained between a plurality of through images acquired continuously before the release button is pressed, and an area including the pixel is detected as a moving object area. To do. For example, as shown in FIG. 4, when three through images G1 to G3 are acquired, a subject with a large movement is a bird. For this reason, the moving body detection part 36 detects a bird as a moving body. In the present embodiment, an area composed only of a moving object may be detected as a moving object, but an area in a predetermined range surrounding the moving object may be detected as a moving object.

  The moving object detection unit 36 detects the moving direction and moving speed of the moving object, and estimates the moving position of the moving object at the time of charge reading based on the moving direction and moving speed. Note that the movement direction is calculated by calculating the movement vector of the pixels in the corresponding moving object region when a plurality of through images are superimposed. FIG. 5 is a diagram for explaining the calculation of the movement vector. Note that FIG. 5 will be described using two through images G1 and G2 among the three through images G1 to G3 shown in FIG. Since the bird is a moving body in the through images G1 and G2, as shown in FIG. 5, the beak position O1 of the bird in the through image G1 moves to the position O2 in the through image G2. Therefore, the moving object detection unit 36 obtains pixel positions in the through images G1 and G2 for the pixels corresponding to each other in the moving object region corresponding to the moving object of the through image, and based on the difference between the coordinate values of the pixel positions, the moving vector Is calculated. FIG. 5 shows a state where the movement vector V0 of the bird's beak is calculated. Then, the moving speed of the moving object is calculated by dividing the magnitude of the moving vector by the shooting time interval of the through image.

  Thus, by calculating the moving direction and moving speed of the moving object, it is possible to predict the position of the moving object at the time of charge reading in multi-frame mode imaging. Note that the moving object detection unit 36 outputs information on the detected position of the moving object and the predicted movement position to the CPU 40.

  The synthesizing unit 37 synthesizes a plurality of images acquired by shooting in the multi-frame mode and corrected in brightness so that the positions of moving objects are the same position, thereby generating a synthesized image. Note that the composition may be performed by adding the corresponding pixel values between the plurality of images after performing the positioning of the moving object for a plurality of images, or by calculating an average value of the pixel values. Good. Here, alignment is performed by selecting one reference image from a plurality of images and rotating, enlarging or reducing other images so that the position of the moving object in the other image matches the position of the moving object included in the reference image. This can be done by deforming.

  The CPU 40 controls each part of the main body of the digital camera 1 according to signals from various processing units such as the release button 2, the operation button 8, and the AE processing unit 29.

  The data bus 41 is connected to various processing units, the frame memory 26, the CPU 40, and the like, and exchanges image data and various instructions.

  Next, processing performed in the present embodiment will be described. FIG. 6 is a flowchart showing processing performed in the first embodiment. Since the present invention has a feature in the shooting process in the multi-frame mode, in the embodiment described below, the process when the multi-frame mode is set will be described.

  After setting to the multi-frame mode, the moving object detection unit 36 detects the position of the moving object (step ST1). Then, the CPU 40 determines whether or not the release button 2 is half-pressed (step ST2), and if step ST2 is negative, the process returns to step ST1. When the release button 2 is half-pressed, step ST2 is affirmed, the imaging system 9 performs pre-shooting (step ST3), and the AE processing unit 29 performs AE processing using the pre-image acquired by pre-shooting (step ST3). Step ST4). As a result, the exposure time T and the number of charge readouts n for one shooting in the multi-frame mode are set.

  Next, the CPU 40 starts monitoring whether or not the release button 2 has been fully pressed (step ST5). When step ST5 is affirmed, the main photographing is started, and the charge readout count is set to an initial value (i = 1). Step ST6). Next, the CPU 40 reads out charges from only the moving object region corresponding to the moving object in the image sensor 18 based on the position of the moving object detected by the moving object detection unit 36 or the information on the moving position thereof (step ST7).

  FIG. 7 is a diagram for explaining the first charge reading. As shown in FIG. 7, the CPU 40 reads out charges from only the pixels corresponding to the moving body region D <b> 1 in the image sensor 18. In FIG. 7 and the following description, the moving object region is indicated by a rectangle. Here, the read electric charges are converted into digital image data as described above, and are temporarily stored in the frame memory 26. In addition, an image represented by image data acquired by reading out charges from only the pixels of the moving object region is referred to as a first image G1-i (i = 1 to n−1). In the first charge readout, as shown in FIG. 7, the exposure time is 1T in the moving body region D1 and other regions in the image sensor 18, that is, the entire region.

  Next, the CPU 40 determines whether or not the next reading is the last reading (step ST8). If the determination in step ST8 is negative, the number of charges read is set next (i = i + 1, step ST9). Return to step ST7. In step ST7, the charge is read from the moving object region Di based on the information on the moving position of the moving object predicted by the moving object detection unit 36. On the other hand, when step ST8 is affirmed, the readout time of electrolysis is set to the last time (i = n, step ST10), and charge is read from the entire area of the image sensor 18 (step ST11). Thereby, the second image G2 is acquired.

  8 to 11 are diagrams showing the positions of moving object regions and the exposure time of the image sensor 18 from the second time to the fifth time when the number of exposures n = 5. First, at the time of reading the charge for the second time, as shown in FIG. 8, the charge is read from the moving body region D2 to obtain the first image G1-2. At this time, since the charge has been read from the moving object region D1 last time, the exposure time of the moving object region D1 is 1T. Further, in the moving object region D2, the exposure time of the region D1 ′ overlapping with the moving object region D1 is 1T, the exposure time of the moving object region D2 (excluding the region D1 ′) is 2T, and the exposure time of other regions is 2T.

  At the time of reading the charge for the third time, as shown in FIG. 9, the charge is read from the moving body region D3 to obtain the first image G1-3. At this time, the exposure time of the moving object region D1 (excluding the region D1 ′) is 2T, and the exposure time of the moving object region D2 is 1T. In the moving body region D3, the exposure time of the region D2 ′ overlapping the moving body region D2 is 1T, the exposure time of the moving body region D3 (excluding the region D2 ′) is 3T, and the exposure time of the other regions is 3T.

  At the time of reading the charge for the fourth time, as shown in FIG. 10, the charge is read from the moving object region D4 to obtain the first image G1-4. At this time, the exposure time of the moving object region D1 (excluding the region D1 ′) is 3T, the exposure time of the moving object region D2 (excluding the region D2 ′) is 2T, and the exposure time of the moving object region D3 is 1T. In the moving body region D4, the exposure time of the region D3 ′ overlapping with the moving body region D3 is 1T, the exposure time of the moving body region D4 (excluding the region D3 ′) is 4T, and the exposure time of other regions is 4T.

  When the charge is read for the fifth time, as shown in FIG. 11, the second image G2 is acquired by reading the charge from the entire area including the moving body area D5. At this time, the exposure time of the moving object region D1 (excluding the region D1 ′) is 4T, the exposure time of the moving object region D2 (excluding the region D2 ′) is 3T, and the exposure time of the moving object region D3 (excluding the region D3 ′) is 2T. The exposure time of the moving body region D4 is 1T. In the moving body region D5, the exposure time of the region D4 ′ overlapping the moving body region D4 is 1T, the exposure time of the moving body region D5 (excluding the region D4 ′) is 5T, and the exposure time of the other regions is 5T.

  Subsequent to step ST11, the image processing unit 31 performs image processing including brightness correction processing on the first image G1-i (i = n-1) and the second image G2 (step ST12). Here, as shown in FIG. 7 to FIG. 11, the first image G1-i and the second image G2 have different exposure times depending on the exposure time because the exposure time differs for each location in the region. It is different. Therefore, the image processing unit 31 performs brightness correction processing corresponding to the exposure time on the first image G1-i and the second image G2. Hereinafter, the brightness correction process will be described. Here, a process when four first images G1-1 to G1-4 and a second image G2 are acquired as shown in FIGS. 7 to 11 will be described.

  First, the first image G1-1 has an exposure time of 1T for the entire region. Here, the exposure time of the region with the longest exposure time in the second image G2 is 5T. In addition, since four first images G1-1 are acquired, when all the first images G1-i are added, the maximum exposure time in the added images is equivalent to 4T. Therefore, when all the first images G1-i and the second image G2 are combined, the maximum exposure time of the combined image is equivalent to 5T. For this reason, the image processing unit 31 does not correct the brightness of the first image G1-1 because the exposure time is 1T.

  Next, in the first image G1-2, the exposure time of the area corresponding to the area D1 ′ overlapping the moving object area D1 is 1T, and the other exposure time is 2T. Therefore, the image processing unit 31 performs the brightness correction process so that the brightness of the area other than the area corresponding to the area D1 ′ in the first image G1-2 is halved. As a result, the brightness of the entire area of the first image G1-2 corresponds to the brightness of the exposure time 1T.

  Next, in the first image G1-3, the exposure time of the area corresponding to the area D2 ′ overlapping the moving object area D2 is 1T, and the other exposure time is 3T. For this reason, the image processing unit 31 performs the brightness correction process so that the brightness of the area other than the area corresponding to the area D2 ′ in the first image G1-3 becomes 1/3. Thereby, the brightness of the entire region of the first image G1-3 corresponds to the brightness of the exposure time 1T.

  Next, in the first image G1-4, the exposure time of the area corresponding to the area D3 ′ overlapping the moving object area D3 is 1T, and the other exposure time is 4T. For this reason, the image processing unit 31 performs the brightness correction process so that the brightness of the area other than the area corresponding to the area D3 ′ in the first image G1-4 becomes ¼. Thereby, the brightness of the entire region of the first image G1-4 corresponds to the brightness of the exposure time 1T.

  Finally, for the second image G2, the moving object regions D1 to D1 are set so that the exposure time of the moving object region D5 corresponds to 1T, and the exposure times of the moving object regions D1 to D4 and the regions D1 ′ to D4 ′ correspond to 5T. The brightness of D5 and areas D1 'to D4' is corrected. Specifically, since the exposure time is 4T in the region corresponding to the moving object region D1 (excluding the region D1 ′) in the second image G2, the brightness is increased to 5/4 times. Correction processing is performed. In addition, since the exposure time is 3T in the region corresponding to the moving object region D2 (excluding the region D2 ′) in the second image G2, the brightness correction process is performed so that the brightness becomes 5/3 times. I do. In addition, since the exposure time is 2T for the area corresponding to the moving object area D3 (excluding the area D3 ′) in the second image G2, the brightness correction process is performed so that the brightness becomes 5/2 times. I do. In addition, since the exposure time is 1T for the region corresponding to the moving object region D4 in the second image G2, the brightness correction process is performed so that the brightness becomes five times. Further, since the exposure time is 5T for the area corresponding to the moving object area D5 (excluding the area D4 ′) in the second image G2, the brightness correction process is performed so that the brightness becomes 1/5 times. . As a result, the brightness of the moving object region D5 corresponds to the exposure for the exposure time 1T. Furthermore, since the exposure time is 5T for the areas other than the moving object areas D1 to D5 in the second image G2, the brightness correction process is not performed.

  Next, the combining unit 37 generates a combined image by superimposing these images so that the positions of the moving object included in the second image G2 and the moving object included in the first image G1-i coincide with each other. (Step ST13). As a result, the brightness of the entire area of the composite image is equivalent to a brightness with an exposure time of 5T.

  Then, the compression / decompression processing unit 32 generates an image file from the image data of the composite image subjected to the image processing (step ST14), and the media control unit 33 records the image file on the recording medium 34 (step ST15). The process is terminated.

  As described above, according to the first embodiment, the charge is repeatedly read out only from the moving object region corresponding to the moving object in the image sensor 18, and therefore, compared to the case where the charge is read out from the entire region of the image sensor 18. When acquiring the image Gi-1, the frame rate can be increased. For this reason, the afterimage of the moving object is less likely to occur in the first image Gi-1, and as a result, the moving object included in the composite image can have high image quality without an afterimage. Further, since the second image G2 is acquired only once at the time of the last exposure, the frame rate is not affected. Further, since the second image G2 is acquired only once, it is possible to reduce the influence of residual noise of the image sensor 18. Therefore, high quality of the composite image can be achieved. In addition, since the second image G2 uses charges read from the entire area of the image sensor 18, the angle of view of the composite image is not reduced.

  In addition, since the position of the moving object and its moving position are predicted, and the charge is read from the moving object region based on the position of the moving object and its moving position, the first image Gi-1 is reliably displayed on the moving object. Can be included.

  In the first embodiment, the charge is read out only from the moving body region at the movement position predicted when the first image G1-i is acquired. The charge may be read out from the moving body region at the moving position. Hereinafter, this will be described as a second embodiment. FIGS. 12 to 16 are diagrams for explaining charge reading in the second embodiment.

  First, as shown in FIG. 12, at the time of reading the charge for the first time, the charges are simultaneously read from the moving body region D1 and the moving body region D2, and further, the charges read from the region other than the region D1 ′ in the moving body region D2 (shown by hatching). Is discarded and the first image G1-1 is acquired. The exposure time is 1T in the moving body region D1 and other regions in the image sensor 18, that is, in all regions.

  Next, at the time of reading the charge for the second time, as shown in FIG. 13, the charges are simultaneously read from the moving object region D2 and the moving object region D3, and further read from the region other than the region D2 ′ in the moving object region D3 (shown by hatching). The charge is discarded and the first image G1-2 is acquired. At this time, since the charge is read from the moving body region D2 at the time of the first charge reading, the exposure time of the moving body region D2 is 1T. Further, the exposure time of the moving body region D1 (excluding the region D1 ′) is 1T, and the exposure time of the other regions is 2T.

  Next, at the time of reading the charge for the third time, as shown in FIG. 14, the charges are simultaneously read from the moving object region D3 and the moving object region D4, and further read from the region other than the region D3 ′ in the moving object region D4 (shown by hatching). The charge is discarded and the first image G1-3 is acquired. At this time, since the charge is read from the moving body region D3 at the time of the second charge reading, the exposure time of the moving body region D3 is 1T. In addition, the exposure time of the moving object region D1 (excluding the region D1 ′) is 2T, the exposure time of the moving object region D2 (excluding the region D2 ′) is 1T, and the exposure time of other regions is 3T.

  Next, at the time of reading charges for the fourth time, as shown in FIG. 15, charges are simultaneously read from the moving object region D4 and the moving object region D5, and further read from regions other than the region D4 ′ in the moving object region D5 (shown by hatching). The charge is discarded and the first image G1-4 is acquired. At this time, since the charge is read from the moving body region D4 when the charge is read for the third time, the exposure time of the moving body region D4 is 1T. The exposure time of the moving object region D1 (excluding the region D1 ′) is 3T, the exposure time of the moving object region D2 (excluding the region D2 ′) is 2T, and the exposure time of the moving object region D3 (excluding the region D3 ′) is 1T. The exposure time for other areas is 4T.

  When the charge is read for the fifth time, as shown in FIG. 16, the second image G2 is acquired by reading the charge from the entire area including the moving object area D5. At this time, since the charge is read from the moving body region D5 at the time of the fourth charge reading, the exposure time of the moving body region D5 is 1T. The exposure time of the moving object region D1 (excluding the region D1 ′) is 4T, the exposure time of the moving object region D2 (excluding the region D2 ′) is 3T, and the exposure time of the moving object region D3 (excluding the region D3 ′) is 2T. The exposure time for the moving object region D4 (excluding the region D4 ′) is 1T, and the exposure time for the other regions is 5T.

  Note that the brightness correction process in the second embodiment is performed as follows. First, in the second embodiment, the exposure time of all the first images G1-i is 1T. For this reason, the image processing unit 31 does not correct the brightness of all the first images G1-i. For the second image G2, the moving object regions D1 to D5 and the moving object regions D1 to D5 and the exposure times of the moving object regions D1 to D4 and the regions D1 ′ to D4 ′ correspond to 5T. The brightness of the areas D1 ′ to D4 ′ is corrected. Specifically, since the exposure time is 4T in the region corresponding to the moving object region D1 in the second image G2 (excluding the region D1 ′), the brightness is increased to 5/4 times. Correction processing is performed. In addition, since the exposure time is 3T in the region corresponding to the moving object region D2 (excluding the region D2 ′) in the second image G2, the brightness correction process is performed so that the brightness becomes 5/3 times. I do. In addition, since the exposure time is 2T for the area corresponding to the moving object area D3 (excluding the area D3 ′) in the second image G2, the brightness correction process is performed so that the brightness becomes 5/2 times. I do. In addition, since the exposure time is 1T for the region corresponding to the moving object region D4 (excluding the region D4 ′) in the second image G2, the brightness correction process is performed so that the brightness becomes 5 times. . In addition, the brightness correction process is not performed on the area corresponding to the moving object area D5 in the second image G2 because the exposure time is 1T. Furthermore, since the exposure time is 5T for the areas other than the moving object areas D1 to D5 in the second image G2, the brightness correction process is not performed.

  As described above, in the second and third embodiments, when one charge is read from the moving body region, the charge is read also from the next moving body region where the charge is read next time, and the charge read from the next moving body region is read. Since the first image G1-i is acquired by discarding, the exposure time of the moving object region can be made the same. For this reason, it is not necessary to correct the brightness of the first image G1-i at the time of generating the composite image, and thus the brightness correction process can be performed at high speed.

  In the second embodiment, the read charges are discarded. However, a synthesized image may be generated using the read charges as they are. Hereinafter, this will be described as a third embodiment.

  In the second embodiment, at the time of the first reading, the charge is simultaneously read from the moving object region D1 and the moving object region D2, and further, the charge read from the region other than the region D1 ′ in the moving object region D2 is discarded. In the third embodiment, the first image G1-1 is acquired using the area other than the area D1 ′ in the moving body area D2 as it is. Further, at the time of the second reading, the charges are simultaneously read from the moving object region D2 and the moving object region D3, and further, the charges read from the regions other than the region D2 ′ in the moving object region D3 are discarded, and the region D2 ′ in the moving object region D3 is discarded. The first image G1-2 is acquired using the other areas as they are. Thereafter, the first image G1-1 to G1-4 and the second image G2 are acquired without discarding the charge in the same manner during the third to fifth readings.

  FIG. 17 is a diagram illustrating first images G1-1 to G1-4 acquired in the third embodiment. As shown in FIG. 17, in the first images G1-1 to G1-4, an image corresponding to the moving object region from which charges are read next time is given. Note that the charge corresponding to the moving object region D5 in the first image G1-4 is discarded.

  Note that the brightness correction processing and composition in the third embodiment are performed as follows. First, in the third embodiment, the exposure time of the area corresponding to the moving object area in all the first images G1-i is 1T. For this reason, the image processing unit 31 does not correct the brightness of the first image G1-i. In addition, regions other than the region corresponding to the moving object region in all the first images G1-i are separated from the first image G1-i and overlapped with the corresponding positions in the second image G2. Thereby, the exposure time of each region on the second image G2 corresponds to that shown in FIG.

  Accordingly, in the second image G2, a region corresponding to the moving object region D1 (excluding the region D1 ′), a region corresponding to the moving object region D2 (excluding the regions D1 ′ and D2 ′), and a region corresponding to the moving object region D3 (region). Since the exposure time is 4T in the area corresponding to the moving object area D4 (except for the areas D3 ′ and D4 ′), the brightness is 5/4 times. Perform brightness correction processing. In addition, since the exposure time is 3T for the areas corresponding to the areas D1 ′, D2 ′, and D3 ′ in the second image G2, the brightness correction process is performed so that the brightness becomes 5/3 times. Do. In addition, the brightness correction processing is not performed on the area corresponding to the moving object area D5 (except the area D4 ′) in the second image G2 because the exposure time is 1T. For the region D4 ′, since the first image G1-i is added here, a brightness correction process for increasing the brightness to ¼ is performed. Furthermore, since the exposure time is 5T for the areas other than the moving object areas D1 to D5 in the second image G2, the brightness correction process is not performed.

  As described above, in the third embodiment, when one charge is read from the moving body region, the charge is also read from the next moving body region where the charge is read next time, and the first reading is performed together with the charge read from the next moving body region. By acquiring the image G1-i, the exposure time of the moving object region can be made the same. For this reason, it is not necessary to correct the brightness of the first image at the time of generating the composite image, whereby the brightness correction process can be performed at high speed. Further, since the read charges can be used effectively, the degree of brightness correction at the time of generating the composite image can be reduced, and as a result, the image quality of the composite image can be further improved.

  In the first to third embodiments, the first image G1-i and the second image are set so as to correspond to the brightness of the region having the maximum exposure time in the second image G2. Although the brightness of G2 is corrected, the brightness of the first image G1-i and the second image G2 may be corrected so as to have a predetermined brightness.

  In the second and third embodiments, charges are read from the moving object region Di to be read out of charges and the moving object region Di + 1 that reads out charges next, but the two moving object regions Di and Di + 1 You may make it read an electric charge from the area | region containing both. For example, when reading charges from the moving object regions D1 and D2, the charges may be read from a region A1 circumscribing the moving object regions D1 and D2, as shown in FIG. In this case, in the second embodiment, the first image G1-1 is acquired by discarding the charges read from the region A1 other than the moving body region D1 (indicated by hatching). In the third embodiment, an image corresponding to the region A1 is acquired as the first image G1-1. In this case, the charge A can be easily read out from the image sensor 18 by setting the area A1 from which the charge is read out to a rectangular area. Note that the brightness correction processing of the second image may be performed in the same manner as in the second and third embodiments, depending on the exposure time of the area from which the charge has been read.

  The digital camera 1 according to the embodiment of the present invention has been described above. However, the computer functions as a unit corresponding to the AE processing unit 29, the moving object detection unit 36, and the synthesis unit 37, and the processing as illustrated in FIG. A program for performing the above is also one embodiment of the present invention. A computer-readable recording medium that records such a program is also one embodiment of the present invention.

The figure which shows the external appearance of the digital camera to which the imaging device by embodiment of this invention is applied (front side) The figure which shows the external appearance of the digital camera to which the imaging device by embodiment of this invention is applied (back side) 1 is a schematic block diagram showing an internal configuration of a digital camera to which a photographing apparatus according to an embodiment of the present invention is applied. Diagram for explaining detection of moving object Diagram for explaining movement vector calculation The flowchart which shows the process performed in 1st Embodiment. The figure for demonstrating the reading of the electric charge of the 1st time in 1st Embodiment. The figure for demonstrating the reading of the electric charge of the 2nd time in 1st Embodiment. The figure for demonstrating the reading of the charge of the 3rd time in 1st Embodiment. The figure for demonstrating the reading of the charge of the 4th time in 1st Embodiment. The figure for demonstrating the reading of the charge of the 5th time in 1st Embodiment. The figure for demonstrating the reading of the electric charge of the 1st time in 2nd Embodiment. The figure for demonstrating the reading of the electric charge of the 2nd time in 2nd Embodiment. The figure for demonstrating the read-out of the charge of the 3rd time in 2nd Embodiment. The figure for demonstrating the reading of the charge of the 4th time in 2nd Embodiment. The figure for demonstrating the reading of the charge of the 5th time in 2nd Embodiment. The figure which shows the 1st image in 3rd Embodiment. The figure for demonstrating the brightness correction process in 3rd Embodiment. The figure which shows the other example of the reading of an electric charge

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Digital camera 5 Monitor 9 Imaging system 10 Shooting lens 11 Focus lens drive part 23 Light emission control part 24 Flash 28 AF process part 29 AE process part 36 Moving body detection part 37 Synthesis | combination part 40 CPU

Claims (8)

An image sensor that photoelectrically converts a subject image to obtain an image signal;
By repeatedly reading out charges from only the moving object region corresponding to the moving object included in the subject image in the image sensor in response to the movement of the moving object region until the exposure time reaches a predetermined time from the start of exposure. The plurality of first image signals are acquired, and when the predetermined time is reached, the image sensor is controlled to acquire the second image signal by reading out charges from the entire area of the image sensor. An imaging apparatus comprising: an imaging control means for performing Based on an image represented by an image signal acquired by the image sensor before the start of exposure, further comprising a moving object detecting means for detecting the position of the moving object and predicting its moving position;
The image pickup control means is means for controlling the image pickup device so as to read out the electric charge only from the moving object region based on the position of the moving object and the moving position. Shooting device.   The imaging control unit reads the charge from the region including the next moving body region where the charge is read next time, and reads the charge read from the region including the next moving body region when reading the charge from the moving body region once. The photographing apparatus according to claim 1, wherein the photographing device is means for controlling the image sensor so as to discard the first image signal.   The imaging control unit reads the charge from the region including the next moving body region where the charge is read next time, and reads the charge from the region including the next moving body region when reading the charge from the moving body region once. The photographing apparatus according to claim 1, wherein the photographing device is a unit that controls the image pickup device so as to acquire the first image signal.   The image processing apparatus further includes a combining unit that combines the plurality of first images represented by the plurality of first image signals and the second image represented by the second image signal to generate a combined image. The imaging device according to claim 1, wherein: Brightness correction means for correcting the brightness of the first image and the second image according to the exposure time of each pixel in the plurality of first images and the second image;
6. The means for synthesizing the plurality of first images and the second image after the brightness correction is performed to generate the synthesized image. Shooting device. A method for controlling an imaging apparatus including an image sensor that photoelectrically converts a subject image to acquire an image signal,
By repeatedly reading out charges from only the moving object region corresponding to the moving object included in the subject image in the image sensor in response to the movement of the moving object region until the exposure time reaches a predetermined time from the start of exposure. Obtaining a plurality of first image signals;
When the predetermined time is reached, a second image signal is obtained by reading out charges from the entire area of the image sensor, and a method for controlling the photographing apparatus. A program for causing a computer to execute a control method of an imaging device including an imaging device that photoelectrically converts a subject image to acquire an image signal,
By repeatedly reading out charges from only the moving object region corresponding to the moving object included in the subject image in the image sensor in response to the movement of the moving object region until the exposure time reaches a predetermined time from the start of exposure. Obtaining a plurality of first image signals;
A program for causing a computer to execute a procedure of acquiring a second image signal by reading out electric charges from all areas in the image sensor when the predetermined time is reached.

Images