JP4404823B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus such as a digital still camera or a digital video camera.

  An imaging apparatus capable of simultaneously capturing and recording moving images and still images has been proposed. This type of imaging apparatus creates a still image during moving image shooting by extracting only one frame of the moving image and performing a still image processing. Alternatively, the moving image shooting is temporarily interrupted to perform still image shooting, and the moving image shooting is resumed after the still image shooting is completed.

  Further, in Patent Document 1 below, when still image shooting is instructed, the aperture is controlled so as to be closed from the aperture value at the time of moving image shooting, and the aperture is completely closed after the start of the closing operation of the aperture. An image pickup apparatus provided with means for controlling the image pickup device to perform charge sweeping is disclosed.

Japanese Patent No. 3308716

  When shooting a scene with a lot of movement as a still image, it is originally desirable to shoot so as to stop the movement of the subject by increasing the shutter speed. On the other hand, in moving image shooting, it is desirable to perform shooting so as to be exposed over one frame period of a moving image in order to smoothly express the motion of a moving object.

  However, in the conventional method in which one frame of a moving image is extracted and still image processing is performed, the exposure time over one frame period of the moving image becomes longer with respect to the movement of the subject, and subject blur (moving subject blur) increases. It is impossible to obtain a clear still image in which the movement of the subject desired by the person is stopped. This problem becomes prominent when trying to obtain a still image of a fast-moving subject. Also, in the method of temporarily interrupting the moving image shooting, the moving image is interrupted and becomes unsightly. The technique described in Patent Document 1 cannot sufficiently solve the above problem.

  Accordingly, the present invention provides an imaging apparatus that can obtain an appropriate still image even when still image shooting is performed during moving image shooting, and that can suppress deterioration in quality of the moving image caused by still image shooting. With the goal.

  In order to achieve the above object, an imaging apparatus according to the present invention includes an imaging element that photoelectrically converts an incident optical image, an amplification circuit that amplifies and outputs a signal from the imaging element, and an output signal of the amplification circuit. In an imaging apparatus that includes a signal processing unit that processes image data obtained, and a main control unit that controls exposure settings of the imaging device and controls the shooting of the optical image, and can capture moving images and still images, When taking a still image during moving image shooting, the main control unit changes the exposure setting from moving image shooting to still image shooting and then takes the still image, and the signal processing unit takes the still image. Moving image frame data during still image shooting is created using still image data which is image data obtained by shooting.

  Thereby, it can be expected that an appropriate still image is obtained even during moving image shooting. In addition, since moving image frame data during still image shooting is created using the still image data, it can be expected that quality degradation of moving images due to still image shooting is suppressed.

  For example, the exposure setting controlled by the main control unit when shooting a still image during moving image shooting includes an aperture that specifies the exposure time of the image sensor and the amount of light incident on the image sensor per unit time. And / or the amplification degree of the amplifier circuit.

  Further, for example, the signal processing unit corrects subject blurring due to movement of the subject included in the still image data according to a difference between the exposure time for moving image shooting and the exposure time for still image shooting, The moving image frame data during still image shooting is created based on the corrected still image data.

  As a result, it is possible to apply a subject blur suitable for a moving image to the moving image frame data during still image shooting to obtain a smooth moving image.

  Specifically, for example, the signal processing unit is configured to detect the movement of the subject based on a difference between the still image data and image data obtained by photographing before and after photographing for obtaining the still image data. A subject blur region specifying unit for obtaining a certain subject blur region, and a subject blur fluctuation amount derived from a difference between the exposure time for moving image shooting and the exposure time for still image shooting in the still image data; The subject blur amount is corrected by adding to the data corresponding to the subject blur region.

  Further, for example, the exposure setting controlled by the main control unit when shooting the still image during moving image shooting includes at least the exposure time and the aperture opening amount, and the still image is captured during moving image shooting. When the time required for changing the aperture of the diaphragm for still image shooting is required for one or more frame periods of moving image shooting, the main control unit uses a plurality of frame periods to The image processing unit performs the still image shooting after changing the aperture amount for the still image shooting, and the signal processing unit detects each subject blur amount due to the movement of the subject included in each image data obtained during the plurality of frame periods. Is corrected according to the difference between the exposure time for moving image shooting and the exposure times used during the plurality of frame periods, thereby creating each frame data of the plurality of frame periods.

  With the above configuration, even when it takes more than one frame period for moving image shooting to change the aperture amount of the aperture for still image shooting, acquisition of appropriate still images and still image shooting are required. It can be expected to suppress the deterioration of the quality of the moving image due to the.

  Further, for example, when the still image during moving image shooting is performed over a period longer than one frame period of the moving image shooting, the signal processing unit performs shooting for obtaining the still image data and the still image data. The moving image frame data during still image shooting is created by weighted addition synthesis with image data obtained by shooting before or after shooting, and the weighting coefficient in the weighted addition is the image to be subjected to weighted addition synthesis It is determined by the data shooting timing and the still image data shooting timing.

  If moving image frame data during still image shooting is created by performing the above weighted addition synthesis, the amount of subject blur included in the movie represented by the moving image frame data approximates the amount of subject shake that should be, and motion It is possible to obtain a smooth video.

  At this time, for example, the signal processing unit performs weighted addition synthesis only for a portion where the difference between the image data to be subjected to weighted addition synthesis and the still image data is a predetermined threshold value or more. Also good.

In addition, for example, the exposure setting controlled by the main control unit when shooting the still image during moving image shooting includes the exposure time of the image sensor and the amplification degree of the amplification circuit. In the case of shooting the still image, when it is determined that exposure is insufficient even if the exposure time is set to a predetermined upper limit value by changing the exposure setting by the main control unit,
The main control unit captures the still image while compensating for insufficient exposure by increasing the amplification factor.

  In addition, for example, when shooting the still image during moving image shooting, exposure is insufficient even if the exposure time is set to a predetermined upper limit value by changing the exposure setting by the main control unit. When the determination is made, the main control unit sets the exposure time to be less than or equal to the upper limit value and takes a plurality of images, and the signal processing unit sets a plurality of image data obtained by shooting the plurality of images. Is used to create data representing the still image to be shot during moving image shooting.

  As a result, a good still image with less noise can be obtained.

  In addition, for example, the signal processing unit includes a brightness detection unit that detects the brightness of each captured moving image and still image based on the output signal of the amplification circuit, and captures the still image during moving image shooting. A brightness correction unit that corrects the still image data so that a difference between the brightness of the still image and the brightness of the moving image before or after the still image is reduced. The moving image frame data during still image shooting is created using the still image data.

  Thereby, it is possible to suppress the flickering of the brightness of the moving image that is caused by the variation in the brightness between the still image data and the moving image before or after the change in the exposure setting for still image shooting.

  At this time, for example, when the still image is shot during moving image shooting, the brightness correction unit is set to a single or plural set at the same position in the moving image before or after the still image and the still image. By correcting the still image data so that the average brightness of the area matches between the still image and the previous or subsequent movie, the brightness of the still image and the brightness of the previous or subsequent movie are corrected. To reduce the difference.

  Also, for example, the exposure setting controlled by the main control unit when shooting the still image during moving image shooting includes at least the exposure time and the aperture opening amount, and the signal processing unit A peripheral light amount correction unit that corrects a variation in decrease in the peripheral light amount associated with a change in the aperture amount of the diaphragm between the still image shooting and the still image capturing may be provided in a stage preceding the brightness correction unit.

  Thereby, it is possible to suppress flickering in the peripheral portion of the moving image, which may occur when still image shooting is performed during moving image shooting.

  As described above, according to the imaging apparatus of the present invention, even when still image shooting is performed during moving image shooting, it is possible to obtain an appropriate still image and to suppress degradation in moving image quality caused by still image shooting. It becomes possible.

<< First Embodiment >>
Hereinafter, a first embodiment of an imaging apparatus according to the present invention will be specifically described with reference to the drawings. FIG. 1 is an overall block diagram of an imaging apparatus 1 according to the first embodiment. The imaging device 1 is, for example, a digital still camera or a digital video camera.

  The imaging apparatus 1 includes an optical system 11, an aperture 12, an imaging device 13, a preprocessing unit 14, a driver 17, a timing generator 18, a main control unit 19, a signal processing unit 20, and a frame memory buffer 40. And an external memory 41 and an operation unit 42. The imaging device 1 is an imaging device capable of capturing both a moving image and a still image (simultaneously).

  The optical system 11 is composed of, for example, a plurality of lenses, and incident light from an imaging target is incident on the image sensor 13 via the lens and the diaphragm 12. The driver 17 controls the optical system 11 based on a control signal from the main control unit 19 and controls the zoom magnification and focal length of the optical system 11. The driver 17 controls the opening amount (size of the opening) of the diaphragm 12 based on a control signal from the main control unit 19. When the optical images incident on the optical system 11 are the same, the amount of incident light per unit time on the image sensor 13 increases as the aperture of the diaphragm 12 increases.

  The image sensor 13 is composed of, for example, a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The image sensor 13 photoelectrically converts an optical image incident through the optical system 11 and the diaphragm 12 and outputs an electrical signal obtained by the photoelectric conversion to the preprocessing unit 14. More specifically, the imaging device 13 includes a plurality of pixels (light receiving pixels; not shown) that are two-dimensionally arranged in a matrix, and in each shooting, each pixel receives a signal charge having a charge amount corresponding to the exposure time. store. Each pixel sequentially outputs an electrical signal having a magnitude proportional to the amount of stored signal charge to the pre-processing unit 14 at the subsequent stage. When the optical images incident on the optical system 11 are the same and the aperture amount of the diaphragm 12 is the same, the magnitude (intensity) of the electrical signal from the image sensor 13 increases in proportion to the exposure time.

  The image sensor 13 has an electronic shutter function, and the exposure time is controlled by an electronic shutter control signal from the timing generator 18. Since the main control unit 19 controls the operation of the timing generator 18, the main control unit 19 and the timing generator 18 function as an exposure time control unit that controls the exposure time. The shutter speed in the electronic shutter control is specified by the exposure time.

  The preprocessing unit 14 includes an amplifier circuit 15 and an A / D converter (analog-digital converter) 16. The amplifier circuit 15 amplifies an analog signal, which is an output signal of the image sensor 13, with a specified amplification level and outputs the amplified signal. The A / D converter 16 converts the output signal of the amplifier circuit 15 into a digital signal according to the signal from the timing generator 18. The preprocessing unit 14 performs further appropriate processing on the digital signal as necessary, and outputs image data representing the captured image. The image data output from the preprocessing unit 14 is given to the signal processing unit 20.

  The amplification degree in the amplifier circuit 15 is normally controlled according to so-called auto gain control. However, as a characteristic point of the present invention, the amplification degree is also controlled by the main control unit 19 via the timing generator 18. The control method will be described in detail later.

  The timing generator 18 controls the timing of each operation of the image sensor 13 and the preprocessing unit 14 in accordance with the shooting timing. Specifically, for example, the timing for reading the output signal from the image sensor 13 (the timing for giving the output signal from the image sensor 13 to the preprocessing unit 14) and the data output timing by the preprocessing unit 14 are controlled.

  The signal processing unit 20 includes a pixel number conversion unit 21, a motion detection unit 22, a moving image processing unit 23, a moving image compression unit 24, a still image processing unit 25, and a still image compression unit 26. Is done. The image data output from the preprocessing unit 14 is given to the pixel number conversion unit 21, the motion detection unit 22, and the still image processing unit 25. Hereinafter, the image data output by the preprocessing unit 14 is particularly referred to as original image data.

  The pixel number conversion unit 21 receives the original image data from the preprocessing unit 14 as input data. The pixel number conversion unit 21 reduces the image size of the original image data by performing cutout processing, filter processing, thinning processing, and the like on the original image data received as input data. Image data obtained by this reduction is given to the moving image processing unit 23. The moving image processing unit 23 performs necessary video processing on the image data from the pixel number conversion unit 21. This video processing includes, for example, generation of luminance signals and color difference signals, gamma correction, contour enhancement processing, and the like. The moving image processing unit 23 outputs the image data obtained through this video processing to the moving image compression unit 24. The moving image compression unit 24 compresses the image data from the moving image processing unit 23 using a predetermined compression method such as MPEG (Moving Picture Experts Group), and outputs the compressed image data to the external memory 41.

  The still image processing unit 25 receives the original image data from the preprocessing unit 14 as input data. The still image processing unit 25 performs necessary video processing on the original image data received as input data. This video processing includes, for example, generation of luminance signals and color difference signals, gamma correction, contour enhancement processing, and the like. The still image processing unit 25 outputs the image data obtained through this video processing to the still image compression unit 26. The still image compression unit 26 compresses the image data from the still image processing unit 25 using a predetermined compression method such as JPEG (Joint Photographic Experts Group), and outputs the compressed image data to the external memory 41. .

  The motion detection unit 22 receives the original image data from the preprocessing unit 14 as input data. The motion detection unit 22 detects a motion vector of the subject using a known representative point matching method or the like based on the original image data sent from the preprocessing unit 14 one after another during moving image shooting. During moving image shooting, the pixel number conversion unit 21 and the moving image processing unit 23 refer to the motion vector detected by the motion detection unit 22 and perform necessary processing (for example, so-called camera shake correction).

  The frame memory buffer 40 is composed of a semiconductor memory such as a DRAM (Dynamic Random Access Memory) and can read and write data. Each part constituting the signal processing unit 20 writes data to be temporarily recorded in the frame memory buffer 40 and reads data to be referred to from the frame memory buffer 40. The external memory 41 is a recording medium such as a semiconductor memory or a memory card.

  The main control unit 19 includes a CPU (Central Processing Unit) and the like, and comprehensively controls the operation of each part in the imaging apparatus 1. When the user operates the operation unit 42 constituted by input keys (not shown) or the like, a signal indicating the operation content is transmitted to the main control unit 19. The main control unit 19 controls the driver 17, the timing generator 18, the signal processing unit 20, and the like in accordance with the signal, so that still image shooting and recording and moving image shooting and recording are performed.

  When performing moving image shooting in response to an operation on the operation unit 42, the main control unit 19 controls the driver 17 and the timing generator 18 to set the exposure setting of the image sensor 13 (exposure setting of the imaging device 1) for moving image shooting. To do. The exposure setting for moving image shooting is an exposure setting suitable (preferably optimum) for shooting a moving image. During moving image shooting, original image data is output from the preprocessing unit 14 one after another at a predetermined frame rate (that is, at predetermined intervals). The original image data representing the captured moving image is sequentially sent to the external memory 41 through the predetermined processing of the pixel number conversion unit 21, the moving image processing unit 23, and the moving image compression unit 24, and is recorded in the external memory 41. The As a result, the moving image is shot and the shot moving image is recorded.

  When taking a still image in response to an operation on the operation unit 42, the main control unit 19 controls the driver 17 and the timing generator 18 to set the exposure setting of the image sensor 13 (exposure setting of the imaging device 1). Use it for. The exposure setting for still image shooting is an exposure setting suitable (preferably optimum) for shooting a still image. Original image data obtained by taking a still image is sent to the external memory 41 through predetermined processing of the still image processing unit 25 and the still image compression unit 26, and is recorded in the external memory 41. As a result, a still image is shot and a recorded still image is recorded.

  When shooting an instantaneous movement of a fast-moving subject as a still image, shooting is performed with a shorter exposure time (faster shutter speed). When the exposure time is sufficiently short relative to the movement of the subject, a clear image without so-called “subject blur” (moving subject blur) can be obtained as shown in an image 61 in FIG. On the other hand, if all frames are shot with a high-speed shutter in moving image shooting, each frame of the moving image frame becomes a clear image without movement, resulting in a moving image with unnatural and lack of smoothness. In other words, the continuity of the moving part cannot be expressed.

  Therefore, in moving image shooting, shooting is performed with the exposure time as long as possible in order to smoothly reproduce the movement of the subject. For example, when the frame rate of moving image shooting is 60 frames / second, shooting is performed with an exposure time as close as possible to 1/60 seconds. Then, as shown in an image 60 in FIG. 2, a so-called “subject blur” (moving subject blur) occurs in a portion where the motion is fast with respect to the exposure time, thereby obtaining a moving image with smooth motion.

  As described above, “subject blur” is a blur that occurs when the subject moves during an exposure period in which signal charges are accumulated in each pixel constituting the image sensor 13 (the length of the exposure period is equal to the exposure time). However, by actively incorporating this “subject blur” in moving image shooting, the motion of the subject can be reproduced smoothly.

  The imaging apparatus 1 realizes improvement of the quality of a captured image when a still image is captured during moving image shooting by adjusting the size (or presence / absence) of the “subject blur”. I will explain the relevant parts.

[First operation example]
First, a first operation example of the operation of the imaging device 1 will be described. In the first operation example, for example, it is assumed that still image shooting is performed during moving image shooting of a fast-moving scene such as an athletic meet. FIG. 3 is a diagram illustrating a control concept of the first operation example. When still image shooting is performed during moving image shooting, the aperture amount of the diaphragm 12, the charge accumulation amount of the image sensor 13, the exposure time, It is a figure which shows the relationship between the original image data read from the pre-processing part 14, and the image data which the moving image process part 23 outputs. The output data of the moving image processing unit 23 representing moving images is hereinafter referred to as “moving image frame data”. Also, in the following description, in order to avoid complication of sentences, an expression as if processing by the pixel number conversion unit 21, the moving image compression unit 24, the still image processing unit 25, and the still image compression unit 26 is ignored is used. There is.

  It is assumed that moving image shooting is performed in the order of frames FL1, FL2, FL3, FL4, FL5, FL6, and FL7. In addition, it is assumed that during moving image shooting, a still image is shot during a period corresponding to the frame FL4 in accordance with a still image shooting operation on the operation unit 42. For this reason, it can be considered that the moving image shooting frame FL4 does not actually exist, but for the sake of convenience, the moving image shooting frame FL4 is considered to exist. In this case, the exposure settings in the frames FL1 to FL3 and FL5 to FL7 are set for moving image shooting, and only the exposure settings in the frame FL4 are set for still image shooting.

  Specifically, in order to make the exposure time at the time of still image shooting suitable for still image shooting, the exposure time at the frame FL4 is made shorter than the exposure times at the frames FL1 to FL3 and FL5 to FL7, while the exposure is performed. In order to compensate for the underexposure due to the shortening of time, the opening amount of the diaphragm 12 in the frame FL4 is made larger than those in the frames FL1 to FL3 and FL5 to FL7. By adjusting the exposure time and the opening amount of the diaphragm 12, the charge accumulation amount in the frames FL1 to FL7 is made uniform to an appropriate level 63. That is, the brightness of the still image is (substantially) the same as that of the moving image. Here, the “charge accumulation amount” may be the average of the charge accumulation amounts of all the pixels constituting the image sensor 13 or the average of the charge accumulation amounts of the pixels located at the center of the imaging surface of the image sensor 13. . Further, the average charge accumulation amount of pixels located in a plurality of regions on the imaging surface of the image sensor 13 may be used. These averages may be used as weighted averages.

  Further, instead of adjusting the aperture amount of the diaphragm 12, the amplification degree of the amplifier circuit 15 may be adjusted, and thereby the charge accumulation amount in the frames FL1 to FL7 may be equalized to the appropriate level 63. In this case, the amplification degree of the amplifier circuit 15 in the frame FL4 is made larger than those in the frames FL1 to FL3 and FL5 to FL7.

  Further, the exposure time, the opening amount of the diaphragm 12 and the amplification degree of the amplifier circuit 15 may all be adjusted, and thereby the charge accumulation amount in the frames FL1 to FL7 may be made uniform to the appropriate level 63. For example, in order to make the exposure time at the time of still image shooting suitable for still image shooting, the exposure time at the frame FL4 is made shorter than the exposure times at the frames FL1 to FL3 and FL5 to FL7, while the exposure time is shortened. In order to compensate for the underexposure due to the above, the opening amount of the diaphragm 12 in the frame FL4 and the amplification degree of the amplifier circuit 15 are made larger than those in the frames FL1 to FL3 and FL5 to FL7.

  As described above, the exposure setting adjusted by the main control unit 19 when shooting a still image during moving image shooting includes at least one of the exposure time, the aperture amount of the diaphragm 12 and the amplification degree of the amplifier circuit 15. And are included.

  All the original image data obtained by the above photographing are sent to the signal processing unit 20. With respect to the frames FL1 to FL3 and FL5 to FL7, each moving image frame data based on each original image data is obtained by the image processing unit 23, and the moving image frame data undergoes a predetermined process and is represented as data representing a moving image in the external memory 41. Stored in The original image data corresponding to the still image captured by the frame FL4 is treated as “still image data”, and is processed by the still image processing unit 25 and the still image compression unit 26 as data representing a still image from the external memory. 41. The moving image frame data corresponding to the frame FL4 during still image shooting is created using the still image data, and stored in the external memory 41 as data representing the moving image. For this reason, even if still image shooting is performed during moving image shooting, the moving image does not stop for a moment or drop frames.

  When moving image frame data is generated using still image data, if the movement of the subject is slow or the difference between the exposure time for moving image shooting and the exposure time for still image shooting is small, the still image data Even if it is adopted as moving image frame data as it is, since the difference in subject blur between the still image data and the preceding and subsequent moving image frames is small, there is no sense of unnatural motion in the moving image. However, when the difference in the amount of subject blur is large, if the still image data is used as it is as moving image frame data, the image looks clear only by that frame, and flickering or unnaturalness may occur.

  In order to suppress such unnaturalness, the moving image processing unit 23 includes the still image data (included in the still image captured in the frame FL4) when generating the moving image frame data using the still image data. The amount of subject blur due to subject movement is corrected according to the difference in exposure time, thereby creating moving image frame data during still image shooting.

  A method of creating moving image frame data during still image shooting using still image data by the moving image processing unit 23 will be described with reference to FIGS. FIG. 4 is an internal configuration diagram of the moving image processing unit 23 showing only a part related to creation of moving image frame data from still image data. In the description of the specific examples using FIGS. 4 to 10, the presence of the pixel number conversion unit 21 is ignored for the sake of simplification. That is, it is assumed that the original image data from the preprocessing unit 14 is directly given to the moving image processing unit 23.

  Assume that the frame rate is 60 frames / second, the exposure time for moving image shooting in frames FL2, FL3, and FL5 is T, and the exposure time for still image shooting in frame FL4 is t, as shown in FIG. The exposure times T and t are 1/60 seconds and 1/180 seconds, respectively, and the intermediate time point of the exposure period of the frame FL4 coincides with the intermediate time point of the frame period of the frame FL4. Then, as shown in FIG. 6, a situation is assumed in which a bright spot P having a size of one pixel in an image whose background is all black is moving at a constant speed in the right direction while maintaining a constant luminance. It is assumed that the bright spot P moves rightward by 3 pixels while the exposure time T elapses. Then, consider a pixel group that continues in the right direction through which the bright spot P passes between the frames FL2 to FL5. This pixel group is composed of a total of 12 pixels (n + k) as shown in FIG. 7 (where n is a natural number and k is an integer from 0 to 11), and the pixel having a larger number of k is arranged on the right side. It shall be.

  Under the above assumption, when the original image data in the frames FL2, FL3, FL4, and FL5 is expressed as an image, they are as images 70, 71, 72, and 73 in FIG. 8, respectively.

  In the image 70, as the bright spot P moves in the right direction, only the pixel data corresponding to the pixels n to (n + 2) is “10”, and the pixel data corresponding to the other pixels is all “0”. ing. Similarly, in the image 71, only the pixel data corresponding to the pixels (n + 3) to the pixel (n + 5) is “10”, and the pixel data corresponding to the other pixels are all “0”. Similarly, in the image 73, only the pixel data corresponding to the pixel (n + 9) to the pixel (n + 11) is “10”, and the pixel data corresponding to the other pixels is all “0”. Thus, in the images 70, 71, and 73 corresponding to the frames FL2, FL3, and FL5, it appears as if the bright spot P having the size of one pixel is blurred between the three pixels. That is, subject blurring occurs.

  Each pixel data is assumed to be data having a value proportional to the amount of charge accumulated during the exposure time (that is, data proportional to luminance). Accordingly, the pixel data for the pixels of the portion that does not receive incident light corresponding to the bright spot P are all “0”. “Image data” refers to pixel data for all pixels collectively.

  In the image 72 corresponding to the frame FL4, the exposure time is 1/3 of the other frames and the aperture amount of the diaphragm 12 (and / or the amplification degree of the amplifier circuit 15) is large. Only the pixel data corresponding to the pixel (n + 7) is “30”, and the pixel data corresponding to the other pixels are all “0”. Thus, no subject blur occurs in the image 72 representing the original image data in the frame FL4 as an image.

  The moving image processing unit 23 performs correction processing on the image data (that is, still image data) obtained by shooting the still image of the frame FL4 so that the subject blur difference between the moving image and the still image is reduced. Movie frame data corresponding to the frame FL4 is created. The moving image processing unit 23 (see FIG. 4) includes a difference calculation unit 31 that calculates and outputs a difference between the image data obtained by the current shooting and the image data obtained by the shooting immediately before and after the shooting, Based on the output data of the difference calculation unit 31, the subject blur region specifying unit 32 for specifying the region where the subject blur occurs or the region where the subject blur should occur, and the subject blur while based on the information on the exposure time and the information on the motion vector. A blur processing unit 33 that performs blur processing on the region (subject blur region) specified by the region specifying unit 32 is included.

  The operation of the moving image processing unit 23 will be described along the above specific example. Again, for simplicity of explanation, the presence of the pixel number conversion unit 21 is ignored. That is, the description will be made assuming that the original image data from the preprocessing unit 14 is directly given to the moving image processing unit 23. The difference calculation unit 31 calculates the difference between the original image data (that is, still image data) obtained by shooting the frame FL4 and the original image data obtained by shooting the frames FL3 and FL5, and the difference value Is output. Specifically, the original image data obtained by photographing the frames FL3 and FL5 is subtracted from the original image data obtained by photographing the frame FL4. Of course, this difference calculation is performed between pixel data corresponding to the same pixel. Thereby, the difference values corresponding to the pixels n to (n + 11) are respectively 0, 0, 0, −10, −10, −10, 0, 30, 0, −10, −10, as shown in FIG. -10.

  The subject blur area specifying unit 32 compares the magnitude of the difference value corresponding to each pixel with a predetermined threshold value, and determines the subject blur area (that is, the area where the subject blur occurs or the area where the subject blur should occur). Identify. For example, when the threshold is set to “5”, pixels (n + 3) to (n + 5), (n + 7), and (n + 9) to (n + 11) having a difference value of 5 or more are set as subject blur areas.

  The blur processing unit 33 blurs the original image data (that is, still image data) obtained by shooting the frame FL4 according to the difference between the exposure time T for moving image shooting and the exposure time t for still image shooting. Apply processing. At this time, a motion vector near the shooting timing of the still image of the frame FL4 is referred to. For example, the average motion vector of the motion vector between the frames FL2 to FL3 and the motion vector between the frames FL5 to FL6 is referred to. Since the direction and magnitude of the motion vector between the frames FL2 and FL3 and the motion vector between the frames FL5 and FL6 are both “rightward” and “3 pixels”, the direction and magnitude of the average motion vector Are “rightward” and “3 pixels”.

  In this case, if the exposure time is T, the movement in which the orientation and size are “rightward” and “3 pixels” is caused by setting the exposure time to t for still image shooting. In the still image data, it is regarded as a movement of (3 pixels) × t / T, that is, a movement of one pixel in the right direction. That is, the subject blur of 2 pixels (because 3-1 = 2) does not appear in the still image data. It can be said that the amount of subject blur included in the still image data (the amount of subject blur included in the image represented by the still image data) is zero.

  In consideration of this, the blurring processing unit 33 performs blurring processing for intentionally adding subject blurring for two pixels to still image data. That is, the pixel data is averaged using the peripheral pixels of the pixel specified as the subject blur region. The “peripheral pixels” used for the averaging are set according to the direction and size of the average motion vector. In other words, in this case, the pixel data in each pixel specified as the subject blur region is averaged between the pixel data of the two adjacent pixels on the left and right.

  Thereby, only the pixel data corresponding to the pixel (n + 7) is “30” and the pixel data corresponding to the other pixels are all “0”. As shown in FIG. 4, after the blurring process, only the pixel data corresponding to the pixels (n + 6) to (n + 8) is “10”, and the pixel data corresponding to the other pixels are all “0”. The amount of subject blur included in the still image data after the blurring process (the amount of subject blur included in the image represented by the still image data after the blurring process) is larger than that before the blurring process. For example, it can be expressed as “2”.

  As described above, the blurring processing unit 33 intentionally gives the data corresponding to the subject blur region in the still image data the reduced amount of subject blur caused by shortening the exposure time from T to t. That is, the data corresponding to the subject blur area is corrected so that the image corresponding to the subject blur area is blurred according to the difference between the exposure times T and t. Then, the image data after this correction is output as moving image frame data during still image shooting corresponding to the frame FL4. Thereby, the moving image does not flicker for a moment at the timing of still image shooting, and a smooth moving image can be generated.

  In the averaging process in the blurring process, a weighting coefficient may be appropriately added. In addition, although data proportional to the luminance is exemplified as the pixel data to be subjected to the difference calculation, the color difference signal value corresponding to each pixel and the R (red), G (green) or B (blue) signal corresponding to each pixel A value or the like may be adopted as pixel data. Further, the above blur processing by the blur processing unit 33 is an example, and any method is adopted as long as the image corresponding to the subject blur region is blurred according to the difference between the exposure times T and t. It doesn't matter.

  In addition, in the shooting of moving images and still images, if the timing just in the middle of the exposure period is called the shooting timing (reference timing) of the shooting, in the above first operation example, as shown in FIG. The interval between suitable shooting timings is always constant. In FIG. 11 and FIG. 12, black triangle marks indicate shooting timings of moving images and still images. Thereby, the average time interval (shooting timing interval) between the frame images is constant, and a smoother moving image can be obtained. Such control can be applied when an image sensor capable of high-speed reading that is not required to make the exposure period end timing constant in relation to the frame period is used. When a general CCD or the like is used as the image sensor 13, the exposure period end timing in relation to the frame period is made constant in each frame due to signal read speed and control restrictions. For example, as shown in FIG. 12, in each frame, control is performed so that the end timing of the frame period and the end timing of the exposure period always coincide. In this case, the interval between adjacent shooting timings can vary.

[Second operation example]
Next, a second operation example of the operation of the imaging device 1 will be described. Similar to the first operation example, the second operation example assumes that still image shooting is performed during moving image shooting of a fast-moving scene such as an athletic meet. However, it is assumed that the time required to change the aperture of the aperture 12 from moving image shooting to still image shooting requires a plurality of frame periods for moving image shooting.

  When taking a still image at a shutter speed faster than the shutter speed during movie shooting, it is also possible to keep the brightness of the image constant by increasing the amplification degree of the amplification circuit 15 without changing the aperture amount of the aperture 12. It is. However, when the brightness is secured by increasing the amplification degree, the noise also increases at the same time. For this reason, when still image shooting is performed at a shutter speed faster than the shutter speed during moving image shooting, it is desirable to widen the aperture 12 in accordance with the change in shutter speed in order to keep the brightness of the image constant. However, when the frame rate for moving image shooting is relatively high, it may take more than one frame period of the moving image to change the aperture of the aperture 12 from moving image shooting to still image shooting. The second operation example assumes such a situation.

  FIG. 13 is a diagram illustrating a control concept of the second operation example, and in the case where still image shooting is performed during moving image shooting, the aperture amount of the diaphragm 12, the charge accumulation amount of the image sensor 13, the exposure time, It is a figure which shows the relationship between the original image data read from the pre-processing part 14, and the image data (moving image frame data) which the moving image process part 23 outputs.

  Movie shooting is performed in the order of frames FL11, FL12, FL13, FL14, FL15, FL16, and FL17. Then, it is assumed that during moving image shooting, a still image is shot during a period corresponding to the frame FL14 in accordance with a still image shooting operation on the operation unit 42. For this reason, although it can be considered that the moving image shooting frame FL14 does not actually exist, for the sake of convenience, it is considered that the moving image shooting frame FL14 exists. In the second operation example, it is assumed that a time corresponding to three frame periods of moving image shooting is required because the opening amount of the diaphragm 12 changes from moving image shooting to still image shooting. Similarly, it is assumed that a time corresponding to three frame periods of moving image shooting is required because the opening amount of the diaphragm 12 changes from still image shooting to moving image shooting.

  In this case, the exposure settings in the frames FL11 and FL17 are set for moving image shooting, and only the exposure settings in the frame FL14 are set for still image shooting. The exposure settings in the frames FL12, FL13, FL15, and FL16 are set between moving image shooting and still image shooting.

  Specifically, the opening amount of the diaphragm 12 is gradually increased in the order of the frames FL11, FL12, FL13, and FL14, and the opening amount of the diaphragm 12 is gradually decreased in the order of the frames FL14, FL15, FL16, and FL17. On the other hand, the exposure time is gradually shortened in the order of the frames FL11, FL12, FL13, and FL14 so as to cancel the increase in the exposure amount due to the increase in the aperture amount of the aperture stop 12, and the exposure amount is decreased due to the decrease in the aperture amount of the aperture stop 12. The exposure time is gradually increased in the order of the frames FL14, FL15, FL16, and FL17 so as to cancel out. By adjusting the exposure time and the opening amount of the diaphragm 12, the charge accumulation amount in the frames FL11 to FL17 is made uniform to an appropriate level 63. That is, the brightness of the images obtained by photographing the frames FL11 to FL17 are all (substantially) the same.

  Although it is ideal that the brightness of the image is all the same by the control as described above, the brightness is completely constant due to the restriction of the control accuracy of the aperture amount of the diaphragm 12 that operates mechanically. Difficult to keep in. Therefore, in the subsequent signal processing, amplification processing or the like may be added so that the brightness of the signal finally becomes constant. Specific examples of this processing will be described later in the second and third embodiments.

  Actual still image shooting is performed at the timing when the opening amount of the diaphragm 12 has completely changed, that is, in a period corresponding to the frame FL14. After still image shooting, the aperture of the diaphragm 12 is returned to movie shooting.

  All the original image data obtained by the above photographing are sent to the signal processing unit 20. For the frames FL11 and FL17, each moving image frame data based on each original image data is obtained by the image processing unit 23, and the moving image frame data is stored in the external memory 41 as data representing a moving image through a predetermined process. . The original image data corresponding to the still image captured by the frame FL14 is treated as “still image data”, and is processed by the still image processing unit 25 and the still image compression unit 26 as data representing a still image from an external memory. 41.

  The moving image frame data corresponding to the frame FL14 during still image shooting is created using the still image data and stored in the external memory 41 as data representing the moving image. When the moving image frame data corresponding to the frame FL14 is generated using the still image data, the subject blur amount due to the movement of the subject included in the still image data (the still image captured in the frame FL14) is determined as the exposure time. Thus, the moving image frame data during still image shooting is created. The method of creating moving image frame data during still image shooting using still image data by the moving image processing unit 23 is the same as that described in the first operation example.

  Further, since the exposure time in the frames FL12, FL13, FL15 and FL16 is shorter than the exposure time for moving image shooting, when the original image data obtained by shooting the frames FL12, FL13, FL15 and FL16 is represented as an image, The amount of subject blur included in these images is smaller than when the exposure time is set for moving image shooting. Therefore, the subject shake amount is also corrected for each original image data obtained by photographing the frames FL12, FL13, FL15, and FL16, and each image data obtained through the correction is used for the frames FL12, FL13, FL15, and FL16. Each moving image frame data is used. Each moving image frame data of the frames FL12, FL13, FL15 and FL16 is stored in the external memory 41 as data representing moving images corresponding to these frames. Thereby, the moving image corresponding to those frames does not flicker, and a smooth moving image can be obtained.

  The subject blur amount correction method corresponding to the frames FL12, FL13, FL15, and FL16 is basically the same as the subject blur amount correction method corresponding to the frame FL14, and the internal configuration of the moving image processing unit 23 that performs the correction process. Is the same as that shown in FIG. With reference to FIG. 4, focusing on the frame FL12 as an example, this subject blur amount correction method will be described. However, for simplification of description, the presence of the pixel number conversion unit 21 is ignored. That is, the description will be made assuming that the original image data from the preprocessing unit 14 is directly given to the moving image processing unit 23.

  The difference calculation unit 31 calculates a difference between the original image data obtained by photographing the frame FL12 and the original image data obtained by photographing the frames FL11 and FL13, and outputs the difference value. Specifically, the original image data obtained by photographing the frames FL11 and FL13 is subtracted from the original image data obtained by photographing the frame FL12. The subject blur area specifying unit 32 compares the magnitude of the difference value corresponding to each pixel with a predetermined threshold value, and specifies the subject blur area.

  The blur processing unit 33 blurs the original image data obtained by shooting the frame FL12 according to the difference between the exposure time for moving image shooting and the exposure time in the frame of interest (ie, the frame FL12). . At this time, as described in the first operation example, a motion vector near the shooting timing of the moving image of the frame FL12 is referred to.

  For example, when the exposure time for moving image shooting and the exposure time in the frame FL12 are 1/60 seconds and 1/120 seconds, respectively, 1/120 seconds (because 1 / 60-1 / 120 = 1/120) Subject blur for the exposure time is intentionally added (or added) to the original image data obtained by photographing the frame FL12. That is, the blurring process is performed so that the subject blur amount estimated to have occurred if the exposure time in the frame FL12 was 1/60 seconds is included in the moving image frame data corresponding to the frame FL12. The data subjected to the blurring process is output as moving image frame data corresponding to the frame FL12, and is stored in the external memory 41 through a predetermined process.

  Although the subject blur amount correction method has been described focusing on the frame FL12, the subject blur amount is corrected by the same method for the frames FL13, FL15, and FL16.

[Third operation example]
Next, a third operation example of the operation of the imaging device 1 will be described. In the third operation example, it is assumed that still image shooting is performed during moving image shooting of a dark scene such as a room.

  In moving image shooting of dark scenes, the exposure amount may not reach an appropriate level even if exposure is performed using the entire frame period of moving image shooting. In this case, generally, a bright image is generated by increasing the degree of amplification in signal processing. However, such an increase in amplification increases noise as well. Focusing on this, in the third operation example, when still image shooting is performed during moving image shooting of a dark scene, in order to suppress an increase in noise, the above-described amplification degree is set by making the exposure time longer than one frame period of the moving image. Suppresses excessive increase of

  When a still image is shot over a period longer than one frame during movie shooting, it is necessary to compensate for the lack of movie frames corresponding to the exposure time of the still image. In order to make up for the lack of a moving image frame, if the immediately preceding moving image frame is repeatedly displayed or a black frame is inserted, the moving image becomes unsightly. In consideration of this, in the third operation example, a moving image that is missing due to still image shooting using image data obtained by shooting a still image and image data obtained by shooting before and after the still image. Generate a frame. Hereinafter, this method will be described in detail.

  FIG. 14 is a diagram illustrating a control concept of the third operation example, and when a still image is shot during moving image shooting, the aperture amount of the diaphragm 12, the charge accumulation amount of the image sensor 13, the exposure time, FIG. 4 is a diagram illustrating a relationship among a gain increase amount in an amplifier circuit 15, original image data read from a preprocessing unit 14, and image data (moving image frame data) output by a moving image processing unit 23.

  If there is no still image shooting on the way, moving image shooting is performed in the order of frames FL21, FL22, FL23, FL24, FL25, FL26, FL27. Then, it is assumed that during moving image shooting, a still image is shot during a period corresponding to the frames FL23 and FL24 in accordance with a still image shooting operation on the operation unit 42. For this reason, although it can be considered that the frames FL23 and FL24 for moving image shooting do not actually exist, it is considered that the frames FL23 and FL24 for moving image shooting exist for convenience. In FIG. 14, t1 represents the exposure time for moving image shooting of the frames FL21, FL22, FL25, FL26, and FL27, and t1 × 2 (that is, twice t1) used the frames FL23 and FL24. It represents the exposure time for still image shooting.

  In the example shown in FIG. 14, the exposure amount does not reach the appropriate level 63 even when the aperture of the diaphragm 12 is maximized and exposure is performed for one frame period in moving image shooting. For this reason, in moving image shooting, the gain is doubled in signal processing. That is, in the moving image shooting of the frames FL21, FL22, FL25, FL26, and FL27, the amplification degree of the amplifier circuit 15 is set to twice the reference gain. On the other hand, a still image is taken with an exposure time of approximately two frame periods of FL23 and FL24, and the exposure amount reaches an appropriate level 63 without gain increase in signal processing. That is, in still image shooting using a period corresponding to the frames FL23 and FL24, the amplification degree of the amplifier circuit 15 is set to be one time the reference gain. For this reason, it is possible to obtain a good still image with less noise. Note that the opening amount of the diaphragm 12 in the frames FL21 to FL27 is maintained, for example, constant (maximum opening amount).

  All the original image data obtained by the above photographing are sent to the signal processing unit 20. For the frames FL21, FL22, FL25, FL26, and FL27, each moving image frame data based on each original image data is obtained by the image processing unit 23, and these moving image frame data are externally processed as data representing a moving image through predetermined processing. Stored in the memory 41. Original image data corresponding to a still image shot using a period corresponding to the frames FL23 and FL24 is treated as “still image data”, and is processed through the processing by the still image processing unit 25 and the still image compression unit 26. The data representing the image is stored in the external memory 41.

  When a still image is shot using a period corresponding to the frames FL23 and FL24, a moving image frame (moving image frame data) that should correspond to the frame FL23 is lost. On the other hand, the still image data includes information on a missing moving image frame. Therefore, when the frame rate of the moving image is sufficiently large relative to the movement of the subject, even if the image represented by the still image data is displayed as a moving image for two consecutive frames, the movement is recognized to stop for a moment. And the smoothness of the video is not compromised. However, in the case where the frame rate of the moving image cannot be said to be sufficiently large with respect to the movement of the subject, if the image represented by the still image data is displayed continuously for two frames, the movement appears to stop for a moment. .

  Therefore, in the third operation example, in order to obtain a smooth moving image regardless of the frame rate of the moving image, original image data (that is, a still image) representing a still image shot using a period corresponding to the frames FL23 and FL24. Data) and two original image data representing an image shot using the frames FL22 and FL25, to generate moving image frame data during still image shooting. In order to obtain a video that achieves smooth motion reproduction, subject blur that is close to the subject blur that should originally be included in the video is added to the generated video frame data, and the same image is not continuously displayed. That's fine.

  An example of a method of creating moving image frame data corresponding to the frames FL23 and FL24 based on the main point will be described with reference to FIG. However, for simplification of description, the presence of the pixel number conversion unit 21 is ignored. That is, the description will be made assuming that the original image data from the preprocessing unit 14 is directly given to the moving image processing unit 23. Further, in the description using FIG. 15, although slightly different from the illustration of FIG. 14, the length of the exposure time in the frames FL22 and FL25 is assumed to be equal to the length of one frame period of the moving image shooting, and It is assumed that the exposure time for still image shooting using the frames FL23 and FL24 coincides with the length of two frame periods for moving image shooting.

In each shooting of moving images and still images, the timing just in the middle of the exposure period is referred to as shooting timing (reference timing) of the shooting. The moving image shooting timings in the frames FL22 and FL25 are α ₂₂ and α ₂₅ , respectively, and the still image shooting timings using the frames FL23 and FL24 are β ₁ . If the still image is not shot, the shooting timings of the moving images shot in the frames FL23 and FL24 are α ₂₃ and α ₂₄ , respectively.

The moving image frame data corresponding to the frame FL23 includes original image data representing a still image captured using the frames FL23 and FL24 (that is, the still image data), and original image data obtained by capturing the frame FL22. Is created by weighted addition synthesis. The weighting coefficient in this weighted addition synthesis is determined based on the imaging timings α ₂₂ , α ₂₃ and β ₁ . That is, 1/3 of each pixel data constituting the original image data corresponding to the frame FL22 is obtained by multiplying each of the pixel data constituting the still image data photographed using the frames FL23 and FL24 by 2/3. The product of 3 is added, and the image data obtained by this addition is used as moving image frame data corresponding to the frame FL23. Of course, the addition of the pixel data is performed between the pixel data corresponding to the same pixel.

“2/3” as the weighting coefficient is calculated by dividing “the length of time between the shooting timings α ₂₂ and α ₂₃ ” by “the length of time between the shooting timings α ₂₂ and β ₁ ”. The “1/3” as the weighting factor is calculated by dividing “the length of time between the shooting timings α ₂₃ and β ₁ ” by “the length of time between the shooting timings α ₂₂ and β ₁ ”. The Since more image information in the frame FL23 is included on the still image data side, a smoother moving image can be reproduced by determining the weighting coefficient as described above.

  For example, when pixel data of a certain pixel n corresponding to the still image data is “60” and pixel data of the pixel n corresponding to the frame FL22 is “30”, moving image frame data corresponding to the frame FL23 The pixel data for the pixel n that constitutes “50” is “50” because 60 × 2/3 + 30 × 1/3 = 50.

Similarly, the moving image frame data corresponding to the frame FL24 includes original image data representing a still image captured using the frames FL23 and FL24 (that is, the still image data) and an original obtained by capturing the frame FL25. It is created by weighted addition synthesis with image data. The weighting coefficient in this weighted addition synthesis is determined based on the photographing timings β ₁ , α ₂₄ and α ₂₅ . That is, 1/3 of each of the pixel data constituting the original image data corresponding to the frame FL25 is obtained by multiplying each of the pixel data constituting the still image data photographed using the frames FL23 and FL24 by 2/3. The product of 3 is added, and the image data obtained by this addition is used as moving image frame data corresponding to the frame FL24. Of course, the addition of the pixel data is performed between the pixel data corresponding to the same pixel.

“2/3” as the above weighting coefficient is calculated by dividing “the length of time between imaging timings α ₂₄ and α ₂₅ ” by “the length of time between imaging timings β ₁ and α ₂₅ ”. The “1/3” as the above weighting coefficient is calculated by dividing “the length of time between shooting timings β ₁ and α ₂₄ ” by “the length of time between shooting timings β ₁ and α ₂₅ ”. The Since more image information in the frame FL24 is included on the still image data side, a smoother moving image can be reproduced by determining the weighting coefficient as described above.

  If moving image frame data during still image shooting is created by performing the above-described weighted addition synthesis, the amount of subject blur included in the moving image represented by the moving image frame data approximates the amount of subject blur that should originally exist. That is, moving image frame data in which the subject blur amount is adjusted based on still image data is created, and a moving image with smooth motion can be obtained.

Further, as shown in FIG. 16, the still image shooting timing when the exposure time of still image shooting is required longer than one frame period is controlled so as to be positioned exactly between the shooting timings of the preceding and subsequent moving image frames. It is desirable that This is because the average time interval (shooting timing interval) between the frame images is constant, and a smoother moving image can be obtained. The example shown in FIG. 15 also follows this. In other words, the still image shooting timing β ₁ is controlled so as to be positioned exactly between the shooting timings α ₂₂ and α ₂₅ of the preceding and subsequent moving image frames. In FIG. 16 and FIG. 17, black triangle marks represent shooting timings of moving images and still images, and white triangle marks represent shooting timings of moving image frames generated using still image data. When data is read from the image sensor 13 at the end of the frame period due to restrictions such as the signal readout speed, the shooting timing of the still image is set to exactly the middle between the shooting timings of the preceding and subsequent video frames as shown in FIG. It may not be possible to position.

  Further, the above weighted addition synthesis may be performed on all of the pixel data constituting the image data (that is, may be performed on the entire screen), or may be performed on only a portion having a large difference.

  An operation in the case of performing weighted addition synthesis only for a portion with a large difference will be described under the above-described conditions shown with reference to FIG. For example, as shown in FIG. 18, the pixel data corresponding to the pixels n, (n + 1) and (n + 2) constituting the original image data obtained by photographing the frame FL22 are “10”, “10” and “10”, respectively. 10 ”and the pixel data corresponding to the pixels n, (n + 1) and (n + 2) constituting the original image data (that is, the still image data) obtained by photographing the frames FL23 and FL24 are Suppose that they are “7”, “15” and “40”, respectively. It is assumed that a threshold value of “8” is set in advance.

  The moving image processing unit 23 compares the difference between pixel data for the same pixel with the threshold value, and performs the above-described weighted addition synthesis only for pixel data that gives a difference equal to or greater than the threshold value. In the case of the above example, only the pixel data corresponding to the pixel (n + 2) among the pixel data corresponding to the pixels n, (n + 1) and (n + 2) is subjected to weighted addition synthesis. Then, in the moving image frame data corresponding to the frame FL23, the pixel data corresponding to the pixel (n + 2) becomes “30” by the weighted addition synthesis.

For example, pixel data of still image data is used as it is as pixel data of pixels not subject to weighted addition synthesis. This is because the shooting timing α ₂₃ corresponding to the created moving image frame data is closer to the shooting timing β ₁ than the shooting timing α ₂₂ . However, the pixel data of the original image data corresponding to the frame FL22 may be used as it is as the pixel data of the pixels that are not the object of the weighted addition synthesis.

  In the above example, the exposure time for still image shooting is 2 frame periods. However, when the exposure time is longer than 2 frame periods, the moving image frame data during still image shooting is also used using the same concept. Can be created. However, if the exposure time for still image shooting during moving image shooting becomes longer, the moving image frame data generated by the synthesis increases, so that the reproduced moving image may be unnatural.

  Therefore, an upper limit value (for example, 3 frame periods) may be provided for the exposure time of still image shooting performed during moving image shooting. When it is determined that the exposure is insufficient even when the exposure time of still image shooting is extended to the upper limit (that is, when the exposure amount does not reach the predetermined appropriate level), the gain is increased in the signal processing. Just give it. That is, a still image may be taken after compensating for the underexposure by increasing the amplification degree of the amplifier circuit 15 above the reference gain.

  Whether the exposure is insufficient or not can be determined from the magnitude of the output signal of the amplifier circuit 15. In a state where the amplification degree of the amplifier circuit 15 is set to a predetermined value, if the magnitude of the output signal of the amplifier circuit 15 reaches a predetermined specified value, there is no shortage of exposure, and the magnitude reaches the specified value. If not, exposure is insufficient.

  Also, when an upper limit is set for the exposure time of still image shooting performed during movie shooting, a plurality of still image generation images are continuously shot instead of compensating for insufficient exposure by increasing the gain. May be added and combined (multiple exposure combining) to obtain a still image with optimum exposure. This technique will be described later as a fourth operation example.

[Fourth operation example]
FIG. 19 is a diagram illustrating a control concept of the fourth operation example. When still image shooting is performed during moving image shooting, the aperture amount of the diaphragm 12, the charge accumulation amount of the image sensor 13, the exposure time, FIG. 4 is a diagram illustrating a relationship among a gain increase amount in an amplifier circuit 15, original image data read from a preprocessing unit 14, and image data (moving image frame data) output by a moving image processing unit 23. In the fourth operation example, the presence of the pixel number conversion unit 21 is ignored for the sake of simplicity. That is, the description will be made assuming that the original image data from the preprocessing unit 14 is directly given to the moving image processing unit 23.

  If there is no still image shooting on the way, moving image shooting is performed in the order of frames FL31, FL32, FL33, FL34, FL35, FL36, and FL37. Then, it is assumed that during moving image shooting, a still image is shot using a period corresponding to the frames FL33 to FL36 in accordance with a still image shooting operation on the operation unit 42. For this reason, it can be considered that the moving image shooting frames FL33 to FL36 do not actually exist, but for the sake of convenience, the moving image shooting frames FL33 to FL36 are considered to exist. In FIG. 19, t1 represents the exposure time of moving image shooting of the frames FL31, FL32, and FL37, and t1 × 2 (that is, twice t1) represents the first stationary state using the frames FL33 and FL34. The exposure time for image shooting and the exposure time for second still image shooting using the frames FL35 and FL36 are shown.

  In the fourth operation example, an upper limit value (for example, two frame periods) is provided for the exposure time of still image shooting performed during moving image shooting, and the exposure time for still image shooting is extended to the upper limit value. Also, it is assumed that the exposure is determined to be insufficient (that is, the exposure amount does not reach the predetermined appropriate level 63). In moving image shooting, since the exposure amount does not reach the appropriate level 63 even when the aperture of the aperture 12 is maximized and exposure is performed for one frame period, the gain is increased four times in signal processing. That is, in moving image shooting of the frames FL31, FL32, and FL37, the amplification degree of the amplification circuit 15 is set to four times the reference gain.

  The shooting of one still image during moving image shooting is performed using a total of four frames FL33 to FL36. In this example, the exposure amount does not reach the appropriate level 63 even if the exposure time is extended to the above upper limit value. For this reason, first, the first still image shooting (first combining shooting) is performed using the frames FL34 and FL34, and then the second still image shooting (second shooting is performed using the frames FL35 and FL36. Shooting for compositing).

  Although the first and second still image shootings are performed with exposure times corresponding to approximately two frame periods, each exposure amount is almost half of the appropriate level 63. Further, in the first and second still image shootings, the gain is not increased in the signal processing (that is, the amplification degree of the amplifier circuit 15 is set to one time the reference gain). Therefore, if one still image is generated using the image data obtained from the first and second still image shootings, a good still image with less noise can be obtained. Note that the opening amount of the diaphragm 12 in the frames FL31 to FL37 is maintained constant (maximum opening amount).

  All the original image data obtained by the above photographing are sent to the signal processing unit 20. For the frames FL31, FL32, and FL37, each moving image frame data based on each original image data is obtained by the image processing unit 23, and the moving image frame data is stored in the external memory 41 as data representing a moving image through a predetermined process. Is done.

  The still image processing unit 25 includes the first still image data (first original image data) obtained by the first still image photographing and the second still image obtained by the second still image photographing. The image data representing one still image is generated by adding and synthesizing the data (original image data) (multiple exposure composition). The image data obtained by the addition synthesis is stored in the external memory 41 as data representing a still image through a predetermined process. Of course, the addition synthesis is performed between pixel data corresponding to the same pixel.

  The moving image frame data corresponding to the frames FL33, FL34, FL35, and FL36 is created by the same method as that in the third operation example, and is stored in the external memory 41 as data representing the moving image through a predetermined process. Specifically, the moving image frame data corresponding to the frame FL33 is created by weighted addition synthesis of the original image data corresponding to the frame FL32 and the first still image data, and the moving image frame data corresponding to the frames FL34 and FL35. Are generated by weighted addition synthesis of the first still image data and the second still image data, and the moving image frame data corresponding to the frame FL36 corresponds to the second still image data and the frame FL37. It is created by weighted addition synthesis with original image data. Since the exposure amounts of the first still image data and the second still image data are almost half of the appropriate level 63, the gain is doubled in the signal processing before the weighted addition synthesis. .

  Although slightly different from the content shown in FIG. 19, considering the case where the exposure time of the first and second still image shooting matches the length of the two frame periods of moving image shooting as shown in FIG. , The moving image frame data corresponding to the frame FL34 is obtained by multiplying each of the pixel data constituting the first still image data after the gain increase by 3/4 to the first frame after the gain increase. 2 is created by adding the pixel data constituting the still image data of 2 multiplied by ¼. Of course, this addition is performed between pixel data corresponding to the same pixel.

If the shooting timings for the first and second still image shootings are β ₂ and β _3, and the shooting timing of the moving image that was shot in the frame FL ₃₄ if no still image shooting was taken is α ₃₄ , “3/4” as a weighting coefficient is calculated by dividing “the length of time between the imaging timings α ₃₄ and β ₃ ” by “the length of time between the imaging timings β ₂ and β ₃ ”. Is calculated by dividing “the length of time between the imaging timings β ₂ and α ₃₄ ” by “the length of time between the imaging timings β ₂ and β ₃ ”. .

<< Second Embodiment >>
When changing the exposure setting such as an aperture to capture a still image during moving image shooting, ideally, the exposure setting is changed so that the brightness of the captured image is kept constant. However, in reality, the still image represented by the still image data and the moving images before and after it are limited due to the control accuracy of the mechanically operated aperture, the exposure time setting accuracy, the amplification accuracy setting accuracy, etc. May not be the same brightness. In this case, if moving image frame data is created using the still image data without any correction regarding brightness, flicker may occur in the moving image brightness. The second embodiment described below is an improvement of the first embodiment in order to solve this problem.

  FIG. 21 is an overall block diagram of the imaging apparatus 1a according to the second embodiment. The image pickup apparatus 1a is a digital still camera or the like similar to the image pickup apparatus 1 of FIG. 1, and is configured to be able to take both a moving image and a still image (simultaneously) as with the image pickup apparatus 1. In FIG. 21, the same parts as those in FIG. 1 are denoted by the same reference numerals, and the description of the same parts is omitted. The imaging device 1a has a configuration in which a brightness detection unit (luminance detection unit) 27 and a brightness correction unit (luminance correction unit) 28 are added to the imaging device 1 of FIG. The imaging device 1 and the imaging device 1a are identical except that the brightness detection unit 27 and the brightness correction unit 28 are added and that the code representing the signal processing unit is changed from 20 to 20a with the addition. ing. A signal processing unit 20a in FIG. 21 is obtained by adding a brightness detection unit 27 and a brightness correction unit 28 to the signal processing unit 20 in FIG.

  The brightness correction unit 28 is provided between the preprocessing unit 14 and the pixel number conversion unit 21. In the second embodiment, the original image data output by the preprocessing unit 14 is given to the brightness detection unit 27 and the brightness correction unit 28, and the output data of the brightness correction unit 28 is the pixel number conversion unit 21. The motion detection unit 22 and the still image processing unit 25 are provided. For this reason, the pixel number conversion unit 21, the motion detection unit 22, and the still image processing unit 25 in the second embodiment receive the output data of the brightness correction unit 28 as input data, and perform the same operation as in the first embodiment.

  The brightness detection unit 27 obtains the average luminance of the image represented by the original image data sent from the preprocessing unit 14 one after another based on each original image data. Then, the average brightness corresponding to each obtained original image data is transmitted to the brightness correction unit 28. This average luminance indicates the brightness of the image represented by each original image data.

  Here, the average brightness detected by the brightness detection unit 27 and transmitted to the brightness correction unit 28 may be the average brightness of the entire image, or the average brightness of an area of an arbitrary size in the center of the image. There may be. Further, the average luminance of a plurality of areas set in the image may be used. In addition, a plurality of areas in the image are determined and the average brightness of each determined area is obtained, and the average brightness of each area is weighted and added to obtain the final average brightness transmitted to the brightness correction unit 28. Good.

  As described above, various types of average luminance can be adopted as the average luminance transmitted to the brightness correction unit 28. However, which part of the image the average luminance is to be obtained is determined in all shooting. Must be the same. That is, the brightness detection unit 27 calculates the average luminance of a single region or a plurality of regions set at the same position in the image (still image and moving image) and transmits the average luminance to the brightness correction unit 28. The “entire image” can be regarded as a single region.

  The brightness correction unit 28 performs brightness correction on the original image data sent one after another from the preprocessing unit 14 while referring to the given average luminance, and converts the number of pixels of the image data after the brightness correction. Output to the unit 21, the motion detection unit 22, and the still image processing unit 25.

  For the sake of concrete explanation, an operation in the case of taking a still image during moving image shooting will be considered while following the example shown in FIG. The average luminance of the image represented by the original image data corresponding to the frame immediately before the still image shooting, that is, the frame FL3, and the average luminance of the image represented by the original image data (still image data) obtained by shooting the frame FL4. Suppose that they are “11” and “10”, respectively.

  In this case, the brightness correction unit 28 treats “11” as the target average brightness L1, treats “10” as the correction target average brightness L2, and calculates a brightness correction value by dividing the target average brightness by the correction target average brightness. G (1.1 in this case). Then, the brightness correction unit 28 sets the average brightness of the image represented by the still image data after the brightness correction to G times the average brightness of the image represented by the still image data before the brightness correction (now In this case, the given still image data is corrected (brightness correction) so as to be 1.1 times. For example, when the numerical value of each pixel data constituting the still image data is a numerical value proportional to the luminance, the numerical value of each pixel data constituting the still image data may be uniformly multiplied by 1.1. On the other hand, the brightness correction unit 28 does not perform brightness correction on the original image data corresponding to moving image shooting. However, it is possible to uniformly amplify the signal level of the original image data corresponding to all of the moving image and the still image.

  Each part (moving image processing unit 23 and the like) subsequent to the brightness correction unit 28 uses the still image data after the brightness correction, and performs the same operation as the first to fourth operation examples of the first embodiment. Then, moving image frame data to be stored in the external memory 41 is created.

  Thereby, the still image data is corrected (brightness correction) so that the brightness of the still image is the same (substantially the same) as that of the moving image immediately before the still image. Movie frame data during movie shooting is created based on the still image data after the correction (brightness correction), so that the brightness of the movie that can occur due to restrictions on aperture control accuracy, etc. Flicker is suppressed.

  In addition, while using the frame memory buffer 40, the average luminance of the image immediately after the still image shooting, that is, the image represented by the original image data corresponding to the frame FL5 may be handled as the target average luminance L1. In this case, the still image data is corrected (brightness correction) so that the brightness of the still image and the brightness of the moving image immediately after the still image are the same (substantially the same).

  In addition, the average luminance (for example, “11”) of the image represented by the original image data corresponding to the frame immediately before the still image shooting (that is, the frame FL3) and the frame immediately after the still image shooting (that is, the frame FL5). The average luminance (for example, (11 + 13) / 2 = 12) with the average luminance (for example, “13”) of the image represented by the corresponding original image data may be handled as the target average luminance L1. In this case, the still image data is corrected (brightness correction) so that the brightness of the still image and the average of the brightness of the moving image immediately after and immediately after the still image are the same (substantially the same). .

  Note that the brightness correction unit 28 may be arranged at the subsequent stage of the pixel number conversion unit 21. That is, the brightness correction by the brightness correction unit 28 may be performed after the pixel number conversion by the pixel number conversion unit 21.

<< Third Embodiment >>
Here, the influence of the decrease in the amount of peripheral light when taking a still image during moving image shooting will be considered. As shown in FIG. 22, the diagonal line on the imaging surface of the imaging element 13 is taken as the z axis, and the center of the imaging surface is taken as C (the center C is on the z axis). In this case, when light of uniform luminance is applied to the lens constituting the optical system 11, if the lens is ideal, the amount of light received by each pixel constituting the image sensor 13 is the same. However, as shown in FIG. 23, in actuality, the amount of light received by the pixel located at the center C is maximized, and the amount of light received by the pixels located at the periphery of the imaging surface is smaller than that at the center C. This decrease in the amount of received light at the peripheral portion is “a decrease in the amount of peripheral light”.

  In addition, the degree of the decrease in the amount of peripheral light increases as the diaphragm 12 is closer to the open position (the larger the opening amount of the diaphragm 12). Therefore, as in the example shown in FIG. 3, if the aperture of the aperture 12 is increased only when still image shooting is performed during moving image shooting, the amount of peripheral light is reduced between the still image shooting and the previous and next moving image shooting. The degree of decline will change. If moving image frame data during still image shooting is generated from still image data without considering the change in the degree of decrease in the peripheral light amount, the peripheral portion of the moving image may flicker at the time of still image shooting.

  A third embodiment will be described as an embodiment that also eliminates the influence of such a decrease in the amount of peripheral light. FIG. 24 is an overall block diagram of the imaging apparatus 1b according to the third embodiment. The imaging device 1b is a digital still camera or the like similar to the imaging device 1 of FIG. 1, and is configured to be able to capture both a moving image and a still image (simultaneously), like the imaging device 1. . In FIG. 24, the same portions as those in FIGS. 1 and 21 are denoted by the same reference numerals, and redundant description of the same portions is omitted. The imaging device 1b has a configuration in which a peripheral light amount correction unit 29 is added to the imaging device 1a of FIG. The imaging device 1a in FIG. 21 and the imaging device 1b in FIG. 24 are identical except that the peripheral light amount correction unit 29 is added and the code representing the signal processing unit is changed from 20a to 20b in accordance with the addition. ing. A signal processing unit 20b in FIG. 24 is obtained by adding a peripheral light amount correction unit 29 to the signal processing unit 20a in FIG.

  The peripheral light amount correction unit 29 is provided between the preprocessing unit 14 and the brightness correction unit 28. In the third embodiment, the original image data output by the preprocessing unit 14 is given to the peripheral light amount correction unit 29, and the output data of the peripheral light amount correction unit 29 is sent to the brightness detection unit 27 and the brightness correction unit 28. Is given. The output data of the brightness correction unit 28 is given to the pixel number conversion unit 21, the motion detection unit 22, and the still image processing unit 25, as in the second embodiment. For this reason, the pixel number conversion unit 21, the motion detection unit 22, and the still image processing unit 25 in the third embodiment receive the output data of the brightness correction unit 28 as input data, as in the second embodiment, and the first embodiment Performs the same operation as the form.

  The peripheral light amount correction unit 29 performs peripheral light amount correction on the original image data sent one after another from the preprocessing unit 14 while referring to information specifying the aperture amount of the diaphragm 12 from the main control unit 19, and The image data after the light amount correction is output to the brightness detection unit 27 and the brightness correction unit 28. This peripheral light amount correction is performed so as to cancel the fluctuation of the decrease in the peripheral light amount due to the change of the aperture amount of the diaphragm 12 between the moving image and the still image shooting when the still image shooting is performed during the moving image shooting.

  For example, the peripheral light amount correction unit 29 targets all original image data regardless of a still image or a moving image so that the influence of the decrease in the peripheral light amount does not completely exist in the output data of the peripheral light amount correction unit 29. The peripheral light amount correction is performed. That is, for all the original image data, the peripheral light amount correction is performed so as to cancel the decrease in the amount of received light in the peripheral portion of the imaging surface due to the decrease in the peripheral light amount. For example, when it is known that the amount of light received at the peripheral portion is 0.8 times the amount of light received at the center C with respect to a certain opening amount of the diaphragm 12, all the original image data is processed as a target. Is multiplied by 1 / 0.8.

  Further, for example, a state in which the decrease in the peripheral light amount is the smallest (for example, a state in which the aperture amount of the diaphragm 12 is minimum) is set as the reference state, and the influence of the decrease in the peripheral light amount in each photographing is the decrease in the peripheral light amount in the reference state. Peripheral light amount correction may be performed so as to be equivalent to the influence. In this case, the peripheral light amount correction is also performed on all original image data regardless of a still image or a moving image. Note that “the influence of the decrease in the amount of peripheral light” means a decrease in the luminance of the peripheral portion of the reproduced image caused by the decrease in the amount of peripheral light.

  In addition, for example, the state of moving image shooting at a timing around still image shooting (for example, a moving image frame immediately before or after still image shooting) is set as a reference state, and the influence of a decrease in peripheral light amount in still image shooting is the reference state. The peripheral light amount correction may be performed so as to be equivalent to the influence of the decrease in the peripheral light amount. In this case, the peripheral light amount correction is performed only on the original image data (that is, still image data) corresponding to still image shooting.

  The operations of the brightness detection unit 27 and the brightness correction unit 28 that have received the original image data after the peripheral light amount correction as input data are the same as those in the second embodiment. That is, the operations of the brightness detection unit 27 and the brightness correction unit 28 in the third embodiment are the same as the descriptions of “original image data” and “still image data” in the second embodiment, and “original data after peripheral light amount correction”. “Image data” and “Still image data after peripheral light amount correction”.

  When the peripheral light amount correction is performed as described above, the change in the decrease in the peripheral light amount due to the change in the aperture of the diaphragm 12 between the moving image and the still image shooting is canceled, and the still image shooting is performed during the moving image shooting. In addition, it is possible to prevent the peripheral portion of the moving image from flickering.

  Further, the peripheral light amount correction unit 29 and the brightness correction unit 28 may be arranged at the subsequent stage of the pixel number conversion unit 21. That is, the peripheral light amount correction by the peripheral light amount correction unit 29 and the brightness correction by the brightness correction unit 28 may be performed after the pixel number conversion by the pixel number conversion unit 21. Further, what kind of peripheral light amount correction should be applied to the aperture amount of the diaphragm 12 is preset based on the characteristics of the diaphragm 12 and the like.

  According to the first to third embodiments, a still image scene selection function (program automatic exposure function optimized for a shooting scene) can be used even when a still image is shot during moving image shooting. become.

  The present invention is suitable for an imaging apparatus such as a digital still camera or a digital video camera. The present invention is also suitable for a surveillance camera. Further, it is also suitable for a mobile phone or a computer device having the functions of a digital still camera or a digital video camera.

1 is an overall block diagram of an imaging apparatus according to a first embodiment of the present invention. It is a figure for demonstrating subject blurring in relation to exposure time. It is a figure which shows the control concept of the 1st operation example of the imaging device of FIG. It is a figure which shows an example of an internal structure of the moving image process part of FIG. It is a figure for demonstrating the production method of the moving image frame data in the still image imaging | photography using the data for still images by the moving image processing part of FIG. It is a figure for demonstrating the production method of the moving image frame data in the still image imaging | photography using the data for still images by the moving image processing part of FIG. It is a figure for demonstrating the production method of the moving image frame data in the still image imaging | photography using the data for still images by the moving image processing part of FIG. It is a figure for demonstrating the production method of the moving image frame data in the still image imaging | photography using the data for still images by the moving image processing part of FIG. It is a figure for demonstrating the production method of the moving image frame data in the still image imaging | photography using the data for still images by the moving image processing part of FIG. It is a figure for demonstrating the production method of the moving image frame data in the still image imaging | photography using the data for still images by the moving image processing part of FIG. FIG. 4 is a diagram illustrating an example of shooting timing of each shooting applicable to the first operation example of FIG. 3. It is a figure which shows the other example of the imaging timing of each imaging | photography applicable to the 1st operation example of FIG. It is a figure which shows the control concept of the 2nd operation example of the imaging device of FIG. It is a figure which shows the control concept of the 3rd operation example of the imaging device of FIG. It is a figure for demonstrating the principle of the weighted addition composition by the moving image process part of FIG. It is a figure which shows an example of the imaging timing of each imaging | photography applicable to the 3rd operation example of FIG. FIG. 15 is a diagram illustrating another example of the photographing timing of each photographing that can be applied to the third operation example of FIG. 14. FIG. 15 is a diagram for explaining a technique of weighted addition synthesis applicable to the third operation example of FIG. 14. It is a figure which shows the control concept of the 4th operation example of the imaging device of FIG. It is a figure for demonstrating the principle of the weighted addition composition which concerns on the 4th operation example of FIG. It is a whole block diagram of the imaging device concerning a 2nd embodiment of the present invention. It is a figure which shows the relationship between the imaging surface of the imaging device of FIG. 21, and a z-axis direction. It is a figure showing the fall of a peripheral light quantity. It is a whole block diagram of the imaging device concerning a 3rd embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1a, 1b Image pick-up device 11 Optical system 12 Aperture 13 Image pick-up element 14 Preprocessing part 15 Amplifying circuit 16 A / D converter 17 Driver 18 Timing generator 19 Main control part 20, 20a, 20b Signal processing part 21 Pixel number conversion part 22 motion detection unit 23 moving image processing unit 24 moving image compression unit 25 still image processing unit 26 still image compression unit 27 brightness detection unit 28 brightness correction unit 29 peripheral light amount correction unit 31 difference calculation unit 32 subject blur region specifying unit 33 blur processing 40 Frame memory buffer 41 External memory 42 Operation unit

Claims (11)

An image sensor that photoelectrically converts an incident optical image;
An amplification circuit that amplifies and outputs a signal from the image sensor;
A signal processing unit for processing image data obtained from the output signal of the amplifier circuit;
A main control unit that controls exposure settings of the image sensor and controls photographing of the optical image, and an imaging device capable of photographing moving images and still images;
When taking a still image during moving image shooting, the main control unit changes the exposure setting from moving image shooting to still image shooting and then takes the still image, and the signal processing unit takes the still image. Using the still image data that is the image data obtained by shooting, create the video frame data during the still image shooting,
The exposure setting controlled by the main control unit includes an exposure time of the image sensor, an aperture amount of an aperture that specifies an incident light amount per unit time to the image sensor, and / or an amplification degree of the amplifier circuit, Contains
The signal processing unit corrects subject blurring due to movement of the subject included in the still image data according to a difference between the exposure time for moving image shooting and the exposure time for still image shooting, and after the correction An image pickup apparatus that creates the moving image frame data during still image shooting based on the still image data. The signal processing unit obtains a subject blur area where the subject moves based on a difference between the still image data and image data obtained by photographing before and after photographing for obtaining the still image data. A subject blur region specifying unit is provided, and fluctuations in subject blur resulting from the difference between the exposure time for moving image shooting and the exposure time for still image shooting correspond to the subject blur region in the still image data. The imaging apparatus according to claim 1 , wherein the subject blur amount is corrected by adding the data . An image sensor that photoelectrically converts an incident optical image;
An amplification circuit that amplifies and outputs a signal from the image sensor;
A signal processing unit for processing image data obtained from the output signal of the amplifier circuit;
A main control unit that controls exposure settings of the image sensor and controls photographing of the optical image, and an imaging device capable of photographing moving images and still images;
When taking a still image during moving image shooting, the main control unit changes the exposure setting from moving image shooting to still image shooting and then takes the still image, and the signal processing unit takes the still image. Using the still image data that is the image data obtained by shooting, create the video frame data during the still image shooting,
The exposure setting controlled by the main control unit includes an exposure time of the image sensor, an aperture amount of an aperture that specifies an incident light amount per unit time to the image sensor, and / or an amplification degree of the amplifier circuit, Contains
When the time required for changing the aperture of the diaphragm for still image shooting is longer than one frame period of moving image shooting,
The main control unit uses the plurality of frame periods to change the aperture amount of the diaphragm for still image shooting and then shoots the still image, and the signal processing unit is obtained during the plurality of frame periods. Each subject blur amount due to the movement of the subject included in each image data is corrected according to the difference between the exposure time for moving image shooting and each exposure time used during the plurality of frame periods, and thereby the plurality of frames it characterized <br/> to create each video frame data during imaging device. An image sensor that photoelectrically converts an incident optical image;
An amplification circuit that amplifies and outputs a signal from the image sensor;
A signal processing unit for processing image data obtained from the output signal of the amplifier circuit;
A main control unit that controls exposure settings of the image sensor and controls photographing of the optical image, and an imaging device capable of photographing moving images and still images;
When taking a still image during moving image shooting, the main control unit changes the exposure setting from moving image shooting to still image shooting and then takes the still image, and the signal processing unit takes the still image. Using the still image data that is the image data obtained by shooting, create the video frame data during the still image shooting,
The exposure setting controlled by the main control unit includes an exposure time of the image sensor, an aperture amount of an aperture that specifies an incident light amount per unit time to the image sensor, and / or an amplification degree of the amplifier circuit, Contains
When shooting the still image during movie shooting over a period longer than one frame period of movie shooting,
Before SL signal processing unit, by weighted addition combining the image data obtained by photographing before or after shooting for obtaining the still image data and the still image data, wherein in the still image shooting Create video frame data,
Its weighting weighting factors in addition is weighted addition synthesis subject to the image data of to that imaging device, wherein <br/> be determined a photographing timing by the capturing timing of the data for the still image. An image sensor that photoelectrically converts an incident optical image;
An amplification circuit that amplifies and outputs a signal from the image sensor;
A signal processing unit for processing image data obtained from the output signal of the amplifier circuit;
A main control unit that controls exposure settings of the image sensor and controls photographing of the optical image, and an imaging device capable of photographing moving images and still images;
When taking a still image during moving image shooting, the main control unit changes the exposure setting from moving image shooting to still image shooting and then takes the still image, and the signal processing unit takes the still image. Moving image frame data during still image shooting is created using still image data, which is image data obtained by shooting, and the exposure setting controlled by the main control unit includes the exposure time of the image sensor. , An aperture amount of a diaphragm for specifying an incident light amount per unit time to the image sensor and / or an amplification degree of the amplifier circuit are included.
The signal processing unit corrects subject blurring due to movement of the subject included in the still image data according to a difference between the exposure time for moving image shooting and the exposure time for still image shooting, and after the correction Based on the still image data, create the video frame data during still image shooting,
When shooting the still image during movie shooting over a period longer than one frame period of movie shooting,
The signal processing unit is configured to perform weighted addition synthesis of the still image data and the image data obtained by shooting before or after shooting for obtaining the still image data, thereby moving the moving image during still image shooting. Create frame data,
Its weighting weighting factors in addition is weighted addition synthesis subject to the image data of to that imaging device, wherein <br/> be determined a photographing timing by the capturing timing of the data for the still image. The signal processing unit performs weighted addition synthesis only for a portion in which a difference between the image data to be subjected to weighted addition synthesis and the still image data is a predetermined threshold value or more. The imaging device according to claim 4 or 5 . The exposure setting controlled by the main controller when shooting the still image during moving image shooting includes an exposure time of the image sensor and an amplification degree of the amplifier circuit,
When shooting the still image during moving image shooting, when it is determined that the exposure is insufficient even if the exposure time is set to a predetermined upper limit by changing the exposure setting by the main control unit ,
The main control unit shoots the still image after compensating for insufficient exposure by increasing the amplification degree.
The imaging apparatus according to any one of claims 1 to 6, wherein When shooting the still image during moving image shooting, it is determined that the exposure is insufficient even if the exposure time is set to a predetermined upper limit value by changing the exposure setting by the main control unit. When
The main control unit captures a plurality of images after setting the exposure time to be equal to or less than the upper limit value, and the signal processing unit combines a plurality of image data obtained by capturing the plurality of images. The image pickup apparatus according to any one of claims 1 to 6 , wherein data representing the still image to be shot during moving image shooting is created . The signal processing unit includes a brightness detection unit that detects the brightness of each captured moving image and still image based on an output signal of the amplification circuit, and when the still image is captured during moving image capturing. A brightness correction unit that corrects the still image data so that the difference between the brightness of the still image and the brightness of the moving image before or after the still image is reduced, and the still image after the correction The image pickup apparatus according to any one of claims 1 to 8 , wherein the moving image frame data during still image shooting is created by using image data. When the still image is shot during moving image shooting, the brightness correction unit is configured to calculate the average luminance of a single region or a plurality of regions set at the same position in the moving image before or after the still image and the still image. However, by correcting the still image data so as to match between the still image and the previous or subsequent moving image, the difference between the brightness of the still image and the previous or subsequent moving image is reduced. <br/> to imaging apparatus according to claim 9, characterized in. The exposure setting controlled by the main controller when shooting the still image during moving image shooting includes at least the exposure time and the aperture amount of the diaphragm,
The signal processing unit includes a peripheral light amount correction unit that corrects a variation in decrease in peripheral light amount due to a change in the aperture amount between the moving image and the still image before the brightness correction unit. The imaging apparatus according to claim 9 or 10, wherein:

Images