JP6758950B2 - Imaging device, its control method and program - Google Patents

Description

The present invention relates to an imaging device that executes imaging for acquiring a time-lapse moving image, and a control method and program thereof.

Conventionally, there is a known technique for acquiring a moving image (so-called time-lapse movie) recorded by compressing a temporal change of a subject by sequentially connecting a plurality of images acquired by intermittently capturing the subject. Has been done.

As one of the imaging methods for acquiring this time-lapse movie, Patent Document 1 proposes a technique of generating a time-lapse movie by thinning out and compressing an image from a series of movies (video-based interval shooting). There is.

Japanese Unexamined Patent Publication No. 2015-142327

However, in the technique described in Patent Document 1, when performing video-based interval shooting, a frame image is acquired at a set shooting interval, and the acquired image is suitable for generating a time-lapse movie. It is not considered whether or not it is. Therefore, in the technique described in Patent Document 1, even if the acquired image is not suitable for generating a time-lapse moving image, the time-lapse moving image may be generated using the image. As an image unsuitable for generating this time-lapse moving image, for example, an image in which so-called rolling shutter distortion has occurred, an image in which exposure change and white balance adjustment have not been completed, and the like can be considered. Therefore, in a time-lapse movie generated using an image that is not suitable for generating a time-lapse movie, the shape, brightness, and color appearance of the subject portion change unnaturally, which gives the user a sense of discomfort.

An object of the present invention is to suppress unnatural changes in the subject portion in a time-lapse moving image.

The imaging apparatus of the present invention for achieving the above object includes imaging means, and in order to acquire a time-lapse moving image in which a plurality of images are joined, a plurality of images constituting a moving image acquired by using the imaging means. An imaging device capable of setting a first time-lapse mode for determining an image to be used for generating a time-lapse moving image from among the images. In the first time-lapse mode, the time-lapse mode is selected from a plurality of images constituting the moving image. setting means for setting the first interval of acquiring the image used for generating the video, in the first time-lapse mode, among the plurality of images constituting the moving image, the first of the setting means has set The determination means includes a determining means for determining whether or not the image acquired at intervals is an image suitable for generating a time-lapse moving image, and a determining means for determining an image to be used for generating the time-lapse moving image. In the first time-lapse mode, when the determination means determines that the first image obtained according to the first interval is an image suitable for generating the time-lapse moving image , the first image is used. When the image was determined as the image used for generating the time-lapse moving image and the determination means determined that the first image was not an image suitable for generating the time-lapse moving image, it was obtained at a timing different from the first interval. It is characterized in that the second image is determined as an image used for generating a time-lapse moving image .

According to the present invention, it is possible to suppress unnatural changes in the subject portion in a time-lapse moving image.

It is a block diagram which shows the structural example of the digital camera 1 which is embodiment of the image pickup apparatus which carried out this invention. It is a flowchart which shows the imaging mode determination process which concerns on this invention. It is a flowchart which shows the imaging process in the moving image time-lapse mode which concerns on this invention. It is a figure which exemplify the rolling shutter distortion at the time of imaging a moving body. It is a flowchart which shows the appropriate image determination process which concerns on this invention. It is a figure which illustrates the change of the exposure which follows the change of the luminance of the subject at the time of acquisition of a moving image which concerns on this invention.

(Embodiment)
(Basic configuration of digital camera 1)
Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a block diagram showing a configuration example of a digital camera (hereinafter, simply referred to as a camera) 1 which is an embodiment of an imaging device according to the present invention. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. You may. It may also be realized by a combination of software and hardware. Therefore, in the following description, the same hardware can be realized as the main body even if different functional blocks are described as the main body of operation.

As shown in FIG. 1, the camera 1 of the present embodiment is a so-called interchangeable lens type imaging device including a camera unit 100, an external recording medium 200, and a lens unit 300, but is limited thereto. It's not a thing. For example, the camera unit 100 and the lens unit 300 may be integrally provided.

The main mirror 101 is a first optical member that guides the light flux corresponding to the optical image of the subject guided from the lens unit 300 to the inside of the camera unit 100 to the image sensor 103 side and the optical finder 105 side, which will be described later. The shutter 102 is a light-shielding member for opening and light-shielding an optical path between the lens unit 300 and the image sensor 103. The image pickup device 103 is a charge storage type solid-state image pickup device such as a CCD or CMOS, and is an image pickup means for generating analog image data by photoelectric conversion (imaging) of a light flux of a subject incident on the lens unit 300. .. The pentapris 104 is a second optical member that guides the luminous flux incident through the main mirror to the optical finder 105. The optical finder 105 is a third optical member that allows the user to confirm the optical image of the subject.

The A / D conversion unit 106 is a conversion means for converting the analog image data output from the image sensor 103 into digital image data. The image processing circuit 107 is an image processing means that performs various processes such as white balance adjustment processing and gradation processing on the digital image data output from the A / D conversion unit 106.

The timing generation circuit 108 is a timing signal generating means for generating a signal (control signal such as a clock signal) for operating the image sensor 103, the A / D conversion unit 106, the D / A conversion unit 109 described later, and the like. Is. Further, the timing generation circuit 108 can control the charge accumulation in the image sensor 103 by controlling the reset timing of the accumulated charge in the image sensor 103. The timing generation circuit 108 is controlled by the system control unit 120, which will be described later.

When the memory control circuit 110 controls the A / D conversion unit 106, the image processing circuit 107, the D / A conversion unit 109, and the compression / expansion circuit 111 and writes the acquired image data to the image display memory 112 or the image recording memory 113. It is a memory control means that executes the control of. Further, the image display unit 114 is a display means that employs a TFT type LCD (thin film transistor drive type liquid crystal display), an organic EL element (organic electroluminescence element), or the like. The display digital image data written in the image display memory 112 is converted into display analog image data by the D / A conversion unit 109, and then displayed on the image display unit 114.

The image recording memory 113 is a recording means for storing image data acquired by photographing a subject, and has a storage capacity sufficient for storing a predetermined number of still image data and moving image data. The image recording memory 113 can also be used as a work area of the system control unit 120, which will be described later. The compression / decompression circuit 111 is a compression / decompression means that reads image data stored in the image recording memory 113 and compresses and decompresses the image data according to a predetermined image compression method and image decompression method according to various uses.

The shutter control circuit 115 is a shutter control means for controlling the operation of the shutter 102, and controls the operation of the shutter 102 based on the photometric result of the subject calculated by the system control unit 120. The shutter 102 is controlled in conjunction with the control of the aperture 302 described later.

The focus detection circuit 116 is a focus detection means including a focus sensor (not shown) and detecting the focused state of an optical image corresponding to a light flux incident from the lens unit 300 side via a sub mirror (not shown). The photometric circuit 117 is a photometric means that includes a photometric sensor (not shown) and calculates the brightness (luminance value) of an optical image corresponding to a luminous flux incident from the lens unit 300 side via a photometric lens (not shown). .. The detection results of the focus detection circuit 116 and the photometric circuit 117 are output to the system control unit 120 and used for lens position control (focus control) and exposure control of the focus lens (not shown) included in the lens unit 300. In the present embodiment, the focus control is executed by the phase difference detection method based on the output of the focus detection circuit 116, but the focus control is executed based on the contrast information of the image data acquired while shifting the position of the focus lens. May be configured.

The motion detection circuit 118 includes, for example, a gyro sensor and an acceleration sensor, and is a motion detection means for the camera 1 such as the movement direction and the movement speed of the camera 1 based on the output from the gyro sensor and the acceleration sensor. That is, the motion detection circuit 118 is a blur detection means for detecting the blur and the amount of blur of the camera 1. For the motion detection of the camera 1 by the motion detection circuit 118, a known method other than the above may be adopted.

The moving object detection circuit 119 is a moving object detecting means for detecting a moving object existing within the angle of view corresponding to the image data acquired by using the image sensor 103. Specifically, the moving object detection circuit 119 detects a moving object from a plurality of continuous image data by the background subtraction method. The method for detecting a moving body is not limited to this, and for example, a motion vector may be calculated and the moving body may be detected based on the motion vector. In the present embodiment, the moving object detection circuit 119 executes moving object detection by the background subtraction method for each frame during moving image shooting, and determines that a subject having a movement amount of a predetermined value or more is a moving object, but the processing load of the camera unit 100 is increased. In consideration of this, the motion detection may be performed at predetermined time intervals.

The system control unit 120 is a system control means that comprehensively controls the operation of the camera 1. The system control unit 120 is also an exposure control means that controls the exposure based on the output (luminance value) of the photometric circuit 117. In the present embodiment, as the exposure control described above, parameters such as the aperture value related to the opening degree of the aperture 302, the shutter speed related to the charge accumulation time of the image sensor 103, and the shooting sensitivity related to the analog and digital gain amounts are changed.

Information (table data, etc.) regarding exposure (appropriate exposure) with respect to the luminance value is stored in advance in the main memory 121. The system control unit 120 can set an appropriate exposure according to the brightness value based on this information.

Further, the system control unit 120 is also a control means for executing various controls related to the generation of the time-lapse moving image in the time-lapse mode described later. Details of various controls in the time-lapse mode will be described later.

The main memory 121 is a recording means for recording data related to the operation of the camera 1. In the main memory 121, constants for operations executed by the camera 1, various exposure conditions, calculation formulas, and the like are recorded in advance.

The image synthesizing unit 122 is an image synthesizing means for generating a moving image (time-lapse moving image) in which temporal changes are compressed by joining the image data acquired in the time-lapse mode in the order in which they are captured.

The non-volatile memory 123 is a storage means that can be electrically erased and stored, such as an EEPROM represented by a flash memory or the like.

Each unit described below is an operation means for inputting various operation instructions to the system control unit 120, and is composed of a button, a switch, a dial, a touch panel, a line-of-sight detection device, a voice recognition device, or a combination thereof. Will be done.

The mode dial 130 is an operation member used when setting an arbitrary shooting mode from a plurality of shooting modes that can be set by the camera unit 100. In the present embodiment, a plurality of modes such as a normal moving image mode, a moving image time-lapse mode (first time-lapse mode), and a still image time-lapse mode (second time-lapse mode) can be set as an imaging mode for acquiring a moving image. As the camera 1, not only the moving image mode but also the normal still image mode for executing imaging for acquiring a still image can be set.

The normal moving image mode is a mode in which a plurality of image data acquired by continuously executing charge accumulation (imaging) using the image sensor 103 are sequentially connected and displayed (or recorded). In the moving image time-lapse mode, image data for a time-lapse moving image (interval) based on a preset imaging interval (interval) from a plurality of image data acquired by continuously performing imaging using the image sensor 103 ( Hereinafter, it is a mode for determining an appropriate image). Then, in the moving image time-lapse mode, appropriate images can be connected and displayed (or recorded) in the acquisition order (captured order). The still image time-lapse mode is a mode in which a plurality of image data are acquired by intermittently performing image pickup using the image pickup device 103 based on a preset imaging interval (interval). Then, in the still image time-lapse mode, the acquired image data can be connected and displayed (or recorded) in the acquisition order (captured order).

In the moving image time-lapse mode, the charge accumulation line of the image sensor 103 when imaging the subject is smaller than that in the still image time-lapse mode. Therefore, the number of pixels of the image acquired in the moving image time-lapse mode is smaller than that of the image acquired in the still image time-lapse mode.

Further, in the moving image acquired in the normal moving image mode, the time required for imaging for acquiring the moving image and the reproduction time of the moving image are substantially the same. On the other hand, the time-lapse moving images acquired in the moving image time-lapse mode and the still image time-lapse mode differ in the time required for imaging for acquiring the moving image and the reproduction time of the moving image. Each time-lapse moving image acquired in the above-mentioned two types of time-lapse modes is a moving image obtained by joining intermittent image data acquired in a predetermined period (total imaging time). Therefore, the reproduction time of the moving image is shorter than the total imaging time (from the start to the end of imaging) for acquiring one time-lapse movie.

The camera 1 of the present embodiment can further set individual modes in each of the above-mentioned moving image modes. For example, the camera 1 can set an automatic mode, a program mode, a shutter speed priority mode, an aperture priority mode, a manual mode, a depth of focus priority mode, a portrait mode, a landscape mode, a close-up mode, a sports mode, and a night view mode.

The shutter switch 131 is an operating member used when instructing an imaging preparation operation of a subject or a start of an imaging operation. SW1 is turned on by the first stroke (for example, half-pressing) of the shutter switch 131. When SW1 is turned on, the imaging preparation operation is started, and the system control unit 120 starts focus control, exposure control, auto white balance (AWB) processing, light emission control, and the like.

Further, SW2 is turned on by the second stroke (for example, fully pressed) of the shutter switch 131. When the SW2 is turned on, the image pickup operation is started, and the system control unit 120 starts the exposure process and the recording process related to the charge accumulation (imaging) using the image pickup element 103.

In the exposure process, the signal read from the image sensor 103 is written to the image recording memory 113 as image data via the A / D conversion unit 106 and the memory control circuit 110 in response to an instruction from the system control unit 120. Then, in response to an instruction from the system control unit 120, development processing based on various calculations in the image processing circuit 107 and the memory control circuit 110 is executed on the image data, and the developed image data is stored in the image recording memory. It is written in 113.

In the recording process, the developed image data read from the image recording memory 113 is compressed by the compression / decompression circuit 111 in response to an instruction from the system control unit 120. After that, according to the instruction from the system control unit 120, the image data after the compression process is sent to the external recording medium 200 via the first camera I / F 140, the first camera connector 141, the media connector 203, and the media I / F 202. It is written in the recording unit 201.

The reproduction switch 132 is an operation member for instructing the start of the reproduction process of reading the acquired image data from the image recording memory 113 or the external recording medium 200 and displaying the acquired image data on the image display unit 114. The operation unit 133 is an operation member used for various settings related to menu display and imaging, and various settings related to reproduction. The imaging interval, the total number of imaging times, the total imaging time, and the like in the time-lapse mode can be set by the user operating the operation unit 133. The power switch 134 is an operating member used to switch on / off of power supply from a power supply unit (battery) (not shown) to each unit of the camera 1. By operating the power switch 134, it is possible to switch on / off the power supply not only to the camera unit 100 but also to various accessory devices such as the lens unit 300 connected to the camera unit 100 and the external recording medium 200.

The power supply control circuit 124 is a power supply control means including a battery detection circuit, a DC-DC converter, a switch circuit used for switching an energization block, and the like. The power supply control circuit 124 detects whether or not a battery is installed, the type of battery, and the remaining battery level based on an instruction from the system control unit 120 in response to the operation of the power switch 134, and obtains a required voltage for a required period. Only supplies to each part of the camera 1.

The second camera I / F 150 is provided on the camera mount unit 160 and is an interface for connecting the camera unit 100 and the lens unit 300. The second camera connector 151 is a connecting means for electrically connecting the camera unit 100 and the lens unit 300 via the lens connector 311 and the lens I / F 310. The second camera connector 151 can transmit control signals, state signals, data signals, and the like between the camera unit 100 and the lens unit 300, and can supply currents of various voltages. The second camera connector 151 may have a configuration capable of transmitting not only telecommunications but also optical communication, voice communication, and the like.

The external recording medium 200 is an external recording device such as a memory card or a hard disk. The external recording medium 200 includes a recording unit 201 composed of a semiconductor memory, a magnetic disk, or the like, a media I / F 202 for the camera unit 100, and a media connector 203 for connecting to the camera unit 100.

The lens unit 300 is an optical device that can be attached to and detached from the camera unit 100. The lens mount portion 320 is a connecting means that engages with the camera mount portion 160 to mechanically attach the lens unit 300 to the camera unit 100. Inside the lens mount portion 320, a lens connector 311 that electrically connects the lens unit 300 and the camera unit 100 is provided. The lens connector 311 can transmit control signals, state signals, data signals, and the like, and receive and supply currents of various voltages between the lens unit 300 and the camera unit 100. The lens connector 311 may have a configuration capable of transmitting not only telecommunications but also optical communication, voice communication, and the like.

The image pickup lens group 301 is an optical member including a focus lens, a zoom lens, a shift lens, and the like. The diaphragm 302 is a light amount adjusting member that adjusts the light amount of the light flux of the subject that passes through the image pickup lens group 301 and is incident on the image pickup element 103 side.

The diaphragm control circuit 303 is a diaphragm control means that controls the aperture amount of the diaphragm 302 based on an instruction from the system control unit 120. The system control unit 120 instructs the diaphragm control circuit 303 to change the aperture diameter of the diaphragm 302 so that the aperture amount corresponds to the target diaphragm value. The aperture diameter of the aperture 302 that is changing is sequentially detected by mutual communication between the lens unit 300 and the camera unit 100. Then, the system control unit 120 ends the change of the opening diameter of the diaphragm 302 when the aperture diameter of the diaphragm 302 reaches the opening diameter corresponding to the target diaphragm value.

The lens control circuit 304 is a lens drive control means for controlling the operation (drive) of the image pickup lens group 301. The lens control circuit 304 can detect the lens position (focal length) of the focus lens, and the information regarding the detected lens position is transmitted to the camera unit 100 side.

The lens system control unit 305 is a lens control means that comprehensively controls the lens unit 300. The lens system control unit 305 has a built-in CPU (not shown), a volatile memory, and a non-volatile memory, and the volatile memory stores constants, variables, programs, and the like for operation. Further, the non-volatile memory stores identification information such as a unique number related to the lens unit 300, management information, functional information such as an open aperture value and a minimum aperture value, and a focal length. The above is the basic configuration of the camera 1.

(Imaging mode determination process)
Hereinafter, the operation of determining the imaging mode set in the camera 1 will be described. FIG. 2 is a flowchart showing an imaging mode determination process according to the present invention. The imaging mode determination process is started in response to the operation of the mode dial 130 or the operation unit 133 while the power of the camera 1 is turned on. In step S201, the system control unit 120 detects whether the imaging mode set by the user is the normal still image mode or the moving image mode.

When the system control unit 120 detects that the normal still image mode is set, the process proceeds to step S202 to shift to the standby state of the imaging process in the normal still image mode. In this standby state, an imaging process (exposure process and recording process) for acquiring a still image is executed in response to the operation of the shutter switch 131. The description of the imaging process in the normal still image mode will be omitted.

When the system control unit 120 detects that the moving image mode is set, the system control unit 120 detects the frame rate of the moving image set by the user in step S203. Then, in step S204, the system control unit 120 detects whether the set moving image mode is the normal moving image mode or the time-lapse mode.

When the system control unit 120 detects that the normal moving image mode is set, the process proceeds to step S205, imaging is performed in the normal moving image mode, and the state shifts to the standby state for recording the image acquired by the imaging. In this standby state, a process (imaging process) for recording a normal moving image is executed in response to the operation of the shutter switch 131. The description of the imaging process in the normal moving image mode will be omitted.

When the system control unit 120 detects that any of the time-lapse modes is set, the system control unit 120 sets the imaging interval (interval) and the total number of imagings (or the total imaging time) set by the user in step S206. To detect. Then, in step S207, the system control unit 120 detects whether the set time-lapse mode is the moving time-lapse mode or the still image time-lapse mode.

When the system control unit 120 detects that the still image time-lapse mode is set, the process proceeds to step S208 to shift to the standby state of the imaging process in the still image time-lapse mode. In this standby state, in response to the operation of the shutter switch 131, an imaging process for acquiring a time-lapse moving image is executed in the still image time-lapse mode.

Although detailed description of the image pickup process in the still image time-lapse mode is omitted, the drive line (accumulation line) of the image sensor 103 in the still image time-lapse mode is larger than the drive line (accumulation line) of the image sensor 103 in the moving image time-lapse mode. There are many. This is because the accumulated lines of the image sensor 103 in the moving image time-lapse mode are thinned out from the total number of lines in consideration of the load of the camera 1 at the time of acquiring the moving image. Therefore, in the moving image time-lapse mode, the time required from the start of storage in the first storage line of the image sensor 103 to the completion of storage in the last storage line is shorter than that in the still image time-lapse mode. With this configuration, for example, even when the same subject is imaged, the amount of distortion of the subject portion due to the influence of the rolling shutter distortion is smaller in the moving image time-lapse mode than in the still image time-lapse mode.

Returning to FIG. 2, when the system control unit 120 detects that the moving image time-lapse mode is set, the process proceeds to step S209 to shift to the standby state of the imaging process in the moving image time-lapse mode. In this standby state, in response to the operation of the shutter switch 131, the imaging process for acquiring the time-lapse moving image is executed in the moving image time-lapse mode. The details of the imaging process in the moving image time-lapse mode will be described later.

When the imaging process in each imaging mode is completed, the system control unit 120 determines in step S210 whether or not the imaging mode has been changed. Then, when the system control unit 120 determines that the imaging mode has been changed, the process returns to step S201. When the system control unit 120 determines that the image pickup mode has not been changed, the system control unit 120 shifts to the standby state corresponding to the previously detected image pickup mode in step S211 and ends the image pickup mode determination process. .. The above is the imaging mode determination process according to the embodiment of the present invention.

(Video time-lapse mode)
Hereinafter, the operation of the camera 1 when the imaging mode is set to the moving image time-lapse mode will be described. FIG. 3 is a flowchart showing an imaging process in the moving image time-lapse mode according to the present invention. The flowchart illustrated in FIG. 3 corresponds to the process of step S209 in FIG.

The imaging process is started in response to the user setting the shutter switch 131 to the SW2 state. First, in step S301, the system control unit 120 executes charge accumulation and exposure processing by accumulating lines obtained by thinning out a predetermined number of lines of the image sensor 103, and based on the frame rate detected in step S203, image data (through image). ) Is started.

In parallel with the flow shown in FIG. 3, the system control unit 120 measures the subject for each predetermined frame and calculates a photometric value Bv_R representing the photometric result. This photometric value Bv_R is a target value of exposure control that follows the subject brightness executed when acquiring a moving image, and is information indicating a target value of each exposure parameter.

On the other hand, in the present embodiment, the control value Bv_C representing the target value of the exposure control for each frame is calculated before the acquisition of the through image, and the exposure control is executed for each frame (imaging) based on the control value Bv_C. To do. With this configuration, even when the photometric value Bv_R changes according to the change in the subject brightness, the exposure can be changed stepwise over a plurality of frames without suddenly changing the exposure. It is possible to acquire a moving image in which the brightness changes smoothly. For example, when the aperture 302 is operated to execute exposure control, the control value Bv_C that gradually changes the aperture diameter of the aperture 302 over a plurality of frames is the exposure control of each frame with respect to the photometric value Bv_R. Set as a target value.

Next, in step S302, the system control unit 120 determines whether or not the time measurement by the built-in timer (not shown) provided with the real-time clock has reached the imaging interval (interval) detected in step S206. In other words, the system control unit 120 determines whether or not the time corresponding to the interval has elapsed from the time measurement by the built-in timer.

If the system control unit 120 determines that the time measurement has not reached the imaging interval, the process returns to step S301. When the system control unit 120 determines that the time measurement has reached the imaging interval, the system control unit (determination means) 120 in step S303 determines that the acquired through image is an image suitable for generating a time-lapse moving image (appropriate image). It is determined whether or not it is.

The following can be considered as images that are not suitable for generating a time-lapse movie. For example, when CMOS is used as a charge storage type solid-state image sensor, the reset and read timings are different for each line of the image sensor 103. Therefore, when a subject is imaged using CMOS, the subject portion is unnaturally distorted as shown in FIG. 4 when the subject to be imaged contains a moving object or the image pickup device is blurred. (So-called rolling shutter distortion) occurs.

4A and 4B are diagrams exemplarily explaining the rolling shutter distortion when an image of a moving object is taken. FIG. 4A is a direction in which a predetermined subject A moves, and FIG. 4B is FIG. 4A. The rolling shutter distortion due to the movement of the subject A shown in) is described. As shown in FIG. 4, when the subject A shown in FIG. 4A is imaged with the camera 1 fixed, resetting and reading are performed on the upper side and the lower side of each line constituting the image sensor 103. A time difference Δt occurs at the timing (accumulation timing) of. In this case, since rolling shutter distortion occurs in the image data according to the time difference Δt, the portion corresponding to the subject A is unnaturally distorted.

Since a time-lapse movie is generated by joining a plurality of image data, in a time-lapse movie generated using the distorted image data, the subject part changes unnaturally due to the influence of the distortion, which gives the user a sense of discomfort. It ends up.

Further, when acquiring a moving image, since the quality of the moving image is prioritized, it is common to change the exposure smoothly according to the change of the subject light. However, even if the exposure changes smoothly in the entire moving image, the brightness of each frame (image data) constituting the moving image may deviate from the appropriate exposure for the subject. For example, when the exposure control is executed by changing the opening amount of the diaphragm 302, which is a mechanical configuration, a plurality of frames may be required from the start to the completion of the change of the diaphragm 302. In this case, the plurality of frames will be image data whose exposure is deviated from the target proper exposure.

Therefore, simply setting (determining) image data from each frame that constitutes a moving image according to the set imaging interval uses image data that deviates from the appropriate exposure to generate a time-lapse movie. It may end up. In the time-lapse movie generated in this case, a frame having an unnatural brightness with respect to the actual brightness of the subject exists, which gives the user a sense of discomfort.

Therefore, in the present embodiment, it is determined whether or not the image data acquired based on the imaging interval is an appropriate image, and the image to be used for generating the time-lapse moving image is determined according to the result of the determination. Solve this problem.

FIG. 5 is a flowchart showing a proper image determination process according to the present invention. As shown in FIG. 5, in step S501, the system control unit 120 calculates the amount of blurring of the camera 1 based on the output of the motion detection circuit 118. Then, the system control unit 120 determines that the camera 1 is shaken when the shake amount is larger than the predetermined shake amount.

Further, in step S502, the system control unit 120 calculates the amount of movement of each subject within the angle of view based on the output of the moving object detection circuit 119. Then, the system control unit 120 determines that the moving body exists within the angle of view when the movement amount is larger than the predetermined movement amount.

The processing of steps S501 and S502 is executed based on the background subtraction method with the through image acquired immediately before each time the through image is acquired, and the processing result is recorded in association with the through image. It is not limited to. For example, the camera 1 may be configured to perform blurring amount and motion detection at each timing when the appropriate image determination processing is executed (that is, at a predetermined timing based on the imaging interval). In this case, the amount of blurring of the camera 1 and the motion detection are executed by reading the through image acquired immediately before from the image recording memory 113 and comparing the through images before and after.

Next, in step S503, the system control unit 120 determines whether or not the aperture 302 is driven at the imaging timing for acquiring the corresponding through image. In the present embodiment, the through image acquired (imaged) while driving the aperture 302 is recorded in the image recording memory 113 in a state in which information (aperture drive flag) indicating that the image was acquired while driving the aperture is associated. To. Therefore, the system control unit 120 determines whether or not the image data is the image data acquired (imaged) while the aperture 302 is being driven, depending on the presence or absence of the aperture drive flag added to the corresponding image data. To do.

In this embodiment, it is not determined whether or not the corresponding through image is the image data acquired while the shutter speed or the shooting sensitivity (gain) is changed. This is because if the exposure is controlled by operating a mechanical configuration such as the aperture 302, the difference between the control value Bv_C and the photometric value Bv_R cannot be calculated accurately, so that it is difficult to perform the exposure compensation described later. caused by. In other words, it is possible to correct the exposure of the image data acquired by changing the shutter speed and the shooting sensitivity (gain) after the image data is acquired.

Next, in step S504, the system control unit 120 determines whether or not the difference between the control value Bv_C of the corresponding through image and the photometric value Bv_R is equal to or greater than a predetermined value. Specifically, in step S504, the system control unit 120 determines whether or not the absolute value of the difference between the control value Bv_C and the photometric value Bv_R of the corresponding image data is equal to or greater than a predetermined value. The predetermined value is, for example, 3Ev in APEX conversion, but is not limited to this, and any value can be used as long as the brightness of the image data can be appropriately adjusted when the exposure compensation described later is applied. You may adopt the thing.

If "NO" is determined in any of the processes of steps S501 to S504 described above, the system control unit 120 determines in step S505 that the corresponding image data is an appropriate image. If "YES" is determined in any of the processes of steps S501 to S504, the system control unit 120 determines in step S506 that the corresponding image data is not an appropriate image. That is, the camera 1 of the present embodiment is the image data acquired (imaged) when the camera 1 is not shaken and there is no moving object within the angle of view among the image data acquired based on the imaging interval. Is determined to be a proper image. Further, in the camera 1 of the present embodiment, among the image data acquired based on the imaging interval, the aperture diameter of the aperture 302 has not changed, and the difference between the control value Bv_C and the photometric value Bv_R is smaller than a predetermined value. The image data captured (acquired) in this state is determined to be an appropriate image.

In the present embodiment, it is determined whether or not the image data is an appropriate image depending on whether or not the corresponding image data is the image data acquired (imaged) while the aperture 302 is being driven. However, it is not limited to this. For example, similarly to the aperture 302, the image data acquired (imaged) during the operation of the ND filter (not shown) adopting the mechanical configuration may be determined as an improper image. In addition, among the image data acquired during the exposure control for following the change in the brightness of the subject, the image data whose brightness value exceeds a predetermined threshold range is used regardless of whether or not the mechanical configuration is operated. The configuration may be determined to be an improper image.

Returning to FIG. 3, in step S304, the system control unit 120 determines whether or not the corresponding image is an appropriate image based on the processing result of step S303. If the system control unit 120 determines that the image is not proper, the process returns to step S301. If the system control unit 120 determines that the image is appropriate, the process proceeds to step S305. In step S305, the system control unit 120 performs exposure compensation processing on the acquired image data. The details will be described below.

FIG. 6 is a diagram illustrating an example of a change in exposure that follows a change in brightness of a subject during acquisition of a moving image according to the present invention. As described above, in the present embodiment, a dead zone of (± ΔBv) in a predetermined range is provided around the control value Bv_C. Therefore, the exposure (or control value Bv_C) is not changed even if the photometric value Bv_R changes in the range of the dead zone based on the current control value Bv_C due to the change in the brightness of the subject.

In FIG. 6, the horizontal axis indicates the time axis, and it is assumed that time has elapsed toward the right side in the figure. Further, the vertical axis in the figure indicates a brightness value (Bv value), and in the present embodiment, the Bv value in terms of APEX becomes higher in brightness toward the upper side in the figure. Further, the solid line in the figure indicates the control value Bv_C, the alternate long and short dash line indicates the photometric value Bv_R, and the broken line indicates the range of the dead zone.

As shown in FIG. 6, at the time of <1>, the difference between the control value Bv_C and the photometric value Bv_R is within the dead zone. In this case, the control value Bv_C does not match (do not reach) the photometric value Bv_R, but the system control unit 120 does not change the control value Bv_C in order to suppress frequent exposure changes.

Next, as shown in FIG. 6, at the time of <2>, the system control unit 120 targets the photometric value Bv_R according to the difference between the photometric value Bv_R and the control value Bv_C exceeding the dead zone range. The control value Bv is changed as a value. Then, as shown in FIG. 6, at the time of <3>, the system control unit 120 changes the control value Bv_C in response to the difference between the photometric value Bv_R and the control value Bv_C falling within the dead zone again. (Fix the control value Bv_C).

As described above, when the moving image (through image) is acquired, the photometric value Bv_R and the control value Bv_C allow a difference of ΔBv at the maximum. With this configuration, it is possible to suppress frequent changes in brightness in the through image displayed on the image display unit 114.

However, in the moving image time-lapse mode, the image data used for generating the time-lapse moving image is determined from the through images constituting the moving image. Therefore, if the brightness of the image data deviates from the appropriate brightness by the difference between the photometric value Bv_R and the control value Bv_C, the brightness between frames in the generated time-lapse moving image changes unnaturally. It ends up.

Therefore, in the present embodiment, in the process of step S305, the brightness is corrected by exposing the acquired through image by an amount corresponding to the absolute value of the difference between the photometric value Bv_R and the control value Bv_C. Compensate for the deviation. In the present embodiment, exposure compensation is executed by adjusting the amount of digital gain after acquiring the image data.

For example, when displaying a through image on the image display unit 114, if the above-mentioned exposure compensation is executed every frame, the brightness of the displayed image may change frequently, so the exposure compensation is not executed. .. In other words, the camera 1 of the present embodiment has a configuration in which the above-mentioned exposure compensation is applied only to the image data for recording used for generating the time-lapse moving image, and even if the original data is the same, the image data for display. No exposure compensation is applied to.

Further, in the present embodiment, regarding the white balance adjustment, the white balance is adjusted so that the through image for display is included in the range of the time constant (dead zone based on the predetermined target value) with respect to the predetermined target value. Set the adjustment amount. On the other hand, the image data for recording used for generating the time-lapse moving image is controlled by the image processing circuit (white balance control means) 107 so that the white balance adjustment amount becomes the predetermined target value described above. To.

In this embodiment, with respect to the image data determined to be an appropriate image, the display image of the through image is subjected to exposure compensation and white balance adjustment, and the display image and the recorded image used for generating the time-lapse moving image are shared. However, it is not limited to this. For example, the image data used to generate the time-lapse moving image may not be used for display, and the through image acquired immediately before may be displayed instead.

Returning to FIG. 3, in step S306, the system control unit 120 executes a recording process on the image data after the exposure compensation, and records the image as an image used for generating the time-lapse moving image in the image recording memory 113 and the external recording medium 200.

Next, in step S307, the system control unit 120 determines whether or not the number of imagings (or imaging time) has reached the preset total number of imagings (or total imaging time). Then, when the system control unit 120 determines that the number of times of imaging has not reached the total number of times of imaging, the process returns to the process of step S301. When the system control unit 120 determines that the number of times of imaging has reached the total number of times of imaging, the current imaging process is terminated and the state shifts to the standby state. The above is the imaging process in the moving image time-lapse mode according to the present embodiment.

The plurality of image data used for generating the time-lapse moving image acquired by the above-described configuration are connected in the acquisition order (imaging order) by the image synthesizing unit 122, and one time-lapse moving image is generated. In the present embodiment, image data for a time-lapse movie is extracted from the through image, and the extracted image data is joined to generate a time-lapse movie, but the present embodiment is not limited to this. For example, a time-lapse moving image may be generated by extracting images that are not used for generating the time-lapse moving image from the through images and then connecting (compressing) the remaining images.

With the configuration described above, it is possible to select an image used for generating a time-lapse moving image from the through images constituting the moving image while avoiding an image unsuitable for generating the time-lapse moving image. Therefore, the camera 1 of the present embodiment suppresses unnatural changes in the shape, brightness, and color appearance of the subject portion due to rolling shutter distortion, exposure difference, and white balance in the generated time-lapse moving image. can do.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these, and various modifications and modifications can be made within the scope of the gist thereof. For example, in the above-described embodiment, the time-lapse moving image is generated inside the camera 1 in the time-lapse mode, but the present invention is not limited to this. That is, the camera 1 may be configured to acquire an image used for generating the time-lapse moving image and generate the time-lapse moving image on an external device or a computer network.

Further, in addition to the above-described embodiment, when it is determined that the image data acquired for each imaging interval is not an appropriate image, a configuration is adopted in which image data for interpolation is acquired instead of the image data. May be good. For example, after determining that the image is not appropriate, the system control unit 120 determines whether or not the image data is an appropriate image each time a through image is acquired, regardless of the time measurement by the built-in timer. Then, the image data determined to be an appropriate image is selected as an image to be used for generating the time-lapse moving image. At this time, the time measurement by the built-in timer may be reset, or the time measurement according to the initial imaging interval may be continued.

Further, in addition to the above-described embodiment, in the time-lapse mode, it is controlled so that the photometric calculation with a large degree of weighting for a specific subject (face, human body, etc.) existing in the angle of view is not performed. For example, at the time of normal subject imaging, it is common to calculate the photometric value (luminance value) by increasing the degree of weighting of photometry for a subject having a high probability of being the main subject such as a face or a human body. On the other hand, in a time-lapse movie, the time interval between frames is longer than that of a normal moving image, so if the photometric value changes depending on the presence or absence of a face or human body, the brightness will be unclear between the frames in the time-lapse movie. It may change naturally.

Therefore, as described above, in the time-lapse mode, this problem is solved by controlling the photometric result so as not to change for each specific subject.

Further, in the above-described embodiment, the acquired image data is an image suitable for generating a time-lapse moving image based on the presence / absence of blurring or moving objects generated in the camera 1, the presence / absence of exposure control, the exposure difference between the target value and the control value, and the like. It was determined whether it was data or not, but it is not limited to this. For example, the system control unit 120 may be configured to execute the appropriate image determination process based on the information regarding the focal length acquired from the lens unit 300. Specifically, when the focal length of the lens unit 300 is long, there is a high possibility that a subject relatively distant from the camera 1 will be the main subject as the user's intention. In this case, even if the amount of blurring of the camera 1 is relatively small, there is a high probability that the subject portion in the image data will change unnaturally due to the influence of the rolling shutter distortion. Therefore, the system control unit 120 determines that the image data acquired (captured) when the focal length based on the lens information acquired from the lens unit 300 is larger than a predetermined distance is an improper image.

Further, in the above-described embodiment, the configuration may be such that the still image time-lapse mode and the moving image time-lapse mode are automatically switched according to a predetermined condition. For example, when a moving object exists within the angle of view, or when there is a high probability that a moving object exists, a moving image time-lapse mode in which the influence of rolling shutter distortion is small is set by thinning out reading. Similarly, the moving image time-lapse mode is set when the camera 1 is shaken or the probability of the shake is high.

On the other hand, when there is no moving object within the angle of view or there is a high probability that the moving object does not exist, a still image time-lapse mode is set in which higher pixel image data can be acquired. Similarly, when there is no blurring of the camera 1 or there is a high probability that no blurring will occur, the still image time-lapse mode is set.

For example, the moving image time-lapse mode is set when the amount of blur generated in the camera 1 is equal to or greater than the predetermined amount of blur, and the still image time-lapse mode is set when the amount of blur is smaller than the predetermined amount of blur. Further, the moving image time-lapse mode is set when the moving amount of the subject within the angle of view is equal to or more than the predetermined moving amount, and the still image time-lapse mode is set when the moving amount is smaller than the predetermined moving amount.

In addition, the above-mentioned determination may be executed based on the imaging mode. For example, the system control unit (mode setting means) 120 sets the moving image time-lapse mode because it is considered that the movement of the subject is large when the sports mode is set as the imaging mode. Further, when the landscape mode is set as the imaging mode, the system control unit (mode setting means) 120 considers that the movement of the subject is small and sets the still image time-lapse mode.

Further, the above-mentioned determination may be executed based on the focal length of the lens unit 300. For example, when the focal length is larger than a predetermined distance, the system control unit 120 determines that the influence of the camera shake is large, and sets the moving image time-lapse mode. Further, when the focal length is equal to or less than a predetermined distance, the system control unit 120 determines that the influence of the blur of the camera 1 is small, and sets the still image time-lapse mode. By adopting the above configuration, it is possible to generate a time-lapse movie in which the shape, brightness, and color appearance of the subject portion are suppressed from being unnaturally changed, and to acquire a high-quality time-lapse movie depending on the situation.

Further, in the above-described embodiment, each unit constituting the camera 1, such as the image processing circuit 107, the memory control circuit 110, and the system control unit 120, operates in cooperation with each other to control the operation of the camera 1. However, it is not limited to this. For example, a (computer) program according to the flow shown in FIGS. 2, 3 and 5 described above is stored in the main memory 121 in advance. Then, the system control unit 120 including the microcomputer may execute the program to control the operation of the camera 1.

Further, as long as it has a program function, the form of the program such as object code, a program executed by an interpreter, script data supplied to the OS, etc. does not matter. Further, the recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, or an optical / optical magnetic recording medium.

Further, in the above-described embodiment, the digital camera has been described as an example of the image pickup apparatus for carrying out the present invention, but the present invention is not limited thereto. For example, a configuration may employ an imaging device other than a digital camera, such as a portable device such as a digital video camera or a smartphone, a wearable terminal, or a security camera.

(Other embodiments)
The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1 Digital camera 100 Camera unit 113 Image recording memory 120 System control unit 200 External recording medium 300 Lens unit

Claims (13)

A second image to be used for generating a time-lapse moving image is determined from a plurality of images constituting a moving image acquired by using the imaging means in order to acquire a time-lapse moving image obtained by joining a plurality of images together with an imaging means. An image pickup device that can set the time-lapse mode of 1.
In the first time-lapse mode, a setting means for setting a first interval for acquiring an image used for generating the time-lapse moving image from a plurality of images constituting the moving image, and
In the first time-lapse mode, among the plurality of images constituting the moving image, whether or not the image acquired at the first interval set by the setting means is an image suitable for generating a time-lapse moving image. Judgment means to judge
A determinant that determines the image used to generate a time-lapse movie,
Have,
When the determination means determines in the first time-lapse mode that the first image obtained according to the first interval is an image suitable for generating the time-lapse moving image , the determination means is concerned. When the first image is determined as an image used for generating the time-lapse movie and the determination means determines that the first image is not an image suitable for generating the time-lapse movie, the interval is different from the first interval. An image pickup apparatus characterized in that a second image obtained at a timing is determined as an image used for generating a time-lapse moving image. The determination means is characterized in that an image acquired by taking an image when the amount of blurring of the imaging device is equal to or less than a predetermined amount of blurring is determined to be an image suitable for generating the time-lapse moving image. Item 1. The imaging device according to item 1. The determination means is characterized in that an image acquired by taking an image when the movement amount of the subject within the angle of view is equal to or less than a predetermined movement amount is determined to be an image suitable for generating the time-lapse moving image. The imaging device according to claim 1 or 2. A diaphragm for adjusting the amount of light incident on the imaging means is provided.
Claims 1 to 3 are characterized in that the determination means determines that an image acquired by taking an image when the aperture diameter of the diaphragm has not changed is an image suitable for generating the time-lapse moving image. The image pickup apparatus according to any one of the above. A photometric means that measures the subject at predetermined intervals when acquiring a moving image,
It has an exposure control means that sets a target value of exposure control for each frame based on the photometric result of the photometric means at the time of acquiring a moving image.
Any of claims 1 to 4, wherein the determination means determines that an image in which the difference between the photometric result and the target value is smaller than a predetermined value is an image suitable for generating the time-lapse moving image. The imaging apparatus according to paragraph 1. A display means capable of displaying an image acquired by using the image pickup means and a display means.
It has an exposure compensation means for compensating for an exposure difference between the photometric result and the target value.
The exposure compensation means does not correct the exposure difference between the photometric result and the target value for the image to be displayed on the display means, and the photometric result for the image used for generating the time-lapse moving image. The image pickup apparatus according to claim 5, wherein the exposure difference between the image and the target value is corrected. The determination means determines that an image in which the difference between the photometric result and the target value is smaller than the value corresponding to the correction amount that can be corrected by the exposure compensation means is an image suitable for generating the time-lapse moving image. The image pickup apparatus according to claim 6. It has a white balance control means that controls the amount of white balance adjustment.
In the first time-lapse mode, the white balance control means controls the amount of white balance adjustment so that the image displayed on the display means is included in a predetermined range based on a predetermined target value. The imaging device according to claim 6 or 7, wherein the adjustment amount of the white balance is controlled so as to be the target value for the image used for generating the time-lapse moving image. The imaging device acquires a plurality of images used for generating a time-lapse moving image acquired by intermittently imaging a subject using the imaging means in order to acquire a time-lapse moving image using the imaging means. Time-lapse mode can be set,
The first time-lapse mode is a mode in which the number of pixels of the image to be acquired is smaller than that of the second time-lapse mode.
A claim comprising a mode setting means for controlling switching between the first time-lapse mode and the second time-lapse mode based on the amount of blur generated in the image pickup apparatus or the amount of movement of the subject within the angle of view. Item 2. The imaging device according to any one of Items 1 to 8. The mode setting means sets the first time-lapse mode when the amount of blurring of the imaging device is equal to or greater than a predetermined amount of blurring, and when the amount of blurring of the imaging device is smaller than the predetermined amount of blurring, the mode setting means said. The imaging device according to claim 9, wherein a second time-lapse mode is set. The mode setting means sets the first time-lapse mode when the movement amount of the subject within the angle of view is equal to or greater than the predetermined movement amount, and the movement amount of the subject within the angle of view is larger than the predetermined movement amount. The imaging device according to claim 9, wherein the second time-lapse mode is set when the size is small. When determining an image to be used for generating a time-lapse moving image from a plurality of images constituting a moving image acquired by using the imaging means in order to obtain a time-lapse moving image in which a plurality of images are joined together with an imaging means. It is a control method of the image pickup device of
A setting step of setting a first interval for acquiring an image used for generating the time-lapse moving image from a plurality of images constituting the moving image, and a setting step.
A determination step of determining whether or not the image acquired at the first interval set in the setting step among the plurality of images constituting the moving image is an image suitable for generating a time-lapse moving image,
The decision process to determine the image used to generate the time-lapse movie, and
Have,
In the determination step, when the first image obtained in accordance with said first interval is determined by said determination step that the image suitable for the generation of time-lapse video data the first image When it is determined as an image to be used for generating an immuraps moving image and it is determined in the determination step that the first image is not an image suitable for generating the time-lapse moving image, it is obtained at a timing different from the first interval. A method for controlling an image pickup apparatus, which comprises determining a second image as an image used for generating a time-lapse moving image . A computer-readable program for causing a computer to execute the control method of the imaging device according to claim 12.

Images