JP6452392B2 - IMAGING DEVICE, ITS CONTROL METHOD, PROGRAM, AND STORAGE MEDIUM - Google Patents

Description

  The present invention relates to a high dynamic range (HDR) composition technique for moving images.

  A high dynamic range (HDR) composition technique is known in which a plurality of images taken at different exposures (under exposure, appropriate exposure, and over exposure) are combined to expand the dynamic range of the images. In this HDR composition, by combining appropriate exposure portions in a plurality of images taken with different exposures, whiteout and blackout are suppressed, and an image having good gradation from dark to bright is generated. be able to. This HDR synthesis can be applied not only when shooting a still image but also when recording a moving image. In HDR synthesis of moving images, for example, different exposures (appropriate exposure and underexposure) can be repeatedly captured at a frame rate of 60 fps, and an HDR moving image in which an HDR frame with a frame rate of 30 fps is generated can be recorded.

  When HDR synthesis of a moving image is performed by a camera, various interrupt processing related to moving image recording must be considered in addition to the above-described imaging control. An example of the interrupt process is exposure control. When recording a moving image, different from when shooting a still image, the appropriate exposure gradually changes during recording. Therefore, it is necessary to perform exposure control and moving image recording in parallel. When using a lens that has a restriction that instantaneously drives the aperture, if the aperture is driven during moving image recording, an abrupt luminance change occurs in the image.

  Therefore, when an image inappropriate for a frame is generated by such interrupt processing, a technique called frame stop is known in which frame acquisition is stopped until an image with an appropriate exposure is obtained (Patent Document 1). . According to Patent Document 1, an image obtained by HDR synthesis can be displayed without failure.

JP 2011-097366 A

  In Patent Document 1, when any of the images for configuring the HDR frame becomes unavailable due to the interrupt process, it is not possible to display the HDR frame synthesized with the set including the frame of the image. The period during which images are not updated becomes longer.

  The present invention has been made in view of the above problems, and the object thereof is to make it possible to record a moving image while shortening the frame stop period and maintaining continuity between frames even when a frame stop occurs in HDR synthesis of a moving image. It is to realize technology.

  In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention includes an imaging unit that repeatedly captures a plurality of frames with different exposures, and a composite frame obtained by combining the plurality of frames with different exposures. A combining unit to generate; a determining unit that determines whether there is an unusable frame that cannot be used as a combined frame in the plurality of frames; and when there is the unusable frame, the usable frame before and after the unusable frame can be used. Control means for controlling the synthesizing means so as to perform a first process of generating a synthesized frame using the frame.

  According to the present invention, even when a frame stop occurs in HDR synthesis of a moving image, it is possible to record a moving image while shortening the frame stop period and maintaining continuity between frames.

The block diagram which shows the outline of the apparatus structure of this embodiment. Explanatory drawing of the HDR synthetic | combination process of a moving image. The figure explaining the process at the time of the frame stop at the time of the moving image recording by HDR synthetic | combination. FIG. 6 is a diagram for explaining HDR synthesis processing at the time of frame stop according to the first embodiment. 5 is a flowchart showing HDR frame generation processing at the time of moving image recording in FIG. 4. The figure explaining the motion of the to-be-photographed object in an HDR frame at the time of a frame stop. The figure explaining the HDR synthetic | combination process (when changing a synthetic | combination ratio) at the time of the frame stop of Embodiment 2. FIG. 8 is a flowchart showing HDR frame generation processing at the time of moving image recording in FIG. 7. The figure explaining the HDR synthetic | combination process (when using a suitable frame) at the time of the frame stop of Embodiment 2. FIG. 10 is a flowchart showing HDR frame generation processing at the time of moving image recording in FIG. 9. The figure explaining the HDR synthetic | combination process (when changing a synthetic | combination ratio) at the time of the frame stop of Embodiment 2. 12 is a flowchart showing HDR frame generation processing at the time of moving image recording in FIG. 11. The figure explaining a mode that an unnatural HDR frame is produced | generated by the combination of a flame | frame. FIG. 10 is a diagram for explaining HDR synthesis processing at the time of frame stop according to the third embodiment. The flowchart which shows the HDR frame production | generation process at the time of the moving image recording of FIG.

  Hereinafter, embodiments for carrying out the present invention will be described in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment. Moreover, you may comprise combining suitably one part of each embodiment mentioned later.

  In the following, an example in which the imaging apparatus according to the present invention is applied to a digital camera capable of still image recording and moving image recording will be described. However, the present invention is not limited to a device mainly intended for imaging such as a digital camera. Needless to say. In other words, the present invention can be applied to any device in which an imaging device is built-in or externally connected, such as a smartphone, a personal computer (laptop type, desktop type, tablet type, etc.), a game machine, etc. is there. Therefore, the “imaging device” in this specification is intended to include any device that has an imaging function but is not primarily intended for imaging.

  <Apparatus Configuration> First, with reference to FIG. 1, an outline of the configuration and functions of the imaging apparatus 101 of this embodiment will be described.

  In FIG. 1, a lens 102 collects a light image of a subject and forms it on the imaging surface of the image sensor 104. The diaphragm 103 adjusts the amount of incident light of the optical image of the subject. The image sensor 104 is an image sensor using a CCD, CMOS, or the like in which a plurality of pixels (storage type photoelectric conversion elements) are arranged, and photoelectrically converts a subject image formed by the lens 102.

  The image processing unit 105 performs various image processing such as noise removal, gamma processing, interpolation processing, matrix conversion, and composition processing on the image data obtained by the image sensor 104, and generates an image of a frame constituting a moving image. To do.

  A diaphragm driving unit 109 drives the diaphragm 103. The exposure control unit 110 controls the time (charge accumulation time or exposure time) during which the image sensor 104 accumulates light. Further, the exposure control unit 110 sets the aperture amount and controls the aperture drive unit 109.

  The imaging control unit 111 controls the operation timing of each unit by issuing operation commands to the image sensor 104, the image processing unit 105, the frame recording unit 106, the composition processing unit 107, the moving image recording unit 108, and the exposure control unit 110. When recording a moving image by HDR synthesis, the shooting control unit 111 controls each unit so as to alternately repeat imaging at different exposures (appropriate exposure and underexposure) at a predetermined frame rate. The image obtained in this way includes at least a set of a frame of an appropriate image (appropriate frame) captured with appropriate exposure and a frame of an under image (underframe) imaged with underexposure. In the following description, the appropriate frame and the under frame may be simply referred to as “appropriate” and “under”.

  In addition, the imaging control unit 111 has a function of detecting a moving object that is a subject and an amount of movement thereof from the frame image generated by the image processing unit 105. There are various moving object detection methods. For example, there is a method of taking the absolute difference between two frames. A part having a difference can be estimated as a moving object, and it can be determined that the moving amount of the moving object is larger as the difference amount is larger.

  The frame recording unit 106 is a memory that records a frame for HDR synthesis (recording frame) generated by the image processing unit 105 during moving image recording.

  The composition processing unit 107 reads out a recording frame for performing HDR composition from the frame recording unit 106 and performs HDR composition.

  The moving image recording unit 108 is a storage device such as a built-in hard disk or semiconductor memory that records the combined frame (HDR frame) generated by the combining processing unit 107 as a moving image.

  The operation unit 112 is a switch that accepts user operations, and performs a still image shooting or moving image recording start instruction and other instructions, input, and the like.

  The imaging control unit 111 has a function of detecting a moving object that is a subject and a moving amount thereof from a frame image generated by the image processing unit 105 and recorded in the frame recording unit 106. In the moving object detection method, for example, the absolute value of a difference between two frames is taken, and a portion where there is a difference can be estimated as a moving object.

  Note that the imaging apparatus 101 according to the present embodiment includes a display unit including an LCD panel, an organic EL panel, and the like, and is used for displaying an image captured by the imaging element 104 and reproducing a moving image read from the moving image recording unit 108. .

  <Operation During Recording of Movie by HDR Compositing> First, an HDR frame generation operation of the imaging apparatus 101 at the time of moving image recording by HDR combining will be described with reference to FIG. 1 and FIG.

  When an instruction to start moving image recording by HDR synthesis is issued from the operation unit 112, the shooting control unit 111 performs necessary initialization processing and starts shooting. When an imaging instruction for an appropriate frame is issued from the imaging control unit 111, the exposure control unit 110 calculates an appropriate exposure control value, sets the aperture amount in the aperture drive unit 109, and sets the shutter speed for the image sensor 104. And set the sensitivity. When an imaging instruction is issued from the imaging control unit 111, the light is condensed by the lens 102, the light amount and the depth of field are adjusted by the diaphragm 103, and an image formed on the image sensor 104 is captured to obtain appropriate image data. Is acquired. The appropriate image read from the image sensor 104 is subjected to image processing necessary for making a recording frame in the image processing unit 105, and becomes an appropriate frame. The appropriate frame is recorded in the frame recording unit 106. When the recording of the appropriate frame is completed, the imaging control unit 111 issues an underframe imaging instruction. The exposure control unit 110 calculates an exposure control value so that underexposure is performed, and sets the aperture 103 and the image sensor 104. When an imaging instruction is issued from the imaging control unit 111, the light is condensed by the lens 102, the light amount and the depth of field are adjusted by the aperture 103, an image formed on the imaging element 104 is captured, and the under image data Is acquired. The under image read from the image sensor 104 is subjected to image processing necessary for the recording frame by the image processing unit 105 to become an under frame. The underframe is recorded in the frame recording unit 106. When shooting of the appropriate frame and the under frame is completed, the shooting control unit 111 reads both frames from the frame recording unit 106 with the combining processing unit 107 and combines them as an HDR frame. The HDR frame is stored as a moving image in the moving image recording unit 108.

  FIG. 2 conceptually shows an operation of acquiring an appropriate frame and an under frame alternately and generating an HDR frame by repeatedly performing the above-described operation.

  <Operation at Frame Stop> Next, the HDR frame generation operation of the image pickup apparatus 101 at the frame stop will be described with reference to FIGS.

  In the moving image recording operation of FIG. 2, the exposure control unit 110 sets the aperture amount, and the aperture drive unit 109 drives the aperture 103 so that the aperture amount is set. Here, in the case of an interchangeable lens type digital camera, there is a camera that can drive the aperture 103 only rapidly depending on the restrictions. Since the exposure control unit 110 determines the exposure control value of the next frame based on the image data acquired by the image sensor 104, when the aperture value changes rapidly, the brightness of the image flickers due to the bias of the aperture between frames. May look like this. In this case, the imaging control unit 111 determines that an image captured in a state where the luminance difference between frames is not unnatural is determined as an unusable frame, stops updating the image during that period, and replaces the immediately previous usable frame as a recording frame. Process.

  In the present embodiment, an unusable frame is generated by such various interrupt processes, and the frame update is stopped by replacing it with the immediately previous usable frame is called a frame stop. It is not known when and in what interval the frame stop occurs due to lens restrictions and subject brightness changes.

  FIG. 3 conceptually shows a frame stop operation at the time of moving image recording by HDR synthesis in FIG. 2, and only the under frame or only the appropriate frame may be an unusable frame among the set of the appropriate frame and the under frame. .

  [Embodiment 1] The HDR synthesizing process at the time of frame stop according to Embodiment 1 will be described below with reference to FIGS.

  In FIG. 4, in the steady state other than the frame stop time, the appropriate frame T101 and the under frame U101 are obtained alternately to generate one HDR frame H101. However, when only the under frame U102 is an unusable frame in the appropriate / under pair (T102 / U102), the HDR frame H102 is generated using the previous underframe U101. If only the appropriate frame T103 is an unusable frame in the appropriate / under pair (T103 / U103), the immediately subsequent appropriate frame T104 is used to generate the HDR frame H104 and replace it with the immediately preceding HDR frame H103. .

  Here, with reference to FIG. 5, the HDR frame generation process at the time of moving image recording of Embodiment 1 is demonstrated based on the example of FIG. 5 is started in response to an instruction to start moving image recording by HDR synthesis from the operation unit 112, and is realized by the imaging control unit 111 executing a predetermined control program and controlling each unit.

  In step S101, the synthesis processing unit 107 synthesizes the HDR frame H101 with an appropriate / under combination (T101 / U101).

  In step S102, the imaging control unit 111 determines whether an unusable frame is included in either appropriate or under, or both. As a result of the determination, if there is no unusable frame, the process returns to step S101 to synthesize the next appropriate / under pair (T102 / U102). If there is an unusable frame, the process proceeds to step S103.

  In step S103, the imaging control unit 111 determines whether the unusable frame starts from the appropriate frame T102. If the unusable frame starts from the proper frame, the process proceeds to step S106. If the unusable frame is only the underframe, the shooting control unit 111 proceeds to step S104. move on.

  In step S104, the imaging control unit 111 determines whether the previous underframe U101 is an unusable frame. If it is an unusable frame, the process proceeds to step S106. If it is an unusable frame, step S105 is performed. Proceed to

  In step S105, the composition processing unit 107 generates an HDR frame H102 using the previous underframe U101, and proceeds to step S107.

  In step S106, the imaging control unit 111 performs frame stop and reuses the immediately previous HDR frame H101.

  In step S107, the imaging control unit 111 determines whether or not a pair can be formed using an appropriate / under (T104 / U104) or under / suitable next (U103 / T104) usable frame. As a result of the determination, if any pair can be created, the process proceeds to step S108. If a pair cannot be created, a frame stop (S106) is performed, and the HDR frame H102 immediately before the frame stop is reused.

  In step S108, the imaging control unit 111 determines whether the set is appropriate / under (T104 / U104) or under / suitable after one (U103 / T104), and the first available frame is appropriate. Determine if it is a frame. As a result of the determination, in the case of a suitable / under combination (T104 / U104), the process returns to step S101 and the HDR frame is synthesized in a stationary state. The process proceeds to step S109.

  In step S109, the imaging control unit 111 combines the HDR frame H104 with a combination of under and one after (U103 / T104), and the process proceeds to step S110.

  In step S110, the imaging control unit 111 replaces the last frame H103 of the frame stop with the HDR frame H104 synthesized in step S109.

  Thereafter, in step S111, the imaging control unit 111 repeats the processing from step S101 until there is an instruction to end moving image recording from the operation unit 112.

  According to the first embodiment described above, when performing moving image recording using HDR synthesis in a situation where unusable frames occur irregularly, if there are suitable or under frames that can be used before and after the frame stop, An HDR frame is generated using the available frame. In this way, a maximum of 2 frames can be interpolated. As a result, even when a frame stop occurs in moving image recording using HDR synthesis, it is possible to shorten the frame stop period and perform moving image recording with continuity between frames.

  [Embodiment 2] Next, the HDR synthesizing process at the time of frame stop according to Embodiment 2 will be described with reference to FIGS.

  FIG. 6 shows a change of a moving object as a subject between frames at the time of moving image recording in which an appropriate frame and an under frame are alternately obtained to generate an HDR frame. In FIG. 6, it is assumed that the moving object P201 in the under image is the moving object P201 in the HDR frame. In this situation, when the underframe U201 and the appropriate frame T201 become unavailable in a certain frame, the frame is stopped, and the moving object P201 jumps to the position of the moving object P203.

  Here, with reference to FIG. 7, the HDR synthesizing process by switching the synthesizing ratio at the time of frame stop according to the second embodiment will be described.

  In FIG. 7, in a steady state other than the frame stop time, the appropriate frame T301 and the under frame U301 are alternately obtained to generate one HDR frame H301. Further, when only the under frame U302 is an unusable frame in the appropriate / under combination (T302 / U302), the HDR frame H302 is generated using the previous underframe U301. Here, in the present embodiment, the HDR frame H302 in which the moving object P301 is displayed in a multiplexed manner is generated by switching the composition ratio of the moving object in the appropriate image and the previous under image with respect to the normal time. To determine whether to switch the composition ratio, use the brightness values of the appropriate frame and underframe to determine whether the moving object is whiteout or blacked out. Only one side is prevented from disappearing. Also, if only the appropriate frame T303 is an unusable frame among the appropriate / under pairs (T303 / U303), the HDR frame H304 is generated using the immediately subsequent appropriate frame T304 and replaced with the immediately preceding HDR frame H403. .

  Hereinafter, with reference to FIG. 8, the HDR frame generation processing during moving image recording according to the second embodiment will be described based on the example of FIG. 8 is started in response to an instruction to start moving image recording by HDR synthesis from the operation unit 112, and is realized by the imaging control unit 111 executing a predetermined control program and controlling each unit.

  In step S201, the synthesis processing unit 107 synthesizes the HDR frame H301 with an appropriate / under combination (T301 / U301).

  In step S202, the imaging control unit 111 determines whether an unusable frame is included in either appropriate or under, or both. If there is no unusable frame as a result of the determination, the process returns to step S201 to synthesize the next appropriate / under pair (T302 / U302). If there is an unusable frame, the process proceeds to step S203.

  In step S203, the imaging control unit 111 determines whether the unusable frame starts from the proper frame T302. If the unusable frame starts from the proper frame, the process proceeds to step S209. If the unusable frame is only the underframe, the shooting control unit 111 proceeds to step S204. move on.

  In step S204, the imaging control unit 111 determines whether the previous underframe U301 is an available frame. If it is determined that the frame is usable, the process proceeds to step S205. If the frame is unusable, the process proceeds to step S209.

  In step S205, the imaging control unit 111 determines whether the luminance of the appropriate frame T302 is equal to or lower than the threshold value, and determines whether the moving object is over-exposed. As a result of the determination, if it is equal to or less than the threshold value, it is determined that no whiteout has occurred, and the process proceeds to step S206.

  In step S206, the imaging control unit 111 determines whether the luminance of the previous underframe U301 is equal to or higher than a threshold value, and determines whether the moving object is blacked out. As a result of the determination, if it is equal to or greater than the threshold value, it is determined that black is not crushed, and the process proceeds to step S207.

  In steps S205 and S206, the imaging control unit 111 performs a frame stop when any one of the whiteout of the subject in the appropriate frame T302 and the blackout of the subject in the underframe U301 is detected (S209).

  In step S207, the imaging control unit 111 switches the setting of the composition ratio of the moving object P301 (for example, appropriate 50%, under 50%).

  In step S208, the composition processing unit 107 generates an HDR frame H302 using the previous underframe U301, and proceeds to step S210.

  In step S209, the imaging control unit 111 performs frame stop. That is, when the previous underframe U301 is an available frame (S204), it is determined whether the luminance of the appropriate frame T302 is equal to or lower than a threshold value, and it is determined whether the moving object is not over-exposed (S205). In addition, it is determined whether the luminance of the previous underframe U301 is equal to or higher than a threshold value, and it is determined whether the moving object is blackened (S206). As a result of the determination, if any one of appropriate and under is not within the predetermined luminance range, the frame is stopped (S209).

  In step S209, the imaging control unit 111 performs frame stop and reuses the immediately preceding HDR frame H301.

  In step S <b> 210, the imaging control unit 111 determines whether a pair can be created with an appropriate / under (T304 / U304) or under / suitable one (U303 / T304) usable frame. If any pair can be created, the process proceeds to step S211. If no pair can be created, frame stop (S209) is performed, and the HDR frame H302 immediately before the frame stop is reused.

  In step S211, the imaging control unit 111 determines whether the combination is appropriate / under (T304 / U304) or under / adequate (U303 / T304), and the first available frame is appropriate. Determine if it is a frame. As a result of the determination, in the case of the appropriate / under combination (T304 / U304), the process returns to step S201, and the HDR frame is synthesized in the normal state. The process proceeds to step S212.

  In step S212, the imaging control unit 111 synthesizes the HDR frame H304 with a combination of under and one after (U303 / T304), and proceeds to step S213.

  In step S213, the imaging control unit 111 replaces the last frame H303 of the frame stop with the HDR frame H304 synthesized in step S213. In FIG. 7, for example, when the unusable frame is not detected at the appropriate frame, that is, when the usable frame starts from the appropriate frame, the last frame is not replaced in step S213, Similarly, an HDR frame synthesized from an appropriate / under pair is used.

  Thereafter, in step S214, the imaging control unit 111 repeats the processing from step S201 until there is a moving image recording end instruction from the operation unit 112.

  Next, with reference to FIG. 9, the HDR synthesis process using an appropriate frame at the time of frame stop according to the second embodiment will be described.

  In FIG. 9, the appropriate frame T401 and the underframe U401 are alternately obtained in the normal state, and one HDR frame H401 is generated. In addition, when only the under frame U402 is an unusable frame in the appropriate / under pair (T402 / U402), the HDR frame H402 is generated using the previous underframe U401. If only the appropriate frame T403 is an unusable frame in the appropriate / under pair (T403 / U403), the HDR frame H404 is generated by using the immediately subsequent appropriate frame T404 and replaced with the immediately preceding HDR frame H403. . Here, in the present embodiment, the HDR frame H404 in which the moving object P401 is displayed in a multiplexed manner is generated by switching the composition ratio of the moving object in the appropriate frame T404 to 100%. To determine whether to switch the composition ratio, use the brightness values of the appropriate frame and underframe to determine whether the moving object is whiteout or blacked out. Only one side is prevented from disappearing.

  Hereinafter, with reference to FIG. 10, the HDR frame generation processing during moving image recording according to the second embodiment will be described based on the example of FIG. 10 is started in response to an instruction to start moving image recording by HDR synthesis from the operation unit 112, and is realized by the imaging control unit 111 executing a predetermined control program and controlling each unit.

  In step S301, the synthesis processing unit 107 synthesizes the HDR frame H401 with an appropriate / under combination (T401 / U401).

  In step S302, the imaging control unit 111 determines whether an unusable frame is included in either appropriate or under, or both. As a result of the determination, if there is no unusable frame, the process returns to step S301, and the next appropriate / under combination (T402 / U402) is synthesized. If there is an unusable frame, the process proceeds to step S303.

  In step S303, the imaging control unit 111 determines whether the unusable frame starts from the proper frame T402. If the unusable frame starts from the proper frame, the process proceeds to step S306. If the unusable frame is only an underframe, the shooting control unit 111 proceeds to step S304. move on.

  In step S304, the imaging control unit 111 determines whether the previous underframe U401 is an unusable frame. If it is an unusable frame, the process proceeds to step S306. If it is an unusable frame, step S305 is performed. Proceed to

  In step S305, the composition processing unit 107 generates an HDR frame H402 by using the moving object combining ratio 100% of the appropriate frame T402 and the previous underframe U401 (moving object combining ratio 0%), and the process proceeds to step S307. move on.

  In step S306, the imaging control unit 111 performs frame stop and reuses the immediately preceding HDR frame H401.

  In step S <b> 307, the imaging control unit 111 determines whether a pair can be created using the appropriate / under (T404 / U404) or under / appropriate (U403 / T404) available frames. As a result of the determination, if any pair can be created, the process proceeds to step S308. If a pair cannot be created, frame stop (S306) is performed, and the HDR frame H402 immediately before the frame stop is reused.

  In step S308, the imaging control unit 111 determines whether the set is appropriate / under (T404 / U404) or under / suitable after (U403 / T404), and the first available frame is appropriate. Determine if it is a frame. As a result of the determination, in the case of a suitable / under combination (T404 / U404), the process returns to step S301, where the HDR frame is synthesized in a stationary state, and in the case of an under / appropriate (U403 / T404) combination, The process proceeds to step S309.

  In step S309, the imaging control unit 111 determines whether the luminance of the appropriate frame T404 is equal to or lower than the threshold, and determines whether the moving object that is the subject is not overexposed. As a result of the determination, if it is equal to or less than the threshold value, it is determined that no whiteout has occurred, and the process proceeds to step S310. If it exceeds the threshold value, it is determined that whiteout has occurred and the process returns to step S301.

  In step S310, the imaging control unit 111 determines whether the luminance of the previous underframe U403 is equal to or higher than a threshold value, and determines whether the moving object that is the subject is blacked out. As a result of the determination, if it is equal to or greater than the threshold value, it is determined that the black is not crushed, and the process proceeds to step S311.

  In step S311, the imaging control unit 111 switches the setting of the composition ratio of the moving object P401 (for example, appropriate 50%, under 50%).

  In step S312, the composition processing unit 107 performs composition of the HDR frame H404 with a combination of under and one after (U403 / T404), and the process proceeds to step S313.

  In step S313, the imaging control unit 111 replaces the last frame H403 of the frame stop with the HDR frame H404 synthesized in step S312. In FIG. 9, for example, when an unusable frame is no longer detected at the appropriate frame, that is, when an available frame starts from an appropriate frame, the last frame is not replaced at step S313. Similarly, an HDR frame synthesized from an appropriate / under pair is used.

  Thereafter, in step S314, the imaging control unit 111 repeats the processing from step S301 until there is a moving image recording end instruction from the operation unit 112.

  Next, with reference to FIG. 11, an HDR synthesis process by exchanging the synthesis ratio at the time of frame stop according to the second embodiment will be described.

  In FIG. 11, the appropriate frame T501 and the underframe U501 are alternately acquired in the steady state, and one HDR frame H501 is generated. In the moving body P501 of the HDR frame H501, the moving bodies of the appropriate frame T501 and the underframe U501 are displayed in a multiplexed manner. In addition, when only the under frame U502 is an unusable frame in the appropriate / under pair (T502 / U502), the HDR frame H502 is generated using the previous underframe U501. Here, in the present embodiment, the composition ratio of the appropriate / under group in a steady state (appropriate 30%, under 70%) and the composition ratio of the appropriate / preceding under group of the HDR frame H502 (appropriate 70%). , Under 30%). For example, it is assumed that the composition ratio of the appropriate frame T501 in the moving object P501 of the HDR frame H501 at the normal time is 30% and the underframe U501 is 70%. In this case, the composition ratio of the appropriate frame T502 and the under frame U501 in the moving object P502 of the HDR frame H502 is set to 70% and 30%, respectively.

  In addition, when only the appropriate frame T503 is an unusable frame in the appropriate / under pair (T503 / U503), the HDR frame H504 is generated using the immediately subsequent appropriate frame T504, and the immediately preceding HDR frame H503 is replaced. . In this case as well, the combination ratio of the under / suitable pair is replaced with that in the steady state. For example, it is assumed that the composition ratio of the appropriate frame T504 in the moving object P505 of the HDR frame H505 in the normal state is 30% and the underframe U504 is 70%. In this case, the composition ratio of the appropriate frame T504 and the underframe U503 in the moving object P504 of the HDR frame H504 is set to 70% and 30%, respectively.

  To determine whether to switch the composition ratio, use the brightness values of the appropriate frame and underframe to determine whether the moving object is whiteout or blacked out. Only one side is prevented from disappearing.

  Hereinafter, with reference to FIG. 12, the HDR frame generation processing at the time of moving image recording according to the second embodiment will be described based on the example of FIG. 12 is started in response to an instruction to start moving image recording by HDR synthesis from the operation unit 112, and is realized by the imaging control unit 111 executing a predetermined control program and controlling each unit.

  In step S401, the synthesis processing unit 107 synthesizes the HDR frame H501 with an appropriate / under combination (T501 / U501).

  In step S402, the imaging control unit 111 determines whether an unusable frame is included in either appropriate or under, or both. If there is no unusable frame as a result of the determination, the process returns to step S401, and the next appropriate / under combination (T502 / U502) is synthesized. If there is an unusable frame, the process proceeds to step S403.

  In step S403, the imaging control unit 111 determines whether the unusable frame starts from the proper frame T502. If the unusable frame starts from the proper frame, the process proceeds to step S409. If the unusable frame is only an underframe, the shooting control unit 111 proceeds to step S404. move on.

  In step S404, the imaging control unit 111 determines whether the previous underframe U501 is an available frame. If it is an available frame, the process proceeds to step S405. If it is an unusable frame, step S409 is performed. Proceed to

  In step S405, the imaging control unit 111 determines whether the luminance of the appropriate frame T502 is equal to or lower than the threshold value, and determines whether the moving object that is the subject is not overexposed. As a result of the determination, if it is equal to or less than the threshold value, it is determined that no whiteout has occurred, and the process proceeds to step S406. If it exceeds the threshold value, it is determined that whiteout has occurred, and the process proceeds to step S409.

  In step S406, the imaging control unit 111 determines whether the luminance of the previous underframe U501 is equal to or higher than a threshold value, and determines whether the moving object that is the subject is blacked out. As a result of the determination, if it is equal to or greater than the threshold value, it is determined that black is not crushed, and the process proceeds to step S407. If it is less than the threshold value, it is determined that black is crushed, and the process proceeds to step S409.

  In step S407, the imaging control unit 111 switches the setting of the composition ratio of the moving object P502 (for example, appropriate 70%, under 30%).

  In step S408, the composition processing unit 107 generates an HDR frame H502 using the previous underframe U501, and proceeds to step S410.

  In step S409, the imaging control unit 111 performs frame stop and reuses the immediately preceding HDR frame H502.

  In step S410, the imaging control unit 111 determines whether or not a pair can be created using the appropriate / under (T504 / U504) or under / appropriate (U503 / T504) available frames. As a result of the determination, if any pair can be created, the process proceeds to step S411. If a pair cannot be created, frame stop (S409) is performed, and the HDR frame H502 immediately before the frame stop is reused.

  In step S411, the imaging control unit 111 determines whether the combination is appropriate / under (T504 / U504) or under / adequate (U503 / T504), and the first frame of the available frames is appropriate. Determine if it is a frame. As a result of the determination, in the case of an appropriate / under combination (T504 / U504), the process returns to step S401, and the HDR frame is synthesized in a stationary state. In the case of an under / appropriate (U503 / T504) combination The process proceeds to step S412.

  In step S412, the imaging control unit 111 determines whether the luminance of the appropriate frame T504 is equal to or lower than the threshold value, and determines whether the moving object that is the subject is not overexposed. As a result of the determination, if it is equal to or less than the threshold value, it is determined that no whiteout has occurred, and the process proceeds to step S413. If it exceeds the threshold value, it is determined that whiteout has occurred and the process returns to step S401.

  In step S413, the imaging control unit 111 determines whether the luminance of the previous underframe U503 is greater than or equal to a threshold value, and determines whether the moving object that is the subject is blacked out. As a result of the determination, if it is equal to or greater than the threshold value, it is determined that black is not crushed, and the process proceeds to step S414.

  In step S414, the imaging control unit 111 switches the setting of the composition ratio of the moving object P503 (for example, appropriate 70%, under 30%).

  In step S415, the composition processing unit 107 performs composition of the HDR frame H504 with a combination of under and one after (U503 / T504), and the process proceeds to step S416.

  In step S416, the imaging control unit 111 replaces the last frame H503 of the frame stop with the HDR frame H504 synthesized in step S415. In FIG. 11, for example, when the unusable frame is not detected at the appropriate frame, that is, when the usable frame starts from the suitable frame, the last frame is not replaced in step S416. Similarly, an HDR frame synthesized from an appropriate / under pair is used.

  Thereafter, in step S417, the imaging control unit 111 repeats the processing from step S401 until there is a moving image recording end instruction from the operation unit 112.

  According to the second embodiment described above, in addition to the effects of the first embodiment, when an unusable frame is detected and interpolation is performed on the usable frames before and after the unusable frame, the moving object is displayed in a multiple display and combined. Adjust the ratio. In this way, the moving object can be displayed smoothly so that it cannot be seen by jumping at the frame stop. As a result, even when a frame stop occurs in moving image recording using HDR synthesis, it is possible to shorten the frame stop period and perform moving image recording with continuity between frames.

  [Third Embodiment] Next, the HDR synthesizing process at the time of frame stop according to the third embodiment will be described with reference to FIGS.

  In the present embodiment, as shown in FIG. 13, an HDR frame in which appropriate frames and underframes are alternately obtained and a moving object is displayed in a multiplexed manner during moving image recording is generated. Then, if any one of the appropriate / under pairs is an unusable frame, an HDR frame is generated using the immediately available frame. That is, when the underframe U602 is an unusable frame in the appropriate / under pair (T602 / U602), the HDR frame H602 is generated using the immediately available underframe U601. Further, when the appropriate frame T603 is an unusable frame in the appropriate / under pair (T602 / U602), the HDR frame H603 is generated using the immediately preceding available appropriate frame T602.

  However, if the same under frame U601 is used in the synthesis of the HDR frame H601 and the HDR frame H602 that are temporally before and after, an unnatural frame in which the moving object P601 and the moving object P602 are stopped between frames is generated. In addition, when the same appropriate frame T602 is used in the HDR frame H602 and the HDR frame H604, depending on the moving amount of the moving object, the multiplicity of the moving object P603 is significantly different from the HDR frames H601 and H602 so far, and an unnatural frame is generated. Will be generated.

  Therefore, in the present embodiment, HDR synthesis processing according to the moving amount of the moving object is performed so as to suppress the generation of such an unnatural frame.

  With reference to FIG. 14, the HDR synthesis process at the time of frame stop according to the fourth embodiment will be described.

  In FIG. 14, in the steady state other than the frame stop time, the appropriate frame T701 and the under frame U701 are obtained alternately to generate one HDR frame H701. Here, when only the under frame U702 is an unusable frame in the appropriate / under pair (T702 / U702), the moving object is detected by using the two previous underframes U700 and U701. I do. When the moving amount of the moving object is equal to or smaller than the threshold, the HDR frame H702 is generated using one or more previous underframes U701. When the moving amount exceeds the threshold, the frame is stopped.

  In addition, when only the appropriate frame T703 is an unusable frame in the appropriate / under pair (T703 / U703), the moving object is detected by using the next appropriate frame T704 and the immediately preceding available appropriate frame T702. I do. If the moving amount of the moving object is equal to or smaller than the threshold, the HDR frame H604 is generated using the immediately preceding appropriate frame T602, replaced with the immediately preceding HDR frame H703, and if the threshold is exceeded, the frame stop is set. Do.

  Hereinafter, the HDR frame generation processing at the time of moving image recording according to the third embodiment will be described with reference to FIG. 15 based on the example of FIG. 15 is started in response to an instruction to start moving image recording by HDR synthesis from the operation unit 112, and is realized by the imaging control unit 111 executing a predetermined control program and controlling each unit.

  In step S501, the synthesis processing unit 107 synthesizes the HDR frame H501 with an appropriate / under combination (T701 / U701).

  In step S502, the imaging control unit 111 determines whether an unusable frame is included in either appropriate or under, or both. If there is no unusable frame as a result of the determination, the process returns to step S501 to synthesize the next appropriate / under pair (T702 / U702). If there is an unusable frame, the process proceeds to step S503.

  In step S503, the imaging control unit 111 determines whether the unusable frame starts from the appropriate frame T702. If it is determined that the frame starts from an appropriate frame, frame stop is performed (S508). The immediately preceding HDR frame H701 is reused, and if the unusable frame is only an underframe, the process proceeds to step S504.

  In step S504, the imaging control unit 111 determines whether the second previous underframe U700 and the previous underframe U701 are usable frames. As a result of the determination, if at least one of them is an unusable frame, the process proceeds to step S508, and if both are usable frames, the process proceeds to step S505.

  In step S505, the imaging control unit 111 detects a moving object using the images of the two previous underframes U700 and U701. The moving object detection method is obtained from, for example, an absolute difference value between two frames.

  In step S506, the imaging control unit 111 determines whether the moving amount of the moving object detected in step S505 is less than or equal to the threshold value. If it is equal to or less than the threshold value, the process proceeds to step S507. .

  In step S507, the composition processing unit 107 generates an HDR frame H702 using the previous underframe U701, and proceeds to step S508.

  In step S508, the imaging control unit 111 performs frame stop and reuses the immediately preceding HDR frame H702.

  In step S509, the imaging control unit 111 determines whether or not a pair can be created using the appropriate / under (T704 / U704) or under / appropriate (U703 / T704) available frames. As a result of the determination, if any pair can be created, the process proceeds to step S510. If a pair cannot be created, frame stop (S409) is performed, and the HDR frame H702 immediately before the frame stop is reused.

  In step S510, the imaging control unit 111 determines whether the combination is appropriate / under (T704 / U704) or under / adequate after (U703 / T704), and the first frame of available frames is appropriate. Determine if it is a frame. As a result of the determination, in the case of an appropriate / under combination (T704 / U704), the process returns to step S501, and the HDR frame is synthesized in a stationary state. In the case of an under / appropriate (U703 / T704) combination The process proceeds to step S511.

  In step S511, the imaging control unit 111 performs moving object detection using the appropriate image T702 of the previous available frame and the next appropriate image T704.

  In step S512, the imaging control unit 111 determines whether or not the movement amount of the movement detected in step S511 is equal to or less than a threshold value. .

  In step S513, the composition processing unit 107 synthesizes the HDR frame H704 with a combination of under and one after (U703 / T704), and the process proceeds to step S514.

  In step S514, the imaging control unit 111 replaces the last frame H703 of the frame stop with the HDR frame H704 synthesized in step S513. In FIG. 14, for example, when the unusable frame is not detected at the appropriate frame, that is, when the usable frame starts from the appropriate frame, the last frame is not replaced in step S514. Similarly, an HDR frame synthesized from an appropriate / under pair is used.

  Thereafter, in step S515, the imaging control unit 111 repeats the processing from step S501 until there is a moving image recording end instruction from the operation unit 112.

  According to the third embodiment described above, in addition to the effects of the first embodiment, when an unusable frame occurs, whether moving object detection is performed in frames before and after the unusable frame, and interpolation is performed in the usable frame. Determine whether to stop the frame. In this way, control is performed so that the HDR frame does not become an unnatural image. As a result, even when a frame stop occurs in moving image recording using HDR synthesis, it is possible to shorten the frame stop period and perform moving image recording with continuity between frames.

  In the embodiment described above, an example in which a composite frame is generated from at least an appropriate / under combination is described as an HDR composite process at the time of moving image recording. The present invention is applicable even when used for arbitrary purposes such as display and display. Further, in the above-described embodiment, the interruption due to the diaphragm drive is exemplified as the cause of occurrence of the frame stop. However, the present invention is not limited to this and can be applied regardless of the cause of occurrence.

  Further, the present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of executing reading. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 104 ... Image sensor, 105 ... Image processing part, 106 ... Frame recording part, 107 ... Composition processing part, 109 ... Aperture drive part, 110 ... Exposure control part, 111 ... Imaging control part

Claims (17)

Imaging means for repeatedly imaging a plurality of frames at different exposures;
Combining means for combining the plurality of frames with different exposures to generate one combined frame;
Determining means for determining whether there is an unusable frame that cannot be used as a composite frame among the plurality of frames;
Control means for controlling the synthesizing means so as to perform a first process of generating a synthesized frame using available frames before and after the unavailable frame when there is the unavailable frame. An imaging device that is characterized.   The imaging apparatus according to claim 1, wherein the determination unit determines an unusable frame based on a luminance difference between a plurality of frames.   When there is the unusable frame, the control means uses a usable frame immediately before the unusable frame as a composite frame generated using the unusable frame instead of the first process. The imaging apparatus according to claim 1, wherein control is performed so as to perform a second process of replacing with a synthesized frame generated in this manner.   The control means replaces a final composite frame replaced by the second process with a composite frame generated using an available frame after an unusable frame is no longer detected. Item 4. The imaging device according to Item 3.   The control means performs the first process when the luminances of a plurality of frames used for generating the composite frame are within a predetermined range, and performs the second process when the luminance is not within the predetermined range. The imaging apparatus according to claim 3, wherein the imaging apparatus is controlled as follows. The composition unit is characterized in that a composition ratio when a composite frame is generated in a steady state where there is no unusable frame is different from a composition ratio when a composite frame is generated by the first processing. Item 6. The imaging device according to any one of Items 3 to 5. Detection means for detecting a subject in the image using a plurality of available frames before the unusable frame;
When the luminance of the plurality of frames combined in the first process is within a predetermined range, the combining unit switches the combination ratio of the predetermined subjects in the images of the plurality of frames to change the predetermined subject. The imaging apparatus according to claim 6, wherein a composite frame in which multiple images are displayed is generated.   The imaging apparatus according to claim 7, wherein the predetermined subject is a moving object.   The imaging apparatus according to claim 8, wherein the control unit switches between performing the first process and performing the second process according to a moving amount of the moving body.   10. The imaging apparatus according to claim 6, wherein the control unit interchanges the composition ratio in the steady state and the composition ratio in the first processing. 11. The plurality of frames with different exposures consist of a set of an appropriate frame imaged at an appropriate exposure and an underframe imaged at an underexposure,
The control means controls to perform the second process at least when the appropriate frame is an unusable frame,
If only the under frame is an unusable frame, whether to perform the first process or the second process depending on whether the under frame immediately before the unusable frame is usable or not is used. The imaging apparatus according to claim 3, wherein the imaging apparatus is switched. The plurality of frames with different exposures consist of a set of an appropriate frame imaged at an appropriate exposure and an underframe imaged at an underexposure,
The control means controls to perform the second process at least when the appropriate frame is an unusable frame,
When only the under frame is an unusable frame and the under frame immediately before the unusable frame is usable, control is performed so that the first process is performed using the usable under frame. The imaging device according to any one of claims 3 to 10, wherein:   13. The imaging apparatus according to claim 12, wherein the control unit generates a composite frame by switching an appropriate frame composition ratio to 100% in the first processing.   When the unusable frame immediately before the usable frame is a suitable frame after the unusable frame is no longer detected, the control means uses a suitable frame immediately after the usable frame to obtain a predetermined composition ratio. 14. The imaging apparatus according to claim 12, wherein a composite frame in which a predetermined subject is displayed in a multiplexed manner is generated. An imaging apparatus control method comprising: an imaging unit that repeatedly captures a plurality of frames with different exposures; and a combining unit that combines the plurality of frames with different exposures to generate one combined frame,
Determining whether there is an unusable frame that cannot be used as a composite frame among the plurality of frames;
And, when there is the unusable frame, controlling the synthesizing unit to perform a first process of generating a synthesized frame using available frames before and after the unusable frame. A method for controlling the imaging apparatus.   The program for functioning a computer as each means of the imaging device described in any one of Claims 1 thru | or 14.   A computer-readable storage medium storing a program for causing a computer to function as each unit of the imaging apparatus according to any one of claims 1 to 14.