JP6119235B2 - Imaging control apparatus, imaging system, imaging control method, and program - Google Patents

Description

  The present invention relates to an imaging control device, an imaging system, an imaging control method, and a program, and more specifically, an imaging control device capable of providing appropriate imaging conditions for a plurality of imaging optical systems, and an imaging system including the imaging control device The present invention relates to an imaging control method executed by the imaging control apparatus, and a program for realizing the imaging control apparatus.

  2. Description of the Related Art An omnidirectional imaging system is known that uses a plurality of wide-angle lenses such as fish-eye lenses and super-wide-angle lenses to image omnidirectional (hereinafter referred to as omnidirectional sphere) at once. In the omnidirectional imaging system, an image from each lens is projected onto the sensor surface, and the obtained images are combined by image processing to generate an omnidirectional image. For example, an omnidirectional image can be generated using two wide-angle lenses having an angle of view exceeding 180 degrees. In the image processing, distortion correction and projective transformation are performed on the partial images photographed by each lens optical system, and the partial images are connected using the overlapping regions included in the partial images, thereby producing one omnidirectional image. Is generated.

  Conventionally, an exposure correction technique in a digital camera that obtains an appropriate exposure from a captured image is known. In this exposure correction technique, an image is acquired from each sensor, an integrated value is calculated by an image processing circuit block, and an exposure condition is set based on the integrated value. For this reason, in the prior art, in order to obtain appropriate exposure conditions, it may take several frames.

  For example, Japanese Patent Application Laid-Open No. 11-341342 (Patent Document 1) is known as a technique for increasing the speed of exposure correction as described above. In the prior art of Patent Document 1, in response to the release operation, the shutter speed of the electronic element shutter in the CCD is sequentially set from a high speed to a low speed to obtain a plurality of exposure detection values, and an optimum exposure is obtained from these exposure detection values. A configuration for selecting a detection value is adopted.

  According to the configuration of Patent Document 1, the processing time from the start of the release operation to the main exposure can be shortened. However, before the main exposure, it is necessary to sequentially set the shutter speed from high to low to obtain a plurality of exposure detection values. In particular, when a plurality of imaging optical systems such as an omnidirectional compound eye camera and a panoramic compound eye camera are used, it is necessary to perform adjustment between the sensors in addition to the exposure correction of each of the plurality of sensors. For this reason, in order to obtain appropriate exposure conditions, a longer time tends to be required. Although the prior art of the above-mentioned Patent Document 1 aims at speeding up the monocular exposure control, it has not been able to speed up the compound eye camera.

  The present invention has been made in view of the above-described problems of the prior art, and in the imaging system using a plurality of imaging optical systems, the present invention takes time required to obtain imaging conditions for the plurality of imaging optical systems. An object is to provide an imaging control device, an imaging system, an imaging control method, and a program that can be shortened.

    In order to solve the above problems, the present invention provides an imaging control apparatus having the following characteristics. The imaging control apparatus includes a plurality of independent imaging condition calculation means for calculating imaging conditions independently for each imaging optical system. The imaging control apparatus further calculates an adjusted imaging condition adjusted between the imaging optical systems based on a plurality of captured images taken by each imaging optical system based on the calculation result by the independent imaging condition calculation unit. Adjusting imaging condition calculation means. Thereby, this imaging control apparatus gives the imaging condition with respect to several imaging optical system.

  With the above configuration, in an imaging system using a plurality of imaging optical systems, the time required to obtain imaging conditions for the plurality of imaging optical systems can be shortened.

A sectional view showing an omnidirectional imaging system by this embodiment. The hardware block diagram of the omnidirectional imaging system by this embodiment. The figure explaining the flow of the whole image processing in the omnidirectional imaging system by this embodiment. (A, B) The figure which illustrates the synthesized image obtained by synthesize | combining the captured images A and B image | photographed with the fisheye lens, and (C) the captured images A and B. The figure which illustrates typically the area | region division system of a captured image. 6 is a flowchart showing a main flow of exposure control executed by the imaging system according to the present embodiment. The flowchart which shows the compound eye appropriate exposure calculation process which the imaging system of this embodiment performs. The figure which shows the function which matches an illumination value with an exposure target level. The figure explaining the divided area mutually matched between both sensors in a plurality of picked-up images imaged with omnidirectional imaging system 10 in a specific embodiment. The sequence diagram which shows the exposure control performed in the imaging system of this embodiment.

  Hereinafter, although embodiment is described, it is not limited to embodiment described below. In the following embodiments, as an example of an imaging control device, an imaging body including two fisheye lenses in an optical system is provided, and a function for determining imaging conditions based on a captured image captured by two fisheye lenses is provided. A description will be given using the omnidirectional imaging system 10. That is, the omnidirectional imaging system 10 includes an imaging body that includes three or more fisheye lenses in an optical system, and has a function of determining imaging conditions based on captured images captured by three or more fisheye lenses. Also good. In the embodiment to be described, the fisheye lens includes a so-called wide-angle lens or a super-wide-angle lens.

  Hereinafter, the overall configuration of the omnidirectional imaging system according to the present embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a sectional view showing an omnidirectional imaging system (hereinafter simply referred to as an imaging system) 10 according to the present embodiment. An imaging system 10 shown in FIG. 1 includes an imaging body 12, a casing 14 that holds the imaging body 12 and components such as a controller and a battery (not shown), and a shutter button 18 provided on the casing 14. .

  The imaging body 12 shown in FIG. 1 includes two imaging optical systems 20A and 20B and two solid-state imaging devices 22A and 22B such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor. In the present embodiment, a combination of the imaging optical system 20 and the solid-state imaging device 22 one by one is referred to as an imaging optical system. The imaging optical system 20 can be configured as a fish-eye lens, for example, with 7 elements in 6 groups. In the embodiment shown in FIG. 1, the fisheye lens has a total angle of view greater than 180 degrees (= 360 degrees / n; n = 2), and preferably has an angle of view of 185 degrees or more. Has an angle of view of 190 degrees or more.

  The optical elements (lens, prism, filter, and aperture stop) of the two imaging optical systems 20A and 20B are solid so that their optical axes are positioned orthogonal to the center of the light receiving region of the corresponding solid-state imaging element 22. A positional relationship is determined with respect to the imaging elements 22A and 22B. At the same time, the optical element is positioned so that the light receiving area is the image plane of the corresponding fisheye lens. Each of the solid-state imaging devices 22 is a two-dimensional imaging device in which a light receiving region forms an area area, and converts light collected by the combined imaging optical system 20 into an image signal.

  In the embodiment shown in FIG. 1, the imaging optical systems 20A and 20B have the same specifications, and are combined in opposite directions so that their optical axes match. The solid-state imaging elements 22A and 22B convert the received light distribution into image signals and output them to image processing means on a controller (not shown). In the image processing means, the captured images respectively input from the solid-state imaging devices 22A and 22B are connected and combined to generate an image with a solid angle of 4π radians (hereinafter referred to as “global celestial image”). The omnidirectional image is an image of all directions that can be seen from the shooting point. Here, in the embodiment shown in FIG. 1, the omnidirectional image is generated, but a so-called panoramic image obtained by photographing 360 degrees only on the horizontal plane may be used.

  As described above, since the fisheye lens has a full angle of view exceeding 180 degrees, when composing an omnidirectional image, overlapping image portions represent the same image in the captured images captured by the respective imaging optical systems. It is used as reference data for image stitching as reference data. The generated omnidirectional image is output to, for example, an external storage medium such as a display device, a printing device, an SD (registered trademark) card, or a compact flash (registered trademark) that is included in or externally connected to the imaging system 10. Is done.

  FIG. 2 shows a hardware configuration of the imaging system 10 according to the present embodiment. The imaging system 10 includes a digital still camera processor (hereinafter simply referred to as a processor) 100, a lens barrel unit 102, and various components connected to the processor 100. The lens barrel unit 102 includes the above-described two sets of lens optical systems 20A and 20B and the solid-state imaging elements 22A and 22B. The solid-state imaging device 22 is controlled by a control command from a CPU 130 described later in the processor 100.

  The processor 100 includes an ISP (Image Signal Processor) 108, a DMAC (Direct Memory Access Controller) 110, an arbiter (ARBMEMC) 112 for arbitrating memory access, a MEMC (Memory Controller) 114 for controlling memory access, And a distortion correction / image synthesis block 118. The ISPs 108A and 108B perform automatic exposure (AE) control, white balance setting, and gamma setting on images input through signal processing of the solid-state imaging devices 22A and 22B, respectively.

  The SDRAM 116 is connected to the MEMC 114. Data is temporarily stored in the SDRAM 116 when processing is performed in the ISPs 108A and 180B and the distortion correction / image synthesis block 118. The distortion correction / image synthesis block 118 uses the information from the triaxial acceleration sensor 120 for two captured images obtained from the two imaging optical systems, performs distortion correction and top / bottom correction, and synthesizes the images.

  The processor 100 further includes a DMAC 122, an image processing block 124, a CPU 130, an image data transfer unit 126, an SDRAM C 128, a memory card control block 140, a USB block 146, a peripheral block 150, and an audio unit 152. Serial block 158, LCD driver 162, and bridge 168.

  The CPU 130 controls the operation of each unit of the imaging system 10. The image processing block 124 performs various image processing on the image data. The resize block 132 is a block for enlarging or reducing the size of the image data by interpolation processing. The JPEG block 134 is a codec block that performs JPEG compression and expansion. H. H.264 block 136 is an H.264 block. It is a codec block that performs video compression and decompression such as H.264. The image data transfer unit 126 transfers the image processed by the image processing block 124. The SDRAM C 128 controls the SDRAM 138 connected to the processor 100. The SDRAM 138 temporarily stores image data when various processes are performed on the image data in the processor 100.

  The memory card control block 140 controls reading and writing with respect to the memory card inserted into the memory card slot 142 and the flash ROM 144. The memory card slot 142 is a slot for detachably attaching a memory card to the imaging system 10. The USB block 146 controls USB communication with an external device such as a personal computer connected via the USB connector 148. A power switch 166 is connected to the peripheral block 150.

  The audio unit 152 is connected to a microphone 156 from which a user inputs an audio signal and a speaker 154 from which a recorded audio signal is output, and controls audio input / output. The serial block 158 controls serial communication with an external device such as a personal computer, and is connected with a wireless NIC (Network Interface Card) 160. An LCD (Liquid Crystal Display) driver 162 is a drive circuit that drives the LCD monitor 164, and converts it into signals for displaying various states on the LCD monitor 164.

  The flash ROM 144 stores a control program and various parameters described by codes that can be read by the CPU 130. When the power is turned on by operating the power switch 166, the control program is loaded into the main memory. The CPU 130 controls the operation of each part of the apparatus according to a program read into the main memory, and temporarily stores data necessary for control in the SDRAM 138 and a local SRAM (not shown).

  FIG. 3 is a diagram illustrating the overall flow of image processing in the imaging system 10 according to the present embodiment. FIG. 3 shows main functional blocks for controlling imaging conditions in the present embodiment. First, an image is captured under a predetermined exposure condition parameter by each of the sensors 200A and 200B including the solid-state imaging elements 22A and 22B. Subsequently, an optical black correction process, a defective pixel correction process, a linear correction process, a shading process, and an area division process shown in process 1 are performed on the images output from the sensors 200A and 200B by the ISP 108 shown in FIG. Is stored in memory.

  The optical black correction process is a process in which the output signal of the effective pixel area is clamp-corrected with the output signal of the optical black area in the solid-state image sensor 22 as the black reference level. A solid-state imaging device such as a CMOS is manufactured by forming a large number of photosensitive elements on a semiconductor substrate, and pixel values cannot be captured locally because impurities are mixed into the semiconductor substrate during the manufacturing. Defective pixels may occur. The defective pixel correction process is a process of correcting the pixel value of the defective pixel based on the combined signal from a plurality of pixels adjacent to the defective pixel as described above.

  The linear correction process is a process for performing linear correction for each RGB. The shading correction process is a process for correcting the distortion of the shadow of the effective pixel region by multiplying the output signal of the effective pixel region by a predetermined correction coefficient. In the area dividing process, an image area constituting the captured image is divided into a plurality of areas, and an integrated value (or integrated average value, which will be described later) is calculated for each divided area. In the embodiment to be described, the integrated value for each divided region is calculated in the form of luminance, and typically, an R (red) value, a G (green) value, and a coefficient derived from an empirical rule are used. This is done by a weighted average of B (blue) values.

  When the processing 1 is completed by the ISP 108, the ISP 108 further performs white balance processing, gamma correction processing, Bayer interpolation processing, YUV conversion processing, edge enhancement processing, and color correction processing shown in processing 2, and stores them in the memory. Saved. Where the amount of light transmitted varies depending on the color of the color filter on the solid-state image sensor 22, the white balance process corrects the sensitivity difference between the R, G, and B colors, and increases the gain to make white in the captured image appear white. It is a process to set. Based on the RGB integrated value (or integrated average value) data for each divided area calculated by the area dividing average process, the white balance calculating unit 220 calculates white balance processing parameters. The gamma correction process is a process performed on the input signal so that the output maintains linearity in consideration of the characteristics of the output device.

  In addition, in CMOS, a color filter of any one of R, G, and B is attached to one pixel of the solid-state imaging device 22, and Bayer interpolation processing is an interpolation that interpolates two insufficient colors from surrounding pixels. It is processing. The YUV conversion process is a process for converting the RAW data in the RGB data format into the YUV data format of the luminance signal Y and the color difference signal UV. In the edge enhancement process, an edge part is extracted from the luminance signal of the image, a gain is applied to the edge, and noise of the image is removed in parallel with the edge extraction. In the color correction process, saturation setting, hue setting, partial hue change setting, and color suppression setting are performed.

  When the above-described processing is completed for each of the two sensors 200A and 200B, distortion correction and synthesis processing are performed on each captured image that has been subjected to the above-described processing, appropriately tagged, and the spherical image is stored in the built-in memory or Files are saved to external storage. In the process of the distortion correction and the synthesis process, the inclination upside down correction may be performed by obtaining information from the triaxial acceleration sensor 120. Further, the stored image file may be appropriately compressed. In addition, a thumbnail image may be generated by cropping and cutting out the central area of the image.

  In the overall flow of image processing described above, the exposure condition parameters for the sensors 200A and 200B are determined by exposure control using the output of the sensor 200. In the imaging system 10, in order to display a captured image on a liquid crystal monitor or EVF (Electronic View Finder) that is provided or externally connected, an image signal from the sensor 200 is always read out. In the automatic exposure control, photometry is performed based on the read image signal to determine whether the level is appropriate, and in accordance with the result, the aperture value (F value), exposure time (shutter speed), amplifier gain (ISO) Modify exposure condition parameters such as sensitivity. And this flow is repeated for every frame, and appropriate exposure is obtained.

  On the other hand, when omnidirectional imaging is performed using the omnidirectional imaging system 10 as shown in FIG. 1, two captured images are generated by the two imaging optical systems. At this time, the omnidirectional imaging system 10 has a wider imaging range and more often a direct light source is captured as compared to a normal monocular camera or the like. When a high-luminance body such as the sun is reflected in the shooting scene, flare occurs in the captured image of one imaging optical system, as illustrated in FIGS. 4A and 4B, due to multiple reflections and the like. There is a possibility that the flare spreads over the entire surface centering on the high-luminance body. In such a case, exposure control for both eyes becomes difficult, and in the composite image, as shown in FIG. 4C, due to an increase in one offset, the overall contrast of only one image decreases. As a result, a difference in brightness occurs in the joint portion, and the image quality of the omnidirectional image is impaired. In addition, there may be a case where an object suitable for exposure correction does not exist in an overlapping region serving as a joint between a plurality of captured images (such as an extremely black subject or a white subject).

  Therefore, in a system using a plurality of imaging optical systems, such as the omnidirectional imaging system 10 described above, the exposure conditions are controlled so that proper exposure is obtained by individual sensors and brightness discontinuity is eliminated between the sensors. It needs to be adjusted. Due to the complexity of this exposure control, in order to obtain appropriate exposure condition parameters for the compound eye system as a whole, a larger number of frames are required and a longer time is required compared to a monocular system.

  Therefore, the imaging system 10 according to the present embodiment employs multi-stage exposure control in order to shorten the time required to obtain the appropriate exposure. That is, in the present imaging system 10, individual exposure control units 202A and 202B that perform independent exposure control for each sensor and a subsequent exposure condition control unit 210 that performs exposure control adjusted by the plurality of sensors as a whole are used. A sensor exposure condition parameter is determined. In the preferred embodiment, the adjustment between the plurality of sensors by the rear exposure condition control unit 210 can be started from the time when the calculation of the exposure conditions of the individual sensors has converged by the front exposure condition control units 202A and 202B.

  The pre-stage exposure condition control units 202A and 202B are provided in the sensors for the respective sensors 200A and 200B, calculate the exposure condition parameters independently for each sensor, and set them in their own registers. The exposure control method employed by the pre-stage exposure condition control units 202A and 202B is not particularly limited, and may be an exposure correction method using a captured image, for example. The exposure control of the pre-stage exposure condition control unit 202 is simpler than exposure control performed by adjusting between a plurality of imaging optical systems at least in that the exposure condition parameters are calculated independently for each imaging optical system. It has become. In the present embodiment, the front-stage exposure condition control units 202A and 202B constitute independent imaging condition calculation means for calculating the imaging conditions independently for each imaging optical system.

  The rear exposure condition control unit 210 adjusts the exposure of individual sensors adjusted between the sensors based on a plurality of captured images captured by the sensors 200A and 200B based on the calculation results of the front exposure condition control units 202A and 202B. Calculate condition parameters. The calculated exposure condition parameter after adjustment is set in the register of each sensor by the exposure condition register setting unit 204A, 204B of each sensor. The post-exposure condition control unit 210 constitutes an adjustment imaging condition calculation unit in the present embodiment. The post-exposure condition control unit 210 can be realized by the ISP 108, the CPU 130, and the like.

  More specifically, the post-exposure condition control unit 210 calculates compound exposure condition parameters that are adjusted so that the brightness of the overlapped portion of the sensors 200A and 200B matches that of the pre-stage stability determination unit 212. Part 214. The exposure condition parameter calculated independently for each sensor is input to the rear exposure condition control unit 210 from each of the previous exposure condition control units 202A and 202B. The pre-stage stability determination unit 212 determines whether or not the change in the exposure condition calculated by each of the pre-stage exposure condition control units 202A and 202B falls within a predetermined convergence condition range, and waits for the change to fall within the range. Then, the compound eye condition calculation unit 214 is called. The convergence condition is a standard for detecting that the change in exposure condition parameter (the difference between the previous exposure correction value and the newly acquired exposure correction value) has become smaller and stabilized, and for example, an experiment is performed on the vendor side. Etc. based on the above.

  The compound eye condition calculation unit 214 performs calculation for adjustment using the exposure condition parameters calculated by the preceding stage exposure condition control units 202A and 202B as initial conditions when the preceding stage stability determination unit 212 determines that the stabilization has been achieved. Start. During the period when the control by the rear exposure condition control unit 210 is valid, the calculations of the front exposure condition control units 202A and 202B are invalidated. On the other hand, in a specific embodiment, when the pre-defined condition indicating the calculation result of the inappropriate exposure condition is satisfied as a result of the calculation by the compound eye condition calculation unit 214, the pre-stage exposure condition control units 202A and 202B again Return to calculation. Examples of inappropriate exposure conditions include a case where at least one of them is black saturated or white saturated.

  Further, in process 1 by the ISP 108, an integrated value is calculated for each divided area constituting each captured image captured by each sensor by the area dividing process, and is input to the subsequent exposure condition control unit 210 as area divided integrated data. . FIG. 5 is a diagram illustrating an example of a region division method for a captured image. In the present embodiment, the light incident on the imaging optical system 20 is imaged on the light receiving region of the solid-state imaging device 22 according to a predetermined projection model such as an equidistant projection method. The picked-up image is picked up by a two-dimensional solid-state image pickup device in which the light receiving region forms an area area, and is image data expressed in a plane coordinate system.

  In the embodiment to be described, a so-called circumferential fisheye lens configuration in which an image circle diameter is smaller than an image diagonal line is adopted. Therefore, each obtained captured image becomes a planar image including the entire image circle on which each imaging range is projected as shown in FIGS. 4 (A) and 4 (B).

  In the present embodiment, the entire region of the captured image is a small region in a polar coordinate system (a circular coordinate system using the radius r and the declination θ) as shown in FIG. 5A, or in FIG. It divides | segments into the small area | region in the orthogonal coordinate system (The plane coordinate system using x coordinate and y coordinate) as shown, and let it be a division area, respectively. The outside of the image circle is a light-shielding region that is not exposed, and is not particularly limited, but may be excluded from the target of integration and averaging.

  In the area dividing process by the ISP 108, the captured image is divided into a plurality of small areas in the manner illustrated in FIG. 5A or 5B, and the luminance value of the pixel is determined for each small area. The integrated value or integrated average value is calculated. The integrated value is a value obtained by integrating the luminance values of all the pixels in the divided small area. On the other hand, the integrated average value is obtained by dividing the luminance integrated value by the area of each small area (the number of pixels constituting the small area (the light shielding area). It is a value normalized by)).

  Furthermore, as indicated by hatching in FIG. 5, each captured image is between a single region (hereinafter also referred to as a central region) that does not overlap between the imaging optical systems located at the center of the image circle and the imaging optical system. It is broadly divided into overlapping areas. The overlapping area is located around the image circle, and is hereinafter also referred to as a peripheral area. In the calculation related to the adjustment, the compound eye condition calculation unit 214 distinguishes and evaluates the single area for each sensor and the overlapping area between the sensors based on the input integrated value for each divided area, and according to this evaluation. Then, an adjusted exposure condition parameter for each of the plurality of sensors is determined. Whether a small region corresponds to an overlapping region or a single region can be determined by a distance from the calibrated image center coordinate value.

  The pixel average value is calculated for each single region and each overlapping region, and the pixel average value for each single region and each overlapping region is used as a region-based evaluation value for evaluating the captured image state for each region of the captured image. To be served. Note that the mode shown in FIG. 5 is an exemplification, and the region division method and the number of divisions are not particularly limited.

  Hereinafter, the exposure control executed by the imaging system according to the present embodiment will be described in detail with reference to FIGS. FIG. 6 is a flowchart showing a main flow of exposure control executed by the imaging system according to the present embodiment.

  The control shown in FIG. 6 is started from step S100 in response to, for example, the imaging system 10 shifting to the shooting mode by turning on the power or instructing mode switching. In step S101, the imaging system 10 branches control depending on whether or not a detection completion signal has been received from each of the sensors 200A and 200B. Here, the detection completion signal is a signal indicating that the sensor has completed reading of the image, and the calculation of the exposure condition by the pre-stage exposure condition control unit 202 has been appropriately completed. In step S101, this step S101 is looped until a detection completion signal is received (during NO). If a detection completion signal is received in step S101 (YES), control proceeds to step S102.

  In step S102, the imaging system 10 branches the control depending on whether or not the pre-stage control is effective. If the pre-stage control is valid in step S102 (YES), the control proceeds to step S103. In step S103, the imaging system 10 uses the subsequent exposure condition control unit 210 to inquire the previous exposure condition control units 202A and 202B about the calculation result of the previous exposure condition, and obtains the exposure condition parameter calculated in the previous process. To do.

  In step S104, the imaging system 10 determines whether the exposure conditions calculated by the previous exposure condition control units 202A and 202B have been stabilized by the previous-stage stability determination unit 212. If it is determined in step S104 that the previous exposure condition and the newly calculated exposure condition are not stabilized because they are significantly different from the reference (NO), the process loops to step S101. Here, the control proceeds to the next frame without proceeding to the subsequent control. On the other hand, if it is determined in step S104 that the predetermined convergence condition is satisfied and stabilized (YES), the control proceeds to the subsequent stage to step S105.

  In step S105, the imaging system 10 invalidates the individual upstream control by the upstream exposure condition control units 202A and 202B, and advances the control to step S106. As a result, the exposure condition parameters calculated by the front-stage exposure condition control units 202A and 202B are not reflected in the setting, and the parameters calculated in the later-stage control described later have priority. Referring back to step S102, if it is determined in step S102 that the pre-stage control is invalid (NO), the predetermined convergence condition is satisfied and stabilized by the processing in the previous frame. Control proceeds directly to S106.

  In step S106, a compound eye proper exposure calculation process described later with reference to FIG. 7 is called. When the exposure condition parameters after adjusting the plurality of sensors are determined by the compound eye proper exposure calculation process, the subsequent exposure condition control unit 210 requests the exposure condition register setting units 204A and 204B to update the exposure condition parameters. . Then, the control loops to step S101, and the control proceeds to the next frame. The determined exposure condition parameter is reflected in the sensor register in accordance with, for example, the vertical synchronization signal of the solid-state imaging device 22.

FIG. 7 is a flowchart showing a compound eye proper exposure calculation process executed by the imaging system of the present embodiment. In the embodiment described below, for the sake of convenience, it is assumed that only the exposure time t and the gain g are controlled exposure condition parameters, and the remaining aperture value a is fixed. A specific exposure condition parameter is an exposure time equivalent value exp _X (exp _X = g _X * t _X ; gain _X equivalent (g _X = 1); where subscript X identifies the camera And A or B.) corresponding to the exposure correction value calculated as follows). However, in other embodiments, all of the exposure time t, the gain g, and the aperture value a may be calculated as exposure condition parameters, or one or more arbitrary selected from the exposure time t, the gain g, and the aperture value a. A combination of these parameters may be calculated as the exposure condition parameter.

  The process shown in FIG. 7 is called at step S106 shown in FIG. 6, and is started from step S200. In step S201, the imaging system 10 subtracts the optical black (OB) value calculated for each captured image from the integrated value for each divided region as a preparation. As the OB value of each captured image, an integrated value (the value of the frame or the previous frame) for each divided region in the light shielding region outside the image circle shown in FIGS. 4A and 4B can be adopted.

In step S202, the imaging system 10 uses the compound eye condition calculation unit 214 to divide each divided region into an overlap region and a single region using the integrated value for each divided region, and calculate an average value of the integrated values. The division of the region can be determined by calibrating the center of the image and determining whether the distance r from the coordinate value of the image center to the center of the small region, for example, exceeds a predetermined threshold r1. In the embodiment to be described, the average value of the single region (r <r1) located at the center of the captured image A is sC _A , the average value of the single region of the captured image B is sC _B , and the overlap is located around the captured image A. The average value of the region (r1 ≦ r <r2: where r2 corresponds to the outer edge of the image circle) is defined as sE _A , and the average value of the overlapping region of the captured image _B is defined as sE _B.

In step S203, the imaging system 10 uses the compound eye condition calculation unit 214 to convert the average value into an illuminance value in order to detect the brightness of the imaging environment (scene). The illuminance value bC _X (X = A, B) of the center area of each sensor is obtained by using the average value sC _X of the integrated values of the individual areas of each sensor, the exposure time t, the amplifier gain g, and the characteristics of the imaging system 10. It can be calculated by the following equation (1) using the coefficient kLx corresponding to. The coefficient kLx is typically set on the vendor side based on experiments or the like. The following formula (1) shows a case where the aperture value a is fixed, and the fixed aperture value is added to the coefficient kLx.

The illuminance value bC _{X in the} central area is an evaluation value for evaluating the brightness of each light receiving area calculated from the pixel value of the actually captured image using the currently set exposure condition parameter. Incidentally, it is possible to intensity values of the surrounding area _{bE X (x = A, B} ) for even, by the same formula as the formula (1) is calculated using the average value sE _X of the integrated value of the overlap region.

In step S <b> 204, the imaging system 10 determines whether or not dark processing is necessary from the illuminance value bC _{X of} each sensor by the compound eye condition calculation unit 214. Whether or not dark processing is necessary can be determined using a threshold (darkTh) for a predefined illuminance value. If it is determined in step S204 that the illuminance value bC _X is less than the predetermined threshold (darkTh) for both sensors and dark processing is necessary (YES), the control proceeds to step S214. On the other hand, if it is determined in step S204 that the illuminance value bC _X of one of the sensors is equal to or greater than a predetermined threshold (darkTh) and the dark process is not required (NO), the control proceeds to step S205.

In step S205, the imaging system 10, the multi-eye condition calculation unit 214, the target exposure level _tL A for each sensor, to determine the tL _B. The exposure target level tL _X is preferably determined according to the brightness of the scene that can be determined from the converted illuminance value bC _X. FIG. 8 is a diagram illustrating a function for associating an illuminance value with an exposure target level. The function shown in FIG. 8 is determined based on the dynamic range characteristics of each sensor, and is stored as an approximate function coefficient or table. According to the function shown in FIG. 8, the target level is set brighter in a bright scene, and the target level is set darker in a dark scene.

In step S206, the imaging system 10 causes the compound eye condition calculation unit 214 to apply a gain to be applied to the central region for converging the average value of the central region to the determined exposure target level (hereinafter referred to as a central gain). Is calculated. The center gain can be calculated by using the following formula (2) within a predetermined range (for example, −4 EV <updownC _X <+4 EV), where the center gain of the sensor X (X = A, B) is updC _X. it can. When the value falls outside the predetermined range, the center gain uploadC _X is limited to a lower limit value (for example, −4 EV) or an upper limit value (for example, +4 EV).

In step S207, the imaging system 10 determines whether or not the center gain updateC _X has a value that suggests saturation, according to the obtained center gain updateC _A updownC _B. When the center gain updC _X is extremely large, it indicates the possibility that the pixel average value sC _X is small and black-saturated. On the other hand, when it is extremely small, the average value sC _X may be large and white-saturated. Showing gender. Therefore, if the center gain UpdownC _X extreme is believed that exposure is greatly deviated from the correct exposure. Therefore, if it is determined in step S207 that the center gain of any sensor is extremely small or large (YES), the control is branched to step S213. That is, the logical condition {sC _A > thHi OR sC _A <thLo OR sC _B > thHi OR sC _B <thLo} is satisfied using the upper limit threshold (thHi) and the lower limit threshold (thLo) with respect to the average value sC _X of the central region. If so, control branches to step S213.

  In step S213, the imaging system 10 validates the individual pre-stage control by the pre-stage exposure condition control units 202A and 202B, ends the process in step S212, and returns to the control after step S106 illustrated in FIG.

On the other hand, if it is determined in step S207 that the center gain is appropriate (NO), control branches to step S208. In step S208, the imaging system 10 uses the compound eye condition calculation unit 214 to calculate a peripheral gain to be applied to the overlapping region that converges to the average value sE of the overlapping regions of both sensors. Peripheral gain, sensor X (X = A, B) surrounding the gain as UpdownE _X, a predetermined range _{(e.g., -4EV <updownE X <+ 4EV} ) within, be calculated using the following equation (3) it can. If a predetermined range, the peripheral gain UpdownE _X is limited to the lower limit value (for example, -4 eV) or the upper limit value (e.g. + 4EV).

In step S209, the imaging system 10 uses the compound eye condition calculation unit 214 to calculate the maximum difference (maxSurroundDiff) of the integrated value (attribute value) in the overlapping region between the plurality of sensors. In a specific embodiment, the captured image captured by the omnidirectional imaging system 10 is inverted between the sensor A and the sensor B as shown in FIG. In the process for obtaining the maximum difference, first, a difference between integrated values is calculated for each divided region corresponding to each other between the two sensors. As shown in FIG. 9, when the image is inverted between the sensors, the image center coordinates in the combined image are (cx _A , cy _A ) and (cx _B , cy _B ), respectively, and H _B is the image of the sensor B. Assuming the width, the relative position (x _B , y _B ) of sensor B with respect to the position (x _A , y _A ) of sensor _A can be obtained by the following equation. The maximum value among the obtained differences between the divided regions corresponding to each other is the maximum difference (maxSurroundDiff).

In step S210, the imaging system 10 performs weighting and averaging processing by the compound eye condition calculation unit 214, and calculates the maximum difference (maxSurroundDiff) of the overlapping region and the illuminance values bC _A and bC calculated and converted for the two sensors. Based on _B , an exposure condition parameter to be set is determined. The exposure condition parameter is determined within a range of an upper limit and a lower limit of an appropriate change amount of the exposure correction value.

  The exposure correction value is calculated as follows. First, using the maximum difference (maxSurroundDiff) calculated in step S209 and a preset allowable difference (diffNGLevel), the following formula (4) is used to evaluate the central region and the peripheral region. The ratio (rC, rE) is calculated. The evaluation ratio (rC, rE) between the central region and the peripheral region is a value that determines which of the central region and the peripheral region the evaluation weight is placed on. Here, R is a constant (R = 10).

  Further, the tolerance (diffNGLevel) depends on the characteristics of the imaging system 10 and is typically obtained from experiments or the like. Under various photographing conditions, the difference between the sensors of the average value is changed to determine whether blur due to flare or the like occurs on one side, and these are totaled. The degree of conspicuous difference in brightness is expressed by the degree of blurring, and a difference in which the degree of blurring exceeds a predetermined threshold is adopted as a tolerance (diffNGLevel).

  The above formula (4) is obtained by calculating the ratio of the central area so that the evaluation ratio rE with respect to the surrounding area becomes large when the difference in brightness between sensors in the overlapping portion is large with the tolerance (maxSurroundDiff = diffNGLevel) as a boundary. It works so that rC becomes small. On the other hand, if the difference in brightness between the sensors in the overlapping portion is smaller than the tolerance, the above equation (4) is such that the ratio rE of the peripheral area becomes small so that the evaluation ratio with respect to the central area rC increases accordingly. Work.

Subsequently, B is a constant (B = 10), and the illuminance value bC _X (x = A, B) of the central region of each sensor calculated in step S203 is used to calculate The evaluation weights (r _A , r _B ) are calculated. The evaluation weight (r _A , r _B ) of each sensor is a value that determines which sensor is assigned the evaluation weight.

The above formula (5) compares the brightness of the central area between images, and works so that the sensor with the lower illuminance value bC _X converted in the central area is evaluated more heavily. That is, according to the above formula (5), the evaluation weight of the darker sensor is increased.

Subsequently, using the upper and lower limiters (minGain, maxGain) and the buffer parameter dumping so that the exposure correction value does not change abruptly, the following equation (6) and (7) is used to adjust the adjustment gain (k _a, it calculates the _{k B).} The buffer parameter is a constant in the range of 0 <dumping ≦ 1.0.

The formula (6), the center gain UpdownC _X and peripheral gain UpdownE _X converges to 1, i.e. in the direction average pixel value of the central and peripheral regions converge to an average value sE between each exposure target level and sensor, Determine the gain k _X '. The CLIP (x, max, min) function is a function that gives x for min ≦ x ≦ max, gives min for x <min, and gives max for x> max. Therefore, the above equation (7) gives an adjustment gain (gain (k _X '), dumping) in which the value k _X ' calculated by the above equation (6) is buffered in the range between the upper limit and the lower limit.

Finally, final exposure correction values (exp _A , exp _B ) of each sensor are calculated by the following equation (8). Here, T represents the number of iterations, and the updated exposure correction value (exp _X ^{T + 1} ) is obtained from the currently set exposure correction value (exp _X ^T ) and the adjustment gains k _A and k _B. .

When the updated exposure correction value (exp _X ^{T + 1} : exposure time converted value corresponding to gain 1) is obtained, the exposure condition is determined using a table called a program diagram prepared in advance according to the characteristics of the imaging system 10. A combination of parameters (t, g) is determined. In the embodiment in which the aperture value a is also a control target, a combination of exposure condition parameters (t, g, a) is determined. Therefore, the above formulas (6), (7) and (8) give the adjusted exposure condition parameters according to the ratio of the weighting of the evaluation to the central region and the peripheral region and the weighting of the evaluation between the sensors. Work.

  In step S211, the imaging system 10 updates the determined exposure condition parameter setting of each sensor by the compound eye condition calculation unit 214, ends the process in step S212, and the steps after step S106 illustrated in FIG. Return to control. By repeating the processes shown in FIG. 6 and FIG. 7, the exposure condition parameter is converged to an appropriate exposure that matches the brightness in the overlapping area between the sensors.

Referring to step S204 again, if it is determined in step S204 that the illuminance value bC _X is less than the predetermined threshold value (darkTh) for both sensors and dark processing is necessary (YES), the control proceeds to step S214. It is done. This is because, when both sensors are dark, flare or the like is unlikely to occur and a difference in exposure between the two captured images hardly occurs. In step S214, the imaging system 10 uses the compound eye condition calculation unit 214 to determine a common exposure target level tL _AVE among the sensors. The common exposure target level tL _AVE preferably corresponds to the brightness of the scene that can be determined from the average value bC _AVE (bC _AVE = (bC _A + bC _B ) / 2) of the converted illuminance values bC _A and bC _B. Can be determined.

In step S215, the exposure condition parameter of each sensor is determined so that the brighter exposure condition matches the darker exposure condition. For example, when sensor B is darker (bC _A > bC _B ), the same exposure condition is set for both sensors according to the darker sensor B according to the following equation (9).

  Then, the control proceeds to step S211, and the imaging system 10 updates the determined exposure condition parameter setting of each sensor, ends this processing in step S212, and performs the control after step S106 shown in FIG. Return to.

In the embodiment described above, in step S213, the pre-stage control is validated and returned to the pre-stage control. However, the process of step S213 is an exception process, and various modes can be considered. In another embodiment, the exposure correction value for each camera may be directly obtained by the following equation using the calculated center gains upC _A and up _C , without returning to the previous control.

  FIG. 10 is a sequence diagram showing exposure control executed in the imaging system of the present embodiment. FIG. 10 shows a step corresponding to the front exposure condition control independent of each sensor and a step corresponding to the rear exposure condition control targeting a plurality of sensors. At the time of monocular exposure correction, exposure correction is performed independently for each sensor by the pre-stage exposure condition control units 202A and 202B in the sensors 200A and 200B.

  The processing from step S301 to step S305 corresponds to the processing of one frame. Similarly, the processing from step S306 to step S310 corresponds to the processing of the next one frame. In step S301, the sensors 200A and 200B each notify the ISP 108 of reading start, and in step S301, the sensors 200A and 200B respectively notify the ISP 108 of reading completion. When the calculation of the previous exposure condition parameter is completed, the sensors 200A and 200B respectively notify detection completion to the task realizing the rear exposure condition control unit 210 in step S303. The task of the post-stage exposure condition control unit 210 is activated by an interrupt signal from the ISP 108.

  In step S304, the post-exposure condition control unit 210 inquires of the sensors 200A and 200B about the exposure condition, and acquires the exposure condition parameter of each sensor in step S305. The rear-stage exposure condition control unit 210 acquires the exposure conditions of each sensor and determines whether to perform the rear-stage compound eye exposure correction. The exposure condition parameters calculated by the previous exposure condition control units 202A and 202B are applied to the next frame.

  Subsequently, the same processing is performed for each frame in step S306 to step S310 and step S311 to step S315. Here, the description is continued assuming that the exposure control in the previous stage is stabilized. In step S316, the next frame readout notification is sent from the sensors 200A and 200B to the ISP 108. Until the completion notification in step S318 is performed, the rear exposure condition control unit 210 performs the rear exposure condition control. Calculate parameters. In step S317, the rear exposure condition control unit 210 completes the calculation, notifies the system control task of the calculation together with the calculation result, and requests the update of the exposure condition parameter.

  In step S318, when the reading completion notification is performed from the sensors 200A and 200B to the ISP 108 and the control task, in step S319, the control task is synchronized with the sensor vertical synchronization signal at a timing when the sensor is not read. Update the exposure condition parameter in the register of each sensor. That is, in the post-stage exposure condition control, a new exposure condition is applied to the two frames ahead.

  In the preceding control, appropriate exposure can be obtained at high speed independently for each sensor. On the other hand, more accurate exposure control that considers the state of each sensor for a plurality of sensors is performed by subsequent control. Thereby, in an imaging system using a plurality of imaging optical systems, it is possible to apply high-precision exposure conditions adjusted between the imaging optical systems at high speed. Further, preferably, since the post-stage control is performed after the exposure control for each sensor has converged, the use time of resources such as the CPU is shortened, and the power consumption is also reduced. In addition, since the exposure conditions to be finally applied are adjusted between the imaging optical systems, there is a brightness discontinuity seen at the joint when a plurality of captured images by the plurality of imaging optical systems are combined. Reduced and high quality composite images can be provided.

  As described above, according to the present embodiment, in an imaging system using a plurality of imaging optical systems, an imaging control apparatus capable of reducing the time required to obtain imaging conditions for the plurality of imaging optical systems, An imaging system, an imaging control method, and a program can be provided.

  In the above-described embodiment, the imaging system 10 that captures a omnidirectional still image is described as an example of the imaging control device. However, the imaging control device is not particularly limited. In another embodiment, an omnidirectional video imaging system (or imaging device) that captures a moving image of a celestial sphere, a mobile information terminal such as a smartphone or a tablet that has a function of capturing the omnidirectional sphere, and an imaging device in an imaging system. It can also be configured as a digital still camera processor (control device) to be controlled. Further, in the above-described embodiment, the exposure control has a total of two stages including the front stage control and the rear stage control. However, this does not prevent expansion to three or more.

  The functional unit can be realized by a computer-executable program written in a legacy programming language such as an assembler, C, C ++, C #, Java (registered trademark), an object-oriented programming language, or the like. ROM, EEPROM, EPROM , Stored in a device-readable recording medium such as a flash memory, a flexible disk, a CD-ROM, a CD-RW, a DVD-ROM, a DVD-RAM, a DVD-RW, a Blu-ray disc, an SD card, an MO, or through an electric communication line Can be distributed. In addition, a part or all of the functional unit can be mounted on a programmable device (PD) such as a field programmable gate array (FPGA) or mounted as an ASIC (application-specific integration). Circuit configuration data (bit stream data) downloaded to the PD in order to implement the above functional unit on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, VHDL (VHSIC (Very High Speed) Integrated Circuits) Hardware Description Language)), data described in Verilog-HDL, etc. can be distributed by a recording medium.

  Although the embodiments of the present invention have been described so far, the embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art may conceive other embodiments, additions, modifications, deletions, and the like. It can be changed within the range that can be done, and any embodiment is included in the scope of the present invention as long as the effects of the present invention are exhibited.

DESCRIPTION OF SYMBOLS 10 ... Imaging system, 12 ... Imaging body, 14 ... Housing, 18 ... Shutter button, 20 ... Imaging optical system, 22 ... Solid-state image sensor, 100 ... Digital still camera processor, 102 ... Lens barrel unit, 108 ... ISP, 110 ... DMAC, 112 ... Arbiter (ARBMEMC), 114 ... MEMC, 118 ... Distortion correction / image synthesis block, 120 ... Three-axis acceleration sensor, 122 ... DMAC, 124 ... Image processing block, 126 ... Image data transfer unit 128 ... SDRAMC, 130 ... CPU, 132 ... Resize block, 134 ... JPEG block, 136 ... H. H.264 block, 138 ... SDRAM, 140 ... memory card control block, 142 ... memory card slot, 144 ... flash ROM, 146 ... USB block, 148 ... USB connector, 150 ... peripheral block, 152 ... audio unit, 154 ... speaker, 156: Microphone, 158 ... Serial block, 158 ... Serial block, 160 ... Wireless NIC, 162 ... LCD driver, 162 ... LCD driver, 164 ... LCD monitor, 166 ... Power switch, 168 ... Bridge, 200 ... Sensor, 202 ... Previous stage Exposure condition control unit, 204 ... Exposure condition register setting unit, 210 ... Rear stage exposure condition control unit, 212 ... Front stage stability determination unit, 214 ... Compound eye condition calculation unit, 220 ... White balance calculation unit

Japanese Patent Laid-Open No. 11-341342

Claims (9)

An imaging control device that provides imaging conditions for a plurality of imaging optical systems,
A plurality of independent imaging condition calculation means for calculating imaging conditions independently for each imaging optical system;
Determination means for determining whether or not the imaging condition is stabilized;
When it is determined by the determination means that the image has been stabilized, the imaging optical based on a plurality of captured images captured by each of the imaging optical systems with the imaging condition calculated by the independent imaging condition calculation means as an initial condition An imaging control apparatus comprising: adjusted imaging condition calculation means for calculating imaging conditions after adjustment for the plurality of imaging optical systems adjusted between systems. The adjusted imaging condition calculation means calculates an evaluation value for each area for evaluating a shooting state with respect to a single area for each imaging optical system and an overlapping area between the imaging optical systems constituting the plurality of captured images. The adjusted imaging condition calculation means calculates an adjusted imaging condition for each of the plurality of imaging optical systems based on the evaluation values for each of the single area and the overlapping area. The imaging control apparatus according to claim 1 . The adjusted imaging condition calculation means further includes:
Means for calculating an evaluation weight for the overlap region and the single region based on a difference in image attribute values of the overlap region between the captured images;
Means for calculating an evaluation weight between the captured images based on a comparison of brightness between the captured images, and the adjusted imaging condition calculation means includes an evaluation weight for the overlapping region and the single region; The imaging control apparatus according to claim 2 , wherein the adjusted imaging condition is calculated according to a weighting of evaluation between the captured images. During the calculation by the adjustment imaging condition calculation means, the calculation of the independent imaging condition calculation means is invalidated, and if the predefined condition indicating the calculation result of the inappropriate imaging condition is satisfied, the adjustment imaging with the calculation of the condition calculation means is disabled, the calculation of the independent imaging condition calculation means is re-enabled, the imaging control device according to any one of claims 1-3. The imaging condition, aperture, exposure time and gain of at least one image pickup device as claimed in any one of claims 1-4. An imaging system including a plurality of imaging optical systems and the imaging control device according to any one of claims 1 to 5 . An imaging control method for providing imaging conditions for a plurality of imaging optical systems, wherein the imaging control device includes:
An independent calculation step for calculating imaging conditions independently for each imaging optical system;
Under the calculation result in the independent calculation step, an imaging step for causing each of the imaging optical systems to perform imaging,
A determination step of determining whether or not the imaging condition by the independent calculation step is stabilized;
Based on a plurality of captured images captured by each of the imaging optical systems using the imaging conditions calculated in the independent calculation step as initial conditions when determined to be stabilized in the determination step, between the imaging optical systems An adjustment calculation step of calculating an adjusted image pickup condition for the plurality of image pickup optical systems that has been adjusted. The adjustment calculation step includes:
A sub-step of calculating an evaluation value for each region for evaluating a shooting state with respect to a single region for each of the imaging optical systems and an overlapping region between the imaging optical systems constituting the plurality of captured images that have been imaged;
A sub-step of calculating an adjusted imaging condition for each of the plurality of imaging optical systems based on the evaluation value for each of the single region and the overlapping region;
The imaging control method according to claim 7 , comprising: The imaging control device
A plurality of independent imaging condition calculation means for independently calculating imaging conditions for each imaging optical system of the plurality of imaging optical systems ,
Determination means for determining whether or not the imaging condition is stabilized; and
When it is determined by the determination means that the image has been stabilized, the imaging optical based on a plurality of captured images captured by each of the imaging optical systems with the imaging condition calculated by the independent imaging condition calculation means as an initial condition A program for functioning as an adjusted imaging condition calculation means for calculating imaging conditions after adjustment for the plurality of imaging optical systems adjusted between systems.