JP5548023B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to a technique for correcting a misalignment between a plurality of images and generating a composite image by synthesizing images after the misalignment correction.

  2. Description of the Related Art Conventionally, a technique for generating an image with a wide dynamic range by correcting misalignment between a plurality of images captured under different exposure conditions and synthesizing images after misalignment correction (Patent Document 1). reference).

JP 2008-181196 A

  However, in the conventional technique, the positional deviation between the plurality of images is corrected by a uniform method, and the positional deviation correction considering the subject area in the image has not been performed.

  It is an object of the present invention to provide a technique for generating a composite image by performing positional deviation correction corresponding to a region in an image and then synthesizing images after positional deviation correction.

An imaging apparatus according to an aspect of the present invention is an imaging apparatus that can generate a composite image by combining a plurality of images obtained by imaging, and performs image conversion by subjecting subject light received by the imaging element to photoelectric conversion. An image capturing unit that obtains data, an image region selecting unit that selects a part of the image region of the image data, and a positional shift between the plurality of images are detected for the entire image region. Position shift between the plurality of images by a position shift detection method different from the position shift detection method by the first position shift detection unit for the first position shift detection unit and the partial image region. And a second misalignment detecting unit for detecting misalignment and a misalignment between the plurality of images for the entire image region based on the misalignment detected by the first misalignment detecting unit. 1's place A second misalignment correction that corrects misalignment between the plurality of images for the partial image region based on the misalignment detected by the misalignment correction unit and the second misalignment detection unit. A first compositing unit that generates a first composite image by combining a plurality of images in which the positional deviation is corrected for the entire image area, and a positional deviation for the partial image area A second combining unit that combines the plurality of images corrected to generate a second combined image, and a display unit that displays the first combined image and the second combined image , The positional deviation detection accuracy of the second positional deviation detection unit is higher than the positional deviation detection accuracy of the first positional deviation detection unit .

An imaging method according to another aspect of the present invention is an imaging method using an imaging apparatus capable of generating a composite image by combining a plurality of images obtained by imaging, and out of the entire image area of image data. Selecting a partial image region; targeting the entire image region; a first misalignment detecting step for detecting misalignment between the plurality of images; and targeting the partial image region. A second displacement detection step for detecting displacement between the plurality of images by a displacement detection method different from the displacement detection method in the first displacement detection step; and the first displacement detection. Based on the misalignment detected in the step, the misalignment between the plurality of images is corrected for the entire image region, and the misalignment detected in the second misalignment detection step. Based on the misalignment, a step of correcting misalignment between the plurality of images for the partial image region, and a plurality of images corrected for misalignment for the entire image region are synthesized. Generating a first synthesized image; synthesizing a plurality of images whose positional deviations are corrected for the partial image region; and generating a second synthesized image; and And a step of displaying the composite image and the second composite image, and the displacement detection accuracy of the second displacement detection step is higher than the displacement detection accuracy of the first displacement detection step .

  According to the present invention, it is possible to generate a composite image by performing positional deviation correction according to a region in an image and then combining the images after positional deviation correction.

It is a block diagram which shows the structure of the digital still camera which is an imaging device in 1st Embodiment. It is a block diagram which shows the detailed structure of a motion vector calculation part. FIGS. 3A to 3H are diagrams for explaining a method of calculating a motion vector of the entire frame. It is a figure which shows an example of 16 motion vector B0-B15 located in the vicinity of an attention pixel. The composition processing unit combines image data shot with an exposure time shorter than the standard exposure time according to the subject brightness and image data shot with an exposure time longer than the standard exposure time, and has a wide dynamic range. It is a figure which shows the flow of the process which produces | generates a synthesized image. FIGS. 6A and 6B are diagrams for explaining a method of determining a subject of interest to be tracked. It is a flowchart which shows the procedure of the composite image generation process performed by the imaging device in 1st Embodiment. It is a flowchart which shows the detail of the process which acquires tracking area | region data. It is a figure which shows the tracking area | region containing the attention object designated by the photographer. It is a figure which shows an example of the image obtained by synthesize | combining the synthesized image of the whole area | region, and the synthesized image of a tracking area | region. It is a flowchart which shows the detail of the process which synthesize | combines image data, after correct | amending a position shift based on the motion vector of the whole flame | frame. It is a flowchart which shows the detail of the process which synthesize | combines image data, after correct | amending a position shift based on a local motion vector. It is a flowchart which shows the detailed process of weighted composition. It is a figure which shows the relationship between the correlation coefficient value K and the synthetic | combination ratio (alpha). FIG. 15A shows a result of simple synthesis of two image data with a synthesis ratio α = 0.5. FIG. 15B shows a result of weighted synthesis with a synthesis ratio α greater than 0.5. As a result, FIG. 15C is a diagram showing a result when the images are synthesized with the synthesis ratio α = 1. FIG. 16A is a diagram illustrating a tracking area in the evaluation frame, and FIG. 16B is a diagram illustrating an example of a motion vector of a block including the tracking area. 10 is a flowchart illustrating a procedure of composite image generation processing performed by an imaging apparatus according to a third embodiment. 14 is a flowchart illustrating a procedure of composite image generation processing performed by an imaging apparatus according to a fourth embodiment. It is a figure which shows an example of the method of superimposing and displaying on the synthesized image of the whole area | region, without enlarging the synthesized image of a tracking area | region.

-First embodiment-
FIG. 1 is a block diagram illustrating a configuration of a digital still camera that is an imaging apparatus according to the first embodiment. The digital still camera shown in FIG. 1 includes a camera body 1 and an interchangeable lens 2.

  The interchangeable lens 2 includes a lens 1010, a flash memory 1011, a microcomputer 1012, a driver 1013, and a diaphragm 1014. The interchangeable lens 2 is communicably connected to the camera body 1 via the I / F 999.

  The camera body 1 includes a mechanical shutter 101, an image sensor 102, an analog processing unit 103, an analog / digital conversion unit 104 (hereinafter referred to as A / D conversion unit 104), a bus 105, an SDRAM 106, and an image processing unit 107. An AE processing unit 108, an AF processing unit 109, an image compression / decompression unit 110, a memory interface 111 (hereinafter referred to as a memory I / F 111), a recording medium 112, an LCD driver 113, an LCD 114, and a microcomputer 115. An operation unit 116, a flash memory 117, a vibration sensor 118, a motion vector calculation unit 119, a synthesis processing unit 120, and a moving object tracking unit 121.

  The lens 1010 focuses the optical image of the subject on the image sensor 102. The lens 1010 may be a single focus lens or a zoom lens.

  The microcomputer 1012 is connected to the I / F 999, the flash memory 1011 and the driver 1013. The microcomputer 1012 reads and writes information stored in the flash memory 1011 and controls the driver 1013. The microcomputer 1012 can further communicate with the microcomputer 115 via the I / F 999, transmits lens focal length information and the like to the microcomputer 115, and receives information such as an aperture value from the microcomputer 115. Receive.

  In response to an instruction from the microcomputer 1012, the driver 1013 drives the lens 1010 to change the focal length and focus position, and drives the aperture 1014. The aperture 1014 is provided in the vicinity of the lens 1010 and adjusts the amount of light of the subject.

  The mechanical shutter 101 is driven in response to an instruction from the microcomputer 115 to control the time for exposing the subject to the image sensor 102.

  The image sensor 102 is an image sensor in which a Bayer array color filter is arranged in front of a photodiode constituting each pixel. The Bayer array has a line in which R pixels and G (Gr) pixels are alternately arranged in a horizontal direction, and a line in which G (Gb) pixels and B pixels are alternately arranged, and the two lines are further divided. It is configured by alternately arranging in the vertical direction. The imaging element 102 receives the light collected by the lens 1010 by a photodiode that constitutes a pixel and performs photoelectric conversion, and outputs the amount of light to the analog processing unit 103 as a charge amount. The image sensor 102 may be a CMOS type or a CCD type.

  The analog processing unit 103 performs waveform shaping on the electrical signal (analog image signal) read from the image sensor 102 while reducing reset noise and the like, and further increases the gain so that the target brightness is obtained. . The A / D conversion unit 104 converts the analog image signal output from the analog processing unit 103 into a digital image signal (hereinafter referred to as image data).

  A bus 105 is a transfer path for transferring various data generated in the digital camera to each unit in the digital camera. The bus 105 includes an A / D conversion unit 104, an SDRAM 106, an image processing unit 107, an AE processing unit 108, an AF processing unit 109, an image compression / decompression unit 110, a memory I / F 111, an LCD driver 113, and the like. The microcomputer 115, the vibration sensor 118, the motion vector calculation unit 119, the synthesis processing unit 120, and the moving object tracking unit 121 are connected.

  The image data output from the A / D conversion unit 104 is temporarily stored in the SDRAM 106 via the bus 105. The SDRAM 106 temporarily stores various data such as image data obtained by the A / D conversion unit 104 and image data processed by the image processing unit 107, the image compression / decompression unit 110, the synthesis processing unit 120, and the moving object tracking unit 121. Is a storage unit stored in

  The image processing unit 107 includes a white balance correction unit 1071 (hereinafter, WB correction unit 1071), a synchronization processing unit 1072, a color reproduction processing unit 1073, and a noise reduction processing unit 1074 (hereinafter, NR processing unit 1074). Various image processing is performed on the image data read from the SDRAM 106. The WB correction unit 1071 performs processing for correcting the white balance of the image data. The synchronization processing unit 1072 performs a process of synchronizing image data based on the Bayer array into image data including R, G, and B information per pixel. The color reproduction processing unit 1073 performs color reproduction processing that changes the color of an image, and the NR processing unit 1074 performs processing to reduce noise. The image data after the noise reduction processing is stored in the SDRAM 106.

  The vibration sensor 118 detects the movement of the camera body 1 including so-called camera shake. The motion vector calculation unit 119 calculates a motion vector between a plurality of images obtained by continuous shooting. The detailed configuration of the motion vector calculation unit 119 will be described later with reference to FIG.

  The composition processing unit 120 synthesizes a plurality of images photographed continuously. In particular, the composition processing unit 120 synthesizes image data photographed with an exposure time shorter than the standard exposure time according to the subject brightness and image data photographed with an exposure time longer than the standard exposure time, Generate a composite image with a wide dynamic range.

  The moving object tracking unit 121 tracks a subject of interest that is a moving object by a known method. A method for determining the subject of interest that is the tracking target will be described later.

  The AE processing unit 108 calculates subject luminance from the image data. The data for calculating the subject brightness may be an output of a dedicated photometric sensor. The AF processing unit 109 extracts a high-frequency component signal from the image data, and acquires a focus evaluation value by AF (Auto Focus) integration processing.

  When recording still image data, the image compression / decompression unit 110 reads image data from the SDRAM 106, compresses the read image data according to the JPEG compression method, and temporarily stores the compressed JPEG image data in the SDRAM 106. The microcomputer 115 creates a JPEG file by adding a JPEG header necessary for constructing a JPEG file to the JPEG image data stored in the SDRAM 106, and sends the created JPEG file via the memory I / F 111. To the recording medium 112.

  The image compression / decompression unit 110 also reads moving image data from the SDRAM 106 when recording moving image data. The compressed moving image data is temporarily stored in the SDRAM 106 after being compressed according to the H.264 system. The image compression / decompression unit 110 further performs a process of expanding (decompressing) the compressed data based on a command from the microcomputer 115.

  The recording medium 112 is, for example, a recording medium including a memory card that can be attached to and detached from the camera body 1, but is not limited thereto.

  The LCD driver 113 displays an image on the LCD 114. Display of images includes display of moving images such as REC view display that displays image data immediately after shooting for a short time, playback display of a JPEG file recorded on the recording medium 112, and live view display. When reproducing the compressed data recorded on the recording medium 112, the image compression / decompression unit 110 reads out the compressed data recorded on the recording medium 112, performs decompression (decompression) processing, and then temporarily decompresses the decompressed data. It is stored in the SDRAM 106. The LCD driver 113 reads the decompressed data from the SDRAM 106, converts the read data into a video signal, and then outputs it to the LCD 114 for display.

  As will be described later, in the present embodiment, when a tracking target subject exists in the image, the tracking target subject is enlarged and superimposed when the image is displayed on the LCD 114.

  The microcomputer 115 having a function as a control unit comprehensively controls various sequences of the digital camera body 1. An operation unit 116 and a flash memory 117 are connected to the microcomputer 115.

  The operation unit 116 is an operation member such as a power button, a release button, and various input keys. When one of the operation members of the operation unit 116 is operated by the user, the microcomputer 115 executes various sequences according to the user's operation. The power button is an operation member for instructing power on / off of the digital camera. When the power button is pressed, the power of the digital camera is turned on. When the power button is pressed again, the power of the digital camera is turned off. The release button has a two-stage switch of a first release switch and a second release switch. When the release button is pressed halfway and the first release switch is turned on, the microcomputer 115 performs a shooting preparation sequence such as AE processing and AF processing. Further, when the release button is fully pressed and the second release switch is turned on, the microcomputer 115 performs shooting by executing a shooting sequence.

  The flash memory 117 stores a white balance correction value, a low-pass filter coefficient, various parameters necessary for the operation of the digital camera, a manufacturing number for specifying the digital still camera, and the like. The flash memory 117 also stores various programs executed by the microcomputer 115. The microcomputer 115 reads parameters necessary for various sequences from the flash memory 117 according to a program stored in the flash memory 117, and executes each process.

  FIG. 2 is a block diagram illustrating a detailed configuration of the motion vector calculation unit 119. The motion vector calculation unit 119 includes an evaluation frame acquisition unit 20, a luminance signal extraction unit 21, an evaluation frame region setting unit 22, a comparison frame acquisition unit 23, a luminance signal extraction unit 24, and a comparison frame region setting unit 25. The inter-frame correlation processing unit 26, the reliability determination unit 27, the area-specific motion vector calculation unit 28, the entire frame motion vector calculation unit 29, and the local motion vector calculation unit 30 are provided.

  The SDRAM 106 stores at least two pieces of image data taken continuously. This image data is, for example, image data shot with an exposure time shorter than the standard exposure time corresponding to the subject brightness, and image data shot with an exposure time longer than the standard exposure time. The evaluation frame acquisition unit 20 reads image data used as a reference when calculating a motion vector from the two image data stored in the SDRAM 106. The reference image data may be any of the two image data. Here, the reference image data is referred to as an evaluation frame.

  The luminance signal extraction unit 21 extracts the luminance signal of the evaluation frame acquired by the evaluation frame acquisition unit 20.

  The evaluation frame region setting unit 22 divides the evaluation frame acquired by the evaluation frame acquisition unit 20 into blocks having a predetermined size, and a region having a predetermined size is divided into a motion vector calculation region in the divided blocks. Set to.

  FIG. 3A is a diagram illustrating an example of an evaluation frame 31 that is divided into a plurality of blocks 32 by the evaluation frame region setting unit 22. FIG. 3C is a diagram illustrating an example of a motion vector calculation region 35 set in one block 32 by the evaluation frame region setting unit 22. The motion vector calculation area 35 is set for all the blocks 32 in the evaluation frame.

  The comparison frame acquisition unit 23 reads image data to be compared with the evaluation frame from the plurality of image data stored in the SDRAM 106. This image data is image data to be combined with the evaluation frame. Here, the image data to be compared with the evaluation frame is called a comparison frame.

  The luminance signal extraction unit 24 extracts the luminance signal of the comparison frame acquired by the comparison frame acquisition unit 23.

  The comparison frame region setting unit 25 divides the comparison frame acquired by the comparison frame acquisition unit 23 into blocks of a predetermined size, and sets the divided block itself as a comparison region to be compared with the motion vector calculation region. The size of the block into which the comparison frame is divided is the same as the size of the block into which the evaluation frame is divided.

  FIG. 3B is a diagram illustrating an example of the comparison frame 33 divided into a plurality of blocks 34 by the comparison frame region setting unit 25, and FIG. 3D is a diagram illustrating the comparison region 34.

  The inter-frame correlation processing unit 26 obtains a correlation with the comparison frame 33 for each block 32 of the evaluation frame 31. Here, the correlation coefficient value is calculated while scanning the motion vector calculation area 35 set in the block 32 of the evaluation frame 31 within the comparison area 34 of the comparison frame 33 that exists at a position corresponding to the block 32. FIG. 3E is a diagram illustrating a state in which the motion vector calculation area 35 is scanned within the comparison area 34 of the comparison frame 33. The correlation coefficient value is, for example, the sum of absolute values of differences between the pixel value of each pixel in the motion vector calculation area 35 and the pixel value of each pixel in the area to be compared with the motion vector calculation area 35 in the comparison area 34. An error absolute value sum SAD (Sum of Absolute intensity Difference) is obtained. The smaller the SAD, the higher the correlation, and the larger the SAD, the lower the correlation.

  The region-specific motion vector calculation unit 28 determines the region having the smallest value among the correlation coefficient values calculated by the inter-frame correlation processing unit 26 as the region to which the motion vector calculation region 35 is moved, and the amount of movement thereof. Is a motion vector of the block 32 including the motion vector calculation area 35. FIG. 3F is a diagram illustrating an example of a motion vector 38 from the region 36 corresponding to the motion vector calculation region 35 toward the region 37 having the smallest correlation coefficient value in the comparison region 34.

  The above-described motion vector calculation is performed on all blocks divided by the evaluation frame region setting unit 22. FIG. 3G is a diagram illustrating an example of motion vectors calculated for all blocks.

  The reliability determination unit 27 determines the reliability of the motion vector of each block calculated by the region-specific motion vector calculation unit 28. For example, if the correlation coefficient value calculated when obtaining the motion vector is greater than or equal to a predetermined threshold value, it is determined that the reliability of the motion vector is low, and the correlation coefficient value is less than the predetermined threshold value. Determines that the reliability of the motion vector is high. Note that the reliability determination method is not limited to the above-described method, and the reliability may be determined in three or more stages instead of two.

  The entire frame motion vector calculation unit 29 calculates the motion vector of the entire frame based on the motion vector of each block calculated by the region-specific motion vector calculation unit 28 and the reliability determined by the reliability determination unit 27. . Here, among the motion vectors of all the blocks 32, an average vector of other motion vectors excluding motion vectors determined to have low reliability is used as the motion vector of the entire frame. FIG. 3H is a diagram illustrating an example of the motion vector 38 of the entire frame.

  Note that the motion vector weight determined to have low reliability is low, the motion vector weight determined to be high reliability is high, and the weighted addition average calculation of all motion vectors is performed. The obtained motion vector may be a motion vector of the entire frame.

  In addition, a histogram is created using other motion vectors excluding motion vectors determined to have low reliability among the motion vectors of all the blocks 32, and the most frequent motion vector is used as the motion vector of the entire frame. It doesn't matter.

  Based on the motion vector of each block calculated by the region-specific motion vector calculation unit 28, the local motion vector calculation unit 30 calculates a motion vector of each pixel using a method such as Cubic interpolation, for example. Similarly, the correlation coefficient K of each pixel is calculated in the same manner as the motion vector. In Cubic interpolation in the present embodiment, the motion vector of the target pixel is calculated using 16 motion vectors located in the vicinity of the target pixel to be processed.

FIG. 4 is a diagram illustrating an example of 16 motion vectors B0 to B15 located in the vicinity of the target pixel 40. The motion vector Bout of the pixel of interest 40 is obtained by the interpolation calculation of the following equation (1). However, in Expression (1), K _x0 , K _x1 , K _x2 , K _x3 , K _y0 , K _y1 , K _y2 , and K _y3 are interpolation coefficients that are determined according to the position coordinates of the _target pixel 40.

Bout = _Kx0 ( _Ky0 * B0 + _Ky1 * B4 + _Ky2 * B8 + _Ky3 * B12) + _Kx1 ( _Ky0 * B1 + _Ky1 * B5 + _Ky2 * B9 + _Ky3 * B13) + _Kx2 ( _Ky0 * B2 + _Ky1 * B6 + _Ky2 * B10 + _Ky3 * B14) + _Kx3 ( _Ky0 * B3 + _Ky1 * B7 + _Ky2 * B11 + _Ky3 * B15) (1)
The local motion vector calculation unit 30 calculates the motion vector and the correlation coefficient value K for all the pixels of the evaluation frame 31 by the method described above. Here, the motion vector of each pixel is collectively referred to as a local motion vector. In the calculation of the local motion vector, since it is necessary to calculate a motion vector for all the pixels of the evaluation frame 31, the processing time becomes longer than when the motion vector of the entire frame is calculated. However, the calculation accuracy of the local motion vector is higher than the calculation accuracy of the motion vector of the entire frame.

  FIG. 5 shows image data captured by the synthesis processing unit 120 with an exposure time shorter than the standard exposure time corresponding to the subject brightness (hereinafter referred to as short exposure time) and an exposure time longer than the standard exposure time (hereinafter referred to as long). It is a figure which shows the flow of a process which synthesize | combines with the image data image | photographed by exposure time, and produces | generates a synthesized image with a wide dynamic range. The imaging apparatus according to the present embodiment is capable of shooting with a short exposure time and a long exposure time during moving image shooting in a state where a processing mode for generating a composite image with a wide dynamic range is set or during live view display. Repeatedly and alternately.

  The composition processing unit 120 synthesizes the image data 51 captured with the long exposure time and the image data 52 obtained after the image data 51 and captured with the short exposure time. An image 57 is generated. Also, image data obtained after the image data 52, which is image data 53 taken with a long exposure time, and image data obtained after the image data 53, taken with a short exposure time. By combining the data 54, a composite image 58 is generated. Similarly, image data obtained next to the image data 54, which is image data 55 taken with a long exposure time, and image data obtained next to the image data 55, taken with a short exposure time. By combining the image data 56, a composite image 59 is generated.

  A method of determining a subject of interest for which the moving body tracking unit 121 performs tracking processing will be described. FIG. 6 is a diagram for explaining a method of determining a target subject to be tracked. Here, two methods for determining the subject of interest to be tracked will be described.

  FIG. 6A illustrates a method for determining the subject of interest to be tracked by the user touching the subject of interest 62 using the pen 61 from the image displayed on the LCD 114 of the digital still camera. FIG. In other words, the LCD 114 is a touch panel, and the user determines the subject to be tracked by touching the subject to be tracked.

  FIG. 6B illustrates a method of determining the subject of interest to be tracked by operating a predetermined operation button 65 after adjusting the angle of view so that the subject of interest 64 to be tracked is at the center of the LCD 114. It is a figure for doing. That is, when the predetermined operation button 65 is operated, the subject located at the center of the LCD 114 becomes the subject of interest to be tracked.

  FIG. 7 is a flowchart illustrating a procedure of a composite image generation process performed by the imaging device according to the first embodiment. The imaging apparatus according to the first embodiment performs a process of generating a composite image with a wide dynamic range by combining image data captured with a short exposure time and image data captured with a long exposure time. In addition, when there is a subject of interest to be tracked by the moving body tracking unit 121, the image data of the tracking area is synthesized from among a plurality of image data shot at different exposure times, and the image of the tracking area obtained by the synthesis is obtained. A process of enlarging the data and superimposing it on the combined image data of the entire image area is performed. In the following description, it is assumed that there is a subject to be tracked.

  In step S10, AF processing and AE processing are performed. Specifically, first, the AF processing unit 109 calculates a focus evaluation value. The microcomputer 115 issues a command for driving the lens 1010 to the driver 1013 based on the focus evaluation value. Based on this command, the driver 1013 drives the lens 1010 to change the focal length and the focus position. In the AE process, the AE processing unit 108 calculates the subject brightness, and refers to the exposure condition determination table stored in the flash memory 117 based on the calculated subject brightness, so that the ISO sensitivity, aperture, And determine the exposure time. However, in the exposure condition determination table used in the process of step S10, an exposure time shorter than the standard exposure time corresponding to the subject brightness is set.

  In step S20, shooting is performed. The shooting (moving image shooting) is the same as a conventionally used method. The driver 1013 drives the aperture 1014 so as to achieve a set aperture value based on an instruction from the microcomputer 1012. Then, photographing is performed based on the determined exposure time, and image data corresponding to the determined ISO sensitivity is obtained. Here, this image data is referred to as first image data.

  In step S <b> 30, the composition processing unit 120 reads the first image data (RAW data) obtained by the photographing in step S <b> 20 from the SDRAM 106.

  In step S40, AF processing and AE processing are performed again to change the exposure conditions and perform shooting. The AF process is the same as the process of step S10, but the AE process uses an exposure condition determination table in which an exposure time longer than the standard exposure time according to the subject brightness is determined.

  In step S50, shooting (moving image shooting) is performed. Image data obtained by this photographing is referred to as second image data.

  In step S60, the composition processing unit 120 reads the second image data (RAW data) obtained by the photographing in step S50 from the SDRAM 106.

  The processing from step S70 to step S90 is processing for the entire image area. On the other hand, the process from step S100 to step S140 is a process for the tracking area including the target subject to be tracked. The processing from step S70 to step S90 and the processing from step S100 to step S140 may be performed simultaneously in parallel, or either processing may be performed with priority.

  In step S70, the entire frame motion vector calculation unit 29 of the motion vector calculation unit 119 has the entire frame motion vector between the first image data read in step S30 and the second image data read in step S60. Is calculated.

  In step S80, the synthesis processing unit 120 synthesizes the first image data and the second image data. Here, after correcting the positional deviation between the first image data and the second image data based on the entire frame motion vector calculated in step S70, the first image data after the positional deviation correction and the first image data are corrected. A simple synthesis process for synthesizing the two image data with the same synthesis ratio (synthesis ratio α = 0.5) is performed. The details of the process of combining the image data after correcting the positional deviation based on the motion vector of the entire frame will be described later using the flowchart shown in FIG.

  In step S90, the image processing unit 107 performs various image processing such as white balance correction processing, synchronization processing, color reproduction processing, and noise reduction processing on the composite image data generated in step S80.

  On the other hand, in step S100, the moving object tracking unit 121 acquires the image data of the tracking area from the first image data read in step S30. Here, the image data of the tracking area acquired from the first image data is referred to as first tracking area data. Details of processing for obtaining tracking area data from image data will be described with reference to the flowchart shown in FIG.

  As described with reference to FIG. 6, the photographer can designate a subject for which tracking processing is desired on the image displayed on the LCD 114. In step S810, information on the subject of interest designated by the photographer is acquired. Here, pixel information at a position designated by the photographer on the image is acquired.

  In step S820, tracking area setting information is read. The photographer can determine the size of the tracking area after designating the subject of interest on the image. Specifically, in consideration of the size of the subject of interest on the image, a tracking area frame having an appropriate size is selected from a plurality of tracking area frames prepared in advance. It is preferable that the tracking area frame to be selected has a size including the target subject to be tracked. In step S820, the setting information of the tracking area frame selected by the photographer is read.

  In step S830, the tracking area frame selected by the photographer is set around the subject of interest specified by the photographer, and image information in the set tracking area is acquired. FIG. 9 is a diagram showing a tracking area 91 including the target subject 90 designated by the photographer. In this case, image information in the tracking area 91 is acquired.

  Returning to the flowchart shown in FIG. In step S110, the moving object tracking unit 121 acquires the image data of the tracking area from the second image data read in step S60. Here, this image data is referred to as second tracking area data. About the process which acquires tracking area data from image data, it acquires by the method similar to the flowchart shown in FIG.

  When the tracking area data is acquired from the second image data, the first tracking area is the motion vector calculation area 35 in FIG. 3, and a predetermined size (for example, ± 100 pixels) based on the first tracking area. By considering the region set in step 3 as the comparison region 34 in FIG. 3, the second tracking region may be obtained from these motion vectors to obtain the second tracking region data.

  In step S120, the local motion vector calculation unit 30 of the motion vector calculation unit 119 calculates a local motion vector between the first tracking region data acquired in step S100 and the second tracking region data acquired in step S110. calculate.

  In step S130, the synthesis processing unit 120 synthesizes the first tracking area data and the second tracking area data. Here, after correcting the positional deviation between the first tracking area data and the second tracking area data based on the local motion vector calculated in step S120, the first tracking area data after the positional deviation correction is performed. Then, weighted combining is performed to combine the second tracking area data with a combining ratio corresponding to each weight. Details of the process of combining the image data after correcting the positional deviation based on the local motion vector will be described later with reference to the flowchart shown in FIG.

  In step S140, the image processing unit 107 performs various image processing such as white balance correction processing, synchronization processing, color reproduction processing, and noise reduction processing on the composite image data generated in step S130.

  In step S150, the synthesis processing unit 120 synthesizes the synthesized image of the entire area subjected to the image processing in step S90 and the synthesized image of the tracking area subjected to the image processing in step S140. Here, the composite image of the tracking region that has been subjected to image processing in step S140 is enlarged so as to have a predetermined size, and then combined at a predetermined position of the composite image obtained in step S90.

  FIG. 10 is a diagram illustrating an example of an image 100 obtained by combining the combined image of the entire area subjected to image processing in step S90 and the combined image of the tracking area subjected to image processing in step S140. It is. In FIG. 10, the tracking target is the car 101, and the area 102 including the car is set as the tracking area. The image in the tracking area 102 is enlarged so as to have a predetermined size, and the enlarged image 103 is synthesized at the lower right position of the image 100. The composition ratio of the enlarged image 103 at the composition position is 1.

  In step S160, it is determined whether the composite image data is moving image data or live view image data. Here, the moving image data is moving image data to be recorded on the recording medium 112. If it is determined that the data is moving image data, the process proceeds to step S170.

  In step S170, the image compression / decompression unit 110 compresses the composite image data according to a predetermined moving image compression format. In step S180, a header necessary for composing a moving image file is added to the compressed data, and then recorded on the recording medium 112 via the memory I / F 111.

  Note that after determining that the composite image data is moving image data in step S160, the composite image data may be displayed on the LCD 114 after being subjected to display image processing.

  If it is determined in step S160 that the composite image data is live view image data, the process proceeds to step S190. In step S190, image processing for live view display, such as processing for changing the image size for live view display, to the composite image data is performed. In step S200, the image data on which image processing for live view display has been performed is displayed on the LCD 114.

  FIG. 11 is a flowchart showing details of the process in step S80 of the flowchart shown in FIG. 7, that is, the process of combining the image data after correcting the positional deviation based on the motion vector of the entire frame.

  In step S1100, the motion vector (x, y) of the entire frame is acquired. For example, the motion vector of the entire frame calculated by the entire frame motion vector calculation unit 29 of the motion vector calculation unit 119 is stored in the SDRAM 106, and the motion vector (x, y) of the entire frame is read from the SDRAM 106.

  In step S1110, parameters i and j indicating the position coordinates of the processing target pixel are initialized to 0, respectively. The parameter i represents the position coordinate in the x-axis direction of the image data, and the parameter j represents the position coordinate in the y-axis direction.

  In step S1120, the pixel value of pixel (i, j) is read from the first image data.

  In step S1130, in the second image data, it is determined whether or not a pixel (i + x, j + y) obtained by shifting the pixel (i, j) by the motion vector (x, y) exists on the second image data. . If it is determined that the pixel (i + x, j + y) exists on the second image data, the process proceeds to step S1140. If it is determined that the pixel (i + x, j + y) does not exist, the process proceeds to step S1150.

  In step S1140, the pixel value of the pixel (i + x, j + y) is read from the second image data.

  On the other hand, in step S1150, the pixel value of the pixel (i + x, j + y) of the second image data is set to (0, 0).

  In step S1160, when the pixel value of the second image data is (0, 0), the composition ratio α is set to 1.0, and otherwise, the composition ratio α is set to 0.5. A simple synthesis process is performed in which the pixel (i, j) of the image data and the pixel (i + x, j + y) of the second image data are synthesized at the synthesis ratio α.

  In step S1170, 1 is added to the parameter i.

  In step S1180, it is determined whether the parameter i is equal to or greater than the number of pixels M in the x-axis direction of the first image data. If it is determined that the parameter i is less than M, the process returns to step S1120. If it is determined that the parameter i is greater than or equal to M, the process proceeds to step S1190.

  In step S1190, parameter i is set to 0, and 1 is added to parameter j.

  In step S1200, it is determined whether the parameter j is equal to or greater than the number N of pixels in the y-axis direction of the first image data. If it is determined that the parameter j is less than N, the process returns to step S1120. If it is determined that the parameter j is greater than or equal to N, the process of the flowchart ends.

  FIG. 12 is a flowchart showing details of the process of step S130 in the flowchart shown in FIG. 7, that is, the process of combining the image data after correcting the displacement based on the local motion vector. Steps for performing the same processing as the processing in the flowchart shown in FIG. 11 are given the same reference numerals, and detailed description thereof is omitted.

  In step S1110, when the parameters i and j indicating the position coordinates of the processing target pixel are initialized to 0, the process proceeds to step S1210. In step S1210, the motion vector (x, y) of the pixel (i, j) is acquired. However, the pixel (i, j) is a pixel in the tracking area data. For example, the motion vector of each pixel calculated by the local motion vector calculation unit 30 of the motion vector calculation unit 119 is stored in the SDRAM 106, and the motion vector (x, y) of the pixel (i, j) is read from the SDRAM 106. .

  In step S1120A following step S1210, the pixel value of the pixel (i, j) is read from the first tracking area data.

  In step S1130A, in the second tracking area data, it is determined whether or not a pixel (i + x, j + y) obtained by shifting the pixel (i, j) by the motion vector (x, y) exists on the second tracking area data. judge. If it is determined that the pixel (i + x, j + y) exists on the second tracking area data, the process proceeds to step S1140A. If it is determined that the pixel (i + x, j + y) does not exist, the process proceeds to step S1150A.

  In step S1140A, the pixel value of the pixel (i + x, j + y) is read from the second tracking area data.

  In step S1150A, the pixel value of the pixel (i + x, j + y) of the second tracking area data is set to (0, 0).

  In step S1160A, the pixel (i, j) of the first tracking area data and the pixel (i + x, j + y) of the second tracking area data are synthesized. As described above, weighted composition is performed in step S130 of the flowchart shown in FIG. Detailed processing of the weighted synthesis will be described later with reference to the flowchart shown in FIG.

  The processing after step S1170 is the same as the processing of the flowchart shown in FIG.

  Detailed processing of weighted synthesis will be described with reference to the flowchart shown in FIG. In step S1310 of the flowchart shown in FIG. 13, the correlation coefficient value K of the processing target pixel (i, j) is acquired.

  In step S1320, the synthesis ratio α is calculated based on the correlation coefficient value K acquired in step S1310.

  FIG. 14 is a diagram showing the relationship between the correlation coefficient value K and the synthesis ratio α. In FIG. 14, TH is a predetermined threshold value, and K_max is a maximum value that can be taken as the correlation coefficient value K. As shown in FIG. 14, when the correlation coefficient value K is equal to or smaller than a predetermined threshold value TH, the synthesis ratio α is 0.5, and when the correlation coefficient value K becomes larger than the predetermined threshold value TH, The synthesis ratio α is also larger than 0.5. However, when the pixel value of the second image data is (0, 0), the composition ratio α is set to 1.0 regardless of the correlation coefficient value K.

  The flash memory 117 stores a table that defines the relationship between the correlation coefficient value K and the composition ratio α, and the composition ratio α is calculated by referring to this table.

  In step S1330, the pixel (i, j) of the first tracking area data and the pixel (i + x, j + y) of the second tracking area data are synthesized by the following equation (2). However, A is the pixel value of the pixel (i, j) of the first tracking area data, B is the pixel value of the pixel (i + x, j + y) of the second tracking area data, and C is the pixel of the combined pixel Value.

C = α × A + (1−α) × B (2)
As can be seen from FIG. 14 and Expression (2), the correlation between the pixel (i, j) of the first tracking area data and the pixel (i + x, j + y) of the second tracking area data is high, and the correlation When the numerical value K is less than or equal to the predetermined threshold value TH, simple composition processing is performed in which composition is performed with the same composition ratio (composition ratio α = 0.5). On the other hand, when the correlation coefficient value K becomes larger than the predetermined threshold value TH, the specific gravity for synthesizing the pixel (i, j) of the first image data is increased, and the pixel (i + x, j + y) of the second tracking area data is increased. ).

  FIG. 15 is a diagram illustrating an example of a composite image in a case where the composition process is performed in a state in which the positional deviation of the image of the automobile remains even after the positional deviation between the two image data is corrected. FIG. 15A shows the result when simple synthesis is performed at a synthesis ratio α = 0.5, and FIG. 15B shows the result when weighted synthesis is performed at a synthesis ratio α greater than 0.5. FIG. 15 (c) shows the result when synthesis is performed at a synthesis ratio α = 1. As can be seen by comparing FIG. 15A and FIG. 15B, when weighted composition is performed at a composition ratio α greater than 0.5, the image is compared to when simple composition is performed. Blur is reduced. When the synthesis ratio α is set to 1, only image data with the synthesis ratio α set to 1 is used, and image blur is eliminated.

  As described above, according to the imaging apparatus in the first embodiment, the positional deviation between a plurality of images is detected for the entire image area of the image data, and a plurality of the entire image area is targeted based on the detected positional deviation. The first image is generated by correcting the positional deviation between the images. In addition, a position shift between a plurality of images is detected by a position shift detection method different from the position shift detection method for a part of the image area, and based on the detected position shift. Then, the positional deviation between the plurality of images is corrected for a part of the image region, and the second composite image is generated. Then, the first composite image and the second composite image are displayed on the same screen. As a result, it is possible to detect misalignment for a part of the image area including the subject of interest using a method different from the misalignment detection for the entire image area. It can be performed.

  In particular, according to the imaging apparatus of the first embodiment, the accuracy of misregistration detection for a part of the image region is higher than the accuracy of misregistration detection for the entire image region. It is possible to accurately perform misalignment of some of the included image areas. In addition, since the position shift of the entire image area is performed using a method that is less accurate than the position shift detection of a part of the image area, the amount of calculation for position shift detection can be reduced and the time required for image display can be shortened. Can do.

-Second Embodiment-
In the imaging apparatus according to the first embodiment, the entire frame motion vector calculation unit 29 excludes motion vectors determined to have low reliability from the motion vectors calculated for each of a plurality of blocks of the evaluation frame. The average vector of motion vectors was used as the motion vector for the entire frame. In the imaging device according to the second embodiment, the motion vector of the entire frame is increased by increasing the weight of the motion vector of the block including the tracking region among the other motion vectors excluding the motion vector determined to have low reliability. Is calculated. Here, the code of the entire frame motion vector calculation unit in the second embodiment is 29A.

  FIG. 16A is a diagram illustrating a tracking area in the evaluation frame, and FIG. 16B is a diagram illustrating an example of a motion vector of a block including the tracking area. However, FIGS. 16A and 16B show a state where an image is divided into a plurality of blocks.

  In FIG. 16A, the tracking target by the moving object tracking unit 121 is a moving vehicle, and the regions 130 and 131 including the vehicle are tracking regions. In FIG. 16A, for the sake of explanation, the tracking area 130 including the car at the first time and the tracking area 131 including the car at the time after the first time are also shown on one image. .

  Similar to the entire frame motion vector calculation unit 29 in the first embodiment, the entire frame motion vector calculation unit 29A is determined to have low reliability among the motion vectors calculated for each of the plurality of blocks of the evaluation frame. The motion vector of the entire frame is calculated using another motion vector excluding the motion vector. However, the entire frame motion vector calculation unit 29A increases the weight of the motion vector of the block including the tracking region, and adds the weighted average of all other motion vectors excluding the motion vector determined to be low in reliability. The operation is performed, and the motion vector obtained by the operation is set as the motion vector of the entire frame.

  In addition, when calculating a vector using a histogram, it is possible to calculate a motion vector of the entire frame in which the specific gravity of the tracking area is increased by inflating the frequency in the motion vector of the block including the tracking area and counting. it can. In the example shown in FIG. 16, the weights of the motion vectors of the six blocks 132 to 137 (see FIG. 16B) including the tracking areas 130 and 131 are high.

  As described above, according to the imaging apparatus in the second embodiment, when detecting a positional shift for the entire image region, a plurality of positional shifts corresponding to a plurality of positions on the image are detected, and the detected plurality Among the misregistrations, the specific gravity of the misregistration in some image areas is set higher than the specific gravity of the misregistration in areas other than the partial image area, and one representative misregistration is detected. As a result, even when correcting the positional deviation of the entire image region, it is possible to perform the positional deviation correction with an emphasis on the subject of interest, so that the image blur of the subject of interest can be reduced in the composite image.

-Third embodiment-
In the imaging apparatus according to the first embodiment, when detecting a positional shift between a plurality of images for the entire image region, the entire frame motion vector calculation unit 29 calculates the motion vector of the entire frame. In the imaging apparatus according to the third embodiment, even when the positional deviation between a plurality of images is detected for the entire image region, the movement amount of the subject that is the tracking target of the moving object tracking unit 121 is greater than the predetermined amount Lth. First, in order to perform highly accurate alignment, the local motion vector calculation unit 30 calculates a local motion vector.

  FIG. 17 is a flowchart illustrating a procedure of a composite image generation process performed by the imaging device according to the third embodiment. Steps for performing the same processing as the processing in the flowchart shown in FIG. 7 are assigned the same reference numerals, and detailed descriptions thereof are omitted.

  The processing from step S1700 to step S90 is processing for the entire image area. In step S1700, the microcomputer 115 calculates the movement amount of the subject that is the tracking target. The moving body tracking unit 121 calculates a motion vector of a subject to be tracked in order to perform a moving body tracking process. Accordingly, in step S1700, the amount of movement of the subject is calculated based on the motion vector calculated by the moving object tracking unit 121.

  In step S1710, the microcomputer 115 determines whether or not the movement amount of the subject calculated in step S1700 is larger than a predetermined amount Lth. If it is determined that the movement amount of the subject is equal to or less than the predetermined amount Lth, the process proceeds to step S70, and the first image data read in step S30 and the second image read in step S60 by the entire frame motion vector calculation unit 29. The motion vector of the whole frame between data is calculated. On the other hand, if it is determined that the amount of movement of the subject is greater than the predetermined amount Lth, the process proceeds to step S1720.

  In step S1720, the local motion vector calculation unit 30 of the motion vector calculation unit 119 calculates a local motion vector between the first image data read in step S30 and the second image data read in step S60. . Since the calculation method of the local motion vector has already been described, detailed description thereof is omitted here.

  In step S1730, after correcting the positional deviation between the first image data and the second image data based on the local motion vector calculated in step S1720, the first image data after the positional deviation correction and the first image data are corrected. The weighted composition is performed to synthesize the image data of 2 at a composition ratio corresponding to each weight. Since the details of the process of combining the image data after correcting the positional deviation based on the local motion vector have been described using the flowchart shown in FIG. 12, detailed description thereof is omitted here.

  As described above, according to the imaging apparatus of the third embodiment, when the amount of movement of the subject in a part of the image area is larger than the predetermined amount Lth, the positional deviation detection accuracy is higher than when the movement amount is equal to or smaller than the predetermined amount Lth. The position shift between a plurality of images is detected by a high method. As a result, even when the amount of movement of the subject of interest is large and the image blur on the composite image is conspicuous, it is possible to accurately obtain the positional deviation between the plurality of images and reduce the image blur of the subject on the composite image. Can do.

-Fourth Embodiment-
In the imaging apparatus according to the first embodiment, when detecting a positional shift between a plurality of images for the entire image region, the entire frame motion vector calculation unit 29 calculates the motion vector of the entire frame. In the imaging device according to the fourth embodiment, even when the positional deviation between a plurality of images is detected for the entire image region, if the size S of the tracking region is larger than the predetermined size Sth, high accuracy is obtained. In order to perform accurate alignment, the local motion vector calculation unit 30 calculates a local motion vector.

  FIG. 18 is a flowchart illustrating a procedure of a composite image generation process performed by the imaging device according to the fourth embodiment. Steps for performing the same processes as those in the flowcharts shown in FIGS. 7 and 17 are denoted by the same reference numerals, and detailed description thereof is omitted.

  The processing from step S1800 to step S90 is processing for the entire image area. In step S1800, the microcomputer 115 acquires the size S of the tracking area set by the moving body tracking unit 121. As described above, the photographer designates the target subject to be tracked on the image and selects a tracking area frame having an appropriate size from a plurality of tracking area frames prepared in advance. To do. Here, information on the size S of the tracking area frame selected by the photographer is acquired.

  In step S1810, the microcomputer 115 determines whether or not the size S of the tracking area calculated in step S1800 is larger than a predetermined size Sth. If it is determined that the size S of the tracking area is equal to or smaller than the predetermined size Sth, the process proceeds to step S70, and the first image data read in step S30 and the first image data read in step S60 by the entire frame motion vector calculation unit 29. The motion vector of the entire frame between the two image data is calculated. On the other hand, if it is determined that the size S of the tracking area is larger than the predetermined size Sth, the process proceeds to step S1720. The processing in step S1720 and step S1730 is the same as the processing in step S1720 and step S1730 in the flowchart shown in FIG.

  As described above, according to the imaging device of the fourth embodiment, when the size of a part of the image area is larger than the predetermined size Sth, the size of the part of the image area is equal to or less than the predetermined size Sth. A positional deviation between a plurality of images is detected by a method with higher positional deviation detection accuracy than the case. As a result, even when the subject of interest is large and the image blur on the composite image is conspicuous, it is possible to accurately obtain the positional deviation between the plurality of images and reduce the image blur of the subject on the composite image.

  In the description of the first to fourth embodiments described above, hardware processing is assumed as processing performed by the imaging apparatus, but it is not necessary to be limited to such a configuration. For example, a configuration in which processing is performed separately by software is also possible. In this case, the imaging apparatus includes a main storage device such as a CPU and a RAM, and a computer-readable storage medium in which a program for realizing all or part of the above processing is stored. Here, this program is called an imaging program. Then, the CPU reads out the imaging program stored in the storage medium and executes information processing / calculation processing, thereby realizing the same processing as that of the imaging device described above.

  Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, and the like. Further, the imaging program may be distributed to a computer via a communication line, and the computer that has received the distribution may execute the imaging program.

  The present invention is not limited to the first to fourth embodiments described above, and various modifications and applications can be made without departing from the scope of the present invention. For example, the method for calculating the entire frame motion vector described in the second embodiment can also be applied to the imaging devices in the third and fourth embodiments.

  Moreover, you may combine the characteristic in 3rd Embodiment, and the characteristic in 4th Embodiment. That is, when the moving amount of the subject is larger than the predetermined amount Lth or the size S of the tracking area is larger than the predetermined size Sth, the local motion vector may be obtained as the motion vector between the image data. Good.

  In the imaging apparatus according to the third embodiment, the local motion vector is obtained as the motion vector between the image data when the moving amount of the subject to be tracked is larger than the predetermined amount Lth. However, the local motion vector is limited to the local motion vector. Will never be done. That is, it suffices if the motion vector can be obtained by a method with higher accuracy than when the movement amount of the subject to be tracked is equal to or less than the predetermined amount Lth.

  Similarly, in the imaging device according to the fourth embodiment, when the size of the tracking area is larger than the predetermined size Sth, the local motion vector is obtained as the motion vector between the image data. If the motion vector is obtained by a method with higher accuracy than the case where is less than or equal to the predetermined size Sth, it is not limited to the local motion vector.

  In each of the embodiments described above, the process of enlarging the combined image data in the tracking area and displaying the enlarged image is superimposed on the combined image data in the entire image area (see FIG. 10). However, the synthesized image data of the entire image area may be synthesized without enlarging the synthesized image data of the tracking area. The synthesis position is the position of the tracking area, and the synthesis ratio of the tracking area image data at the synthesis position is 1. Further, the combined image data of the tracking area is displayed with a rectangular frame, for example. FIG. 19 is a diagram illustrating an example of a method for displaying a combined image of the tracking area so as to be superimposed on the combined image of the entire area without being enlarged. A synthesized image 192 of the tracking area including the car 191 that is the tracking target is synthesized with the synthesized image 190 of the entire area, and the synthesized image 192 is displayed surrounded by a rectangular frame 193.

DESCRIPTION OF SYMBOLS 1 ... Camera body 2 ... Interchangeable lens 101 ... Mechanical shutter 102 ... Imaging element 103 ... Analog processing part 104 ... Analog / digital conversion part 106 ... SDRAM
107 ... Image processing unit 108 ... AE processing unit 109 ... AF processing unit 110 ... Image compression / decompression unit 112 ... Recording medium 114 ... LCD
115... Microcomputer 117... Flash memory 119... Motion vector calculation unit 120... Synthetic processing unit 121 .. Motion tracking unit 29.

Claims (8)

An imaging device capable of generating a composite image by combining a plurality of images obtained by imaging,
An imaging unit that obtains image data by photoelectrically converting subject light received by the imaging device;
An image area selection unit that selects a part of the image area of the image data;
A first misregistration detector that detects misregistration between the plurality of images for the entire image region;
Second misregistration detection for detecting misregistration between the plurality of images by a misregistration detection method different from the misregistration detection method by the first misregistration detection unit for the partial image region. And
A first misregistration correction unit that corrects misregistration between the plurality of images for the entire image region based on the misregistration detected by the first misregistration detection unit;
A second misregistration correction unit that corrects misregistration between the plurality of images for the partial image region based on the misregistration detected by the second misregistration detection unit;
A first combining unit that generates a first combined image by combining a plurality of images whose positional deviation is corrected for the entire image region;
A second combining unit that generates a second combined image by combining a plurality of images in which misalignment is corrected for the partial image region;
A display unit for displaying the first composite image and the second composite image;
Equipped with a,
An image pickup apparatus characterized in that the positional deviation detection accuracy of the second positional deviation detection unit is higher than the positional deviation detection accuracy of the first positional deviation detection unit . The first misregistration detection unit detects a plurality of misregistrations according to a plurality of positions on the image, and calculates one representative misregistration based on the detected plurality of misregistrations. When calculating one representative positional deviation, the specific gravity of the positional deviation in the partial image area is set higher than the specific gravity of the positional deviation in the area other than the partial image area. The imaging apparatus according to claim 1 . An imaging device capable of generating a composite image by combining a plurality of images obtained by imaging,
An imaging unit that obtains image data by photoelectrically converting subject light received by the imaging device;
An image area selection unit that selects a part of the image area of the image data;
A first misregistration detector that detects misregistration between the plurality of images for the entire image region;
Second misregistration detection for detecting misregistration between the plurality of images by a misregistration detection method different from the misregistration detection method by the first misregistration detection unit for the partial image region. And
A first misregistration correction unit that corrects misregistration between the plurality of images for the entire image region based on the misregistration detected by the first misregistration detection unit;
A second misregistration correction unit that corrects misregistration between the plurality of images for the partial image region based on the misregistration detected by the second misregistration detection unit;
A first combining unit that generates a first combined image by combining a plurality of images whose positional deviation is corrected for the entire image region;
A second combining unit that generates a second combined image by combining a plurality of images in which misalignment is corrected for the partial image region;
A display unit for displaying the first composite image and the second composite image;
A movement amount detection unit that detects a movement amount of a subject in the partial image region based on the plurality of images;
With
When the movement amount detected by the movement amount detection unit is larger than a predetermined amount, the first positional deviation detection unit uses a method with higher positional deviation detection accuracy than when the movement amount is equal to or less than the predetermined amount. An image pickup apparatus that detects a displacement between the plurality of images . An imaging device capable of generating a composite image by combining a plurality of images obtained by imaging,
An imaging unit that obtains image data by photoelectrically converting subject light received by the imaging device;
An image area selection unit that selects a part of the image area of the image data;
A first misregistration detector that detects misregistration between the plurality of images for the entire image region;
Second misregistration detection for detecting misregistration between the plurality of images by a misregistration detection method different from the misregistration detection method by the first misregistration detection unit for the partial image region. And
A first misregistration correction unit that corrects misregistration between the plurality of images for the entire image region based on the misregistration detected by the first misregistration detection unit;
A second misregistration correction unit that corrects misregistration between the plurality of images for the partial image region based on the misregistration detected by the second misregistration detection unit;
A first combining unit that generates a first combined image by combining a plurality of images whose positional deviation is corrected for the entire image region;
A second combining unit that generates a second combined image by combining a plurality of images in which misalignment is corrected for the partial image region;
A display unit for displaying the first composite image and the second composite image;
A size detector for detecting the size of the partial image region based on the plurality of images;
With
The first misregistration detection unit is configured such that when the size of the partial image area detected by the size detection unit is larger than a predetermined size, the size of the partial image region is the predetermined level. An image pickup apparatus , wherein a position shift between the plurality of images is detected by a method having a higher position shift detection accuracy than a case of a size or less . A moving body tracking unit that tracks moving objects on an image is further provided.
The portion of the image area, the image pickup apparatus according to any one of claims 1 to 4, a part of the moving object to be tracked by the moving body tracking portion is characterized in that the area included least. The portion of the image area, the image pickup apparatus according to any one of claims 1 to 4, characterized in that an area for displaying an enlarged. The imaging apparatus according to claim 6 , wherein the display unit superimposes and displays an enlarged image of the second composite image on the first composite image. An imaging method using an imaging device capable of generating a composite image by combining a plurality of images obtained by imaging,
Selecting a part of the image area of the image data,
A first misalignment detection step for detecting misalignment between the plurality of images for the entire image area;
Second misregistration detection for detecting misregistration between the plurality of images by a misregistration detection method different from the misregistration detection method in the first misregistration detection step for the partial image region. Steps,
Correcting the positional deviation between the plurality of images for the entire image region based on the positional deviation detected in the first positional deviation detection step;
Correcting the positional deviation between the plurality of images for the partial image region based on the positional deviation detected in the second positional deviation detection step;
Synthesizing a plurality of images whose positional deviation is corrected for the entire image region to generate a first synthesized image;
Synthesizing a plurality of images in which the positional deviation is corrected for the partial image region, and generating a second synthesized image;
Displaying the first composite image and the second composite image;
Equipped with a,
The imaging method , wherein the positional deviation detection accuracy of the second positional deviation detection step is higher than the positional deviation detection accuracy of the first positional deviation detection step .