JP5869839B2 - Image processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an image processing apparatus for processing a plurality of images obtained from different viewpoints, and a control method therefor.

  In recent years, there has been proposed a multi-view imaging system in which a plurality of cameras are arranged two-dimensionally, and the same target object is simultaneously captured from various viewpoints to obtain image groups having different viewpoint positions (Patent Document 1). There has also been proposed a light field camera that simultaneously captures a plurality of images with different angles of view at various angles of view. Research on multi-view video technology for the purpose of recording and transmitting images with a high degree of realism using a group of images photographed by such a photographing system is underway.

  In the case of a plurality of viewpoint images photographed by the above-described photographing device, supplementary information may be included between the images, the amount of image data becomes enormous, and there is information redundancy. Therefore, if the image data is used for image processing as it is or transmitted, the image processing becomes complicated or high-rate video transmission cannot be performed. Therefore, it is necessary to compress and encode a plurality of viewpoint images (hereinafter also referred to as multiviews).

JP 2010-157826 A

  In the case of a multi-viewpoint image, an image with the number of viewpoints is obtained, so even if compression encoding is performed, the code amount is proportional to the number of viewpoints. One solution is to exclude images at several viewpoint positions from being stored. However, in this case, there will be a restriction related to post-processing by the user (for example, a high-quality image having a viewpoint position designated by the user cannot be generated).

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for reducing encoded data while enabling satisfactory post-processing of a multi-viewpoint image.

In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is,
An image processing apparatus that has M × N imaging units, encodes images at different viewpoint positions obtained from each of the imaging units, and stores the images in a predetermined recording medium,
A setting means for setting whether to set a code amount suppression mode for suppressing a code amount or a non-code amount suppression mode;
When the code amount suppression mode for suppressing the code amount is set by the setting unit, a determination unit that determines whether the shooting mode at the time of shooting is a foreground shooting mode or a distant shooting mode;
When the non-code amount suppression mode is set by the setting unit, all images from the M × N imaging units are determined as storage targets,
When the determination unit determines that the camera is in the distant view shooting mode, among the images from the M × N imaging units, the image from the central imaging unit and all the images from the M × N imaging units Of these , images from I (I <M × N) imaging units including positions that can be the maximum parallax are determined as storage targets,
The determination means when it is determined that the near view imaging mode, the out of the M × N of the imaging unit images, the image from the imaging unit in the center, of the I-number of the storage target in the distant view image capturing mode An image of the imaging unit, and a determination unit that determines, as a storage target , images from J imaging units located closer to the imaging unit located in the central part than the I imaging units to be stored in the distant shooting mode ,
Encoding means for encoding an image in the imaging unit determined by the determining means as a storage target image;
Storage means for storing the encoded data obtained by encoding in a predetermined memory.

  According to the present invention, it is possible to reduce encoded data while enabling satisfactory post-processing of a multi-viewpoint image.

The figure which shows an example of the imaging device of a multi-view system provided with a some imaging part. The block diagram which shows the internal structure of an imaging device. 6 is a flowchart of image processing according to the first embodiment. FIG. 3 is a diagram illustrating an example of an image group acquired by the multi-view imaging apparatus according to the first embodiment. FIG. 6 is a diagram showing an example of image selection in a distant view shooting mode according to the first embodiment. FIG. 6 is a diagram illustrating an example of image selection in a foreground shooting mode according to the first embodiment. The figure which shows an example of the center image which concerns on 1st Embodiment. 10 is a flowchart of image processing according to the second embodiment. The figure which shows an example of the imaging device of a light field camera. 10 is a flowchart of image processing according to the third embodiment. 10 is a flowchart of image processing according to the fourth embodiment. 10 is a flowchart of image processing according to a fifth embodiment.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the components shown in each embodiment described below are merely examples, and the scope of the technology of the present invention is determined by the claims, and is limited by the following individual embodiments. I don't mean.

[First Embodiment]
FIG. 1 is a diagram illustrating an example of an image processing apparatus according to the first embodiment of the present invention, that is, an imaging apparatus based on a multi-view system including a plurality of imaging units.

  The imaging device 100 is a housing and includes nine imaging units 101 to 109 that acquire images and a shooting button 110. In order to simplify the description, it is assumed that the nine imaging units 101 to 109 are evenly arranged on a square lattice.

  When the user presses the shooting button 110, each sensor (imaging device) provided in the imaging units 101 to 109 receives the subject image. The analog video signal obtained by receiving light is A / D converted to generate a plurality of images (digital data). With such a multi-lens imaging device, it is possible to obtain an image group in which the same subject is simultaneously imaged from a plurality of viewpoint positions. Although the number of imaging units is nine here, the number of imaging units is not limited to nine. The present invention is applicable as long as the imaging apparatus has a plurality of imaging units. In addition, although an example in which nine imaging units are equally arranged on a square lattice has been described here, the arrangement of the imaging units is arbitrary. For example, they may be arranged radially or linearly, or may be arranged at random.

<Configuration of photographing apparatus>
FIG. 2 is a block configuration diagram in the image processing apparatus according to the first embodiment of the present invention.

  A CPU (central processing unit; processor) 201 comprehensively controls each unit described below. The imaging unit 202 acquires image data. The photographing unit 202 includes photographing units 101 to 109 in the photographing apparatus of FIG. The bus 203 is a communication path serving as a transfer path for various data, and digital data acquired by the imaging unit 202 is sent to a predetermined processing unit via the bus 203. A display unit 204 displays captured images and characters. For example, a liquid crystal display is used for this. The display unit 204 may have a touch screen function. A display control unit 205 performs display control of a captured image and characters displayed on the display unit 204. The operation unit 206 receives a user instruction. The operation unit 206 includes various buttons including the shooting button 110, a shooting mode dial, and the like. For example, at the time of shooting, a button for the user to specify whether to select the far-field shooting mode or the near-field shooting mode is included, and this is operated to instruct predetermined processing via the CPU 201.

  The imaging control unit 207 controls the imaging system based on an instruction from the CPU 201 such as focusing, opening / closing a shutter, and adjusting a diaphragm. The digital signal processing unit 208 performs various processes such as white balance processing, gamma processing, and noise reduction processing on the digital data received via the bus 203. The image processing unit 209 is an image group acquired by the imaging unit 202, an image group output from the digital signal processing unit 208, or an image required according to the user's designated shooting mode input from the operation unit 206. Select data and perform image processing. The compression / decompression unit 210 performs processing such as conversion of digital data into a file format such as JPEG or MPEG, or image selection and encoding control from the operation unit 206 in accordance with a user-specified shooting mode.

  The internal memory 211 functions as a main memory, work area, and the like of the CPU 201, and stores a control program executed by the CPU 201 and the like. The external memory control unit 212 is an interface for connecting to a removable storage medium (for example, hard disk, memory card, CF card, SD card, USB memory).

<Image processing flow>
The image processing according to the present embodiment relates to the CPU 201, the image capturing unit 202, the data transfer bus unit 203, the operation unit 206, the image processing unit 209, and the internal memory 211 in the image capturing apparatus.

  FIG. 3A is a diagram illustrating a flowchart of image processing according to the present embodiment.

  First, in S10, the CPU 201 detects the operation of a dial or switch provided on the operation unit 206 by the user, and sets ON / OFF of the code amount suppression mode at the time of shooting with the multi-lens imaging device (or Either the code amount suppression mode or the non-code amount suppression mode is set). The CPU 201 stores this setting content in the internal memory 211. When the code amount suppression mode is ON, some of the images of all viewpoints of the multi-view imaging device are excluded from the storage target, and the amount of data to be stored is reduced (details will be described later). On the other hand, when the code amount suppression mode is OFF, the images of all viewpoints possessed by the multi-lens imaging device are the storage targets.

  In the next S11, the CPU 201 refers to the internal memory 211 and confirms ON / OFF of the code amount suppression mode. That is, when the code amount suppression mode is OFF, all the viewpoints are set as the shooting target images in S12 and the setting contents are stored in the internal memory 211 in S12 without depending on the foreground / distant modes of the shooting mode. .

  On the other hand, if the code amount suppression mode is ON, the process proceeds to S13, and the shooting mode is set by the operation unit 206. The shooting mode includes a near view mode in which the object is relatively close and a distant view mode in which the object is far away, and the user selects one of them. The set shooting mode is stored in the internal memory 211. Note that the foreground mode is a mode suitable for photographing portraits of people and the like, and the distant view mode is a mode for photographing landscapes and the like. In S14, the internal memory 211 is referred to, and it is determined which of the foreground mode and the distant view mode is selected as the shooting mode.

  If it is determined that the camera is in the distant view mode, the process proceeds to S15 and image selection processing 1 is executed. On the other hand, if it is determined that the mode is not the distant view mode, that is, if it is determined that the mode is the foreground mode, the process proceeds to S16 and the image selection process 2 is executed. The image selection processes 1 and 2 are processes for selecting a viewpoint position at which the image to be stored is captured from all viewpoints, and the selection contents are stored in the internal memory 211.

  Assume that there are M × N imaging units. At this time, in the image selection processing 1 in the distant view shooting mode, images from the I imaging units including the four corners in the M × N imaging units are determined as storage targets. On the other hand, in the image selection processing 2 in the foreground shooting mode, among the M × N image pickup units, J (where I <J <M × N), which is shorter than the interval between the image pickup units to be stored in the foreground shooting mode. ) Is determined as a storage target.

  In the above embodiment, the code amount suppression mode is set to ON / OFF and then the shooting mode is set. However, the present invention is not limited to this. In short, it is only necessary to be able to discriminate between ON / OFF of the code amount suppression mode and the distant view mode and foreground mode when the code amount suppression mode is ON.

  FIG. 3B shows processing when the photographing button 110 of the operation unit 206 is pressed. If the CPU 201 detects that the photographing button 110 in the operation unit 206 is pressed in S17, the process proceeds to S18. In S18, each imaging unit mounted on the imaging unit 202 is controlled to perform imaging processing. Next, in S19, only the image at the selected viewpoint position stored in the internal memory 211 is encoded, multiplexed and stored in one image file.

  A specific example of the above processing procedure will be described below. As in the embodiment, an image group that captures the same subject from a plurality of viewpoint positions can be obtained by a multi-lens imaging device. Here, the imaging apparatus according to the embodiment will be described as having 5 × 5 imaging units (5 × 5 viewpoints) that are evenly arranged on a square lattice. Since the 5 × 5 imaging unit is provided, it is possible to acquire 5 × 5 25 images as shown in FIG. When these 25 images are present, post-processing switches to a video from a free viewpoint position within the range, generates a stereo image, or in some cases generates a super-resolution image from a plurality of images. You can also. As a technique for generating this super-resolution image, “Multi frame image restoration and registration (TSAI R Y, HUANG T S.)” can be cited. However, the number of images to be stored depends on the number of viewpoints, and the amount of data increases accordingly.

  In the present embodiment, a code amount suppression mode is provided in order to meet the desire to reduce the amount of data even though there are some restrictions, although it is desired to obtain a multi-viewpoint effect. When the code amount suppression mode is turned on, the number of viewpoint images to be saved is smaller than that in the normal case (the code amount suppression mode is OFF), thereby reducing the data amount. However, it should be noted that the smaller the number of viewpoints, the less the effect of multi-viewpoints. Therefore, it is desirable to reduce the amount of data while maintaining the multi-viewpoint effect as much as possible. In the embodiment, when the code amount suppression mode is turned on, the storage target viewpoint position and the number are determined according to the shooting mode, instead of determining the storage target viewpoint position and the number uniformly.

When the distant view mode and the foreground mode are compared, it may be considered that the distance from the photographing device to the subject is longer in the distant view mode. The longer the distance to the subject, the smaller the parallax of each viewpoint image. That is, it can be said that the difference between the images by the 5 × 5 photographing units is small. Therefore, in an extreme case, it is only necessary to store the image at the viewpoint position at the center, but this does not mean that the merit of the multi-view method is utilized. The multi-view method allows various post-processing, so we want to take advantage of its advantages. Therefore, in this embodiment, as shown by the shaded portion of FIG. 5, in a 5 × 5 pieces of the imaging unit, and the viewpoint position of the four corners to achieve the maximum parallax, the viewpoint position of the five places of the center position of the imaging The viewpoint position to be saved was determined. The process of S15 in FIG. 3A described above selects the viewpoint position in FIG.

  On the other hand, in the foreground shooting mode, naturally, the distance from the shooting device to the subject is shorter than in the distance shooting, and the parallax at each viewpoint position is also increased. That is, the difference between the images at each viewpoint position is large and the correlation is high. For this reason, it is desired that a larger number of viewpoint positions be saved than in the distant view shooting mode shown in FIG. 5, and some viewpoint positions need not be saved. Therefore, in the embodiment, the viewpoint positions at intervals shorter than the distance between the storage target viewpoint positions shown in FIG. Specifically, as shown in FIG. 6, the storage target position is determined in a checkered pattern. Here, it is desirable that the central position of the multi-viewpoint is a storage target. This is because the center position is most likely to be referred to when shooting the subject. In addition, by determining the save target viewpoint position in a checkered pattern in this way, an image that can sufficiently withstand camera shake correction, refocus, and free viewpoint image generation processing is obtained in post-processing. Here, digital camera shake correction is a process of restoring an image by calculating movement information of captured image data based on the correlation between the frequency of the shake and the amount of amplitude. The refocus processing is processing for obtaining a subject depth at the time of shooting and adjusting a depth coefficient to focus on an arbitrary portion such as a subject or a background and blurring the rest. The free viewpoint image generation process is a process for obtaining an image of an arbitrary viewpoint that generates an image at an imaging position that is not captured from a captured image of the imaging unit. In the foreground shooting mode, the image processing examples described above require different image data, and therefore image selection is controlled according to the image processing.

<Example of camera shake correction processing>
Camera shake is caused by shaking of a human hand, and camera shake correction is a process of correcting a blurred portion by positioning between consecutive images, that is, between frames. Since the image acquired from the central photographing unit of the photographing apparatus is the image that the user is most likely to pay attention to, a series of images in the blur direction of the photographing camera of the central image is used as an image for image stabilization. Is common. When the selected image shown in FIG. 6 is applied to camera shake correction, for example, in the case of blurring in the vertical direction, an image acquired from the center image and cameras in the vertical direction may be selected. Alternatively, in the case of shaking in the horizontal direction, an image acquired from the center image and the cameras in the matrix direction may be selected. When the blur direction is not fixed, either the vertical direction or the horizontal direction image selection method may be used.

<Example of refocus processing>
The refocusing process is a process of estimating the depth at the time of imaging, which is also referred to as distance below, focusing on the target region based on this depth information, and blurring the rest. Depth estimation can be performed from two images with some degree of parallax, but in order to increase the accuracy of distance estimation, the image at the position where the baseline length becomes long, that is, the position where the parallax information is large, is used as the processing target image. It is common. When the selected image shown in FIG. 6 is applied to refocusing, for example, among the selected images, selected images at both ends of the diagonal line of the photographing apparatus, images at both ends in the vertical direction, or any other arbitrary image What is necessary is just to select the image in the both ends of a direction.

<Example of processing for free viewpoint image generation>
The generation of a free viewpoint image is to generate a captured image from a virtual camera of an arbitrary viewpoint using a viewpoint image acquired in advance or both a viewpoint image and a depth image. Basically, the images acquired from the photographing cameras at the four ends of the photographing device are important, and the intermediate viewpoint image information can be generated therefrom. When the selected image shown in FIG. 6 is applied to the free viewpoint generation process, for example, the central image and five images at the four ends may be used, or densely selected images may be used. .

  In the above embodiment, the code amount suppression mode is divided into ON / OFF, and when it is ON, the shooting mode is roughly divided into the distant view mode and the foreground mode, and the image selection criterion and the number of images are switched. In the case where the camera matrix is odd in both horizontal and vertical directions, an example in which an image acquired from a camera located at the center of the photographing apparatus is used as the central image has been described. When at least one of the horizontal and vertical camera matrices is an even number, one in the vicinity of the center position may be set as the male viewpoint. For example, as shown in FIG. 7, in the case of an imaging apparatus having 6 × 6 imaging units, as shown in the figure, one of the four types is treated as a central viewpoint, and the viewpoint position is set to a distant view or foreground mode. What is necessary is just to make it the viewpoint position which image | photographs a common preservation | save object image.

  In the above-described embodiment, the image selection in the foreground shooting mode includes the central viewpoint image acquired from the central imaging unit of the imaging apparatus, the four corner viewpoint images acquired from the imaging units at the four corners of the imaging apparatus, and the central imaging. A viewpoint image acquired from a photographing device at a fixed interval between the image capturing unit and the photographing unit at the four corners is selected. Here, the regular intervals may be dense or sparse.

  Further, in the above-described embodiment, generation of a super-resolution image is described as image processing in the distant view mode, and camera shake correction, refocus, and free viewpoint image generation processing are described as image processing in the near view mode. . Image processing other than the above may be mentioned.

  Furthermore, although the multi-viewpoint imaging apparatus has been described as an example in the above-described embodiment, for example, a CPU in a PC that connects a multi-viewpoint imaging unit via a USB interface or the like executes the processing described in the above-described embodiment. Therefore, the present invention is not limited to a single device such as an imaging device. This is the same in each embodiment described below.

[Second Embodiment]
In the first embodiment, the user designates either the foreground shooting mode or the foreground shooting mode by operating the operation unit 206. In the present embodiment, an example will be described in which a shooting mode includes a distant view mode, a foreground mode, and an automatic mode. The apparatus configuration is the same as that of the first embodiment, and a description thereof is omitted. In the automatic mode, the determination of the distant view and the close view of the subject is determined from the captured image. In the first embodiment, the viewpoint position corresponding to the image to be saved is determined when the shooting mode is designated. In this second embodiment, when the photographing button 110 is pressed, the viewpoint position of the image to be stored is set according to ON / OFF of the code amount suppression mode and the photographing mode, and FIGS. Alternatively, an example in which either one of all viewpoints is determined and stored will be described.

  Hereinafter, processing when the photographing button 110 is operated will be described with reference to a flowchart of FIG.

  First, in S21, it is determined with reference to the internal memory 211 whether or not the code amount suppression mode is ON. If it is OFF, the process proceeds to S22, and 5 × 5 images obtained by photographing are selected, encoded in S29, multiplexed into one file, and stored.

  On the other hand, when it is determined that the code amount suppression mode is ON, it is determined whether or not it is the automatic mode. If not, since the user has explicitly designated the foreground mode and the distant view mode, it is determined in S26 whether or not the distant view mode is selected. If the distant view mode is selected, the process proceeds to S27. The process proceeds to S28.

  In the case of the automatic mode, in S25, from the captured images of a preset number of viewpoint positions (two of the four corners of the two-dimensional array may be sufficient), the near view mode and the distant view are temporarily set. Determine the mode. In the embodiment, a depth to the subject, that is, a depth image is generated from the image, and parallax determination is performed based on a histogram distribution of pixel values (depth information) of the depth image. If the depth information histogram distribution is flat, the image has a small depth difference and a small parallax, and the shooting mode is temporarily treated as a distant view mode. Whether or not the histogram distribution is flat may be determined as flat (high flatness) when the standard deviation of the histogram distribution is obtained and the value is equal to or less than a preset threshold value. On the other hand, if the histogram distribution of the depth information is non-flat, it is determined that the image has a large depth difference and a large parallax, and the shooting mode is temporarily determined to be the foreground mode. The determination of S26 is performed according to the temporarily determined foreground / distant view mode.

  When the process proceeds to S26, the image is selected as the image storage target at the viewpoint position of the shaded portion in FIG. 5 among the 5 × 5 viewpoints, and the images selected in S29 are each compressed and encoded and multiplexed into one file. Let me save. When the process proceeds to S28, the image is selected as an image storage target at the viewpoint position of the shaded portion in FIG. 6 from the 5 × 5 viewpoints, and the images selected in S29 are compressed and encoded. It will be multiplexed and stored in the file.

[Third Embodiment]
In the first and second embodiments described above, regarding the multi-view imaging apparatus in which a plurality of cameras are spatially arranged, the image selection is switched between the shooting mode and the foreground mode. The third embodiment relates to a photographing device such as a light field camera that optically realizes a multi-lens photographing device, and uses information on a lens attached to the photographing device to perform photographing corresponding to a distant view mode. An example will be described in which shooting is determined as to whether it corresponds to the foreground mode.

  FIG. 9 is a diagram illustrating an example of an imaging device of a light field camera. A light field camera is a photographing device in which a plurality of microlenses are mounted between a normal camera lens (hereinafter also referred to as a main lens) and a sensor. In a normal camera, light rays pass through the lens of the camera and gather at one point on the image sensor (sensor). However, in a light field camera, the light information of the subject enters the main lens, passes through the microlens, Light is received by a sensor (image sensor), and digital data is acquired. A fusion image of pixels projected from each microlens to the imaging element in a different light direction forms a viewpoint image group. The configuration of FIG. 2 can also be applied to a case where the configuration is applied to the internal configuration of the photographing apparatus. Processing when applied to this light field camera will be described with reference to the flowchart of FIG. This figure shows the processing when the shooting button 110 is pressed.

  First, in S30, it is determined whether or not the code amount suppression mode is ON. If the code amount suppression mode is OFF, the process proceeds to S31, where all viewpoint images are determined as storage targets, and in S38, each is compression-encoded, multiplexed into one file, and stored.

  On the other hand, if it is determined that the code amount suppression mode is ON, the process proceeds to S32. In S32, camera lens focus information (hereinafter also referred to as a focal length) is acquired. As an example of acquiring the focus information, when the imaging unit control unit 207 controls the imaging unit 202, the focus information of the lens connected to the imaging unit 202 is read out, stored in the internal memory 211, and extracted by the CPU 201. Next, in S33, the focus information is compared with a preset threshold value (for example, 50 mm). If the focal length is equal to or larger than the threshold value, the process proceeds to S35. If the focal distance is smaller than the threshold value, the process proceeds to S37. Proceed with the process. In S35 and S37, it is determined that the parallax is small, and a preset number of images are selected from all the images. However, the number selected in S35 is smaller than the number selected in S37.

[Fourth Embodiment]
In the above embodiment, when the code amount suppression mode is ON in the multi-lens imaging device having M × N imaging units, the number of images smaller than M × N is stored. However, the number of images to be stored may be fixed to N × M, and encoding with different compression rates may be switched appropriately. That is, when the code amount suppression mode is OFF, all images are encoded in the low compression encoding mode that is a low compression rate in order to give priority to image quality (encoding processing with the first compression rate). When the code amount suppression mode is ON, the number of images to be encoded in the low compression encoding mode (first compression rate) is reduced, and conversely, the high compression encoding (second compression rate) ) To increase the number of images encoded.

  In order to simplify the description, an example applied to the second embodiment described above will be described. The compression / decompression unit 210 is assumed to be a JPEG encoding / decoding unit and the compression rate is changed according to the quantization step value to be set. However, if the compression rate is variable, the compression / decompression unit 210 is not limited to JPEG. Absent. The flowchart of FIG. 11 should be understood as the processing content of S27 of FIG.

  First, one of 5 × 5 images captured in S41 is acquired. The acquisition order is the raster scan order. Then, in S42, it is determined whether or not the acquired image is the selected image by determining whether or not the viewpoint position of the acquired image is a position corresponding to the shaded portion in FIG.

  If it is determined that the image is not a selected image, that is, it is a non-selected image, the process proceeds to S44, a high compression parameter (for example, quantum step Q1) is set in the compression / decompression unit 210, and compression encoding is performed. Let On the other hand, if it is determined that the image is a selected image, the process proceeds to S43, and since it is an image to be encoded with low compression, the compression / decompression unit 210 is notified of the parameters for low compression (for example, quantum step Q0, Q0 <Q1) is set and compression encoding is performed.

  In step S45, the above process is repeated until it is determined that all the 25 images have been encoded.

  As a result, the image at the viewpoint indicated by the hatched portion in FIG. 5 can be encoded with low compression encoding while maintaining high image quality. Then, although the image quality of the image at the viewpoint position other than the shaded portion may be somewhat deteriorated, the code amount is reduced, and the code amount as a whole can be reduced as compared with the case where the code amount suppression mode is OFF.

  In S28 of FIG. 8, the difference is that the encoding is performed with reference to FIG. 6, but the processing content is the same as FIG. That is, image coding is performed at a low compression rate in the hatched portion in FIG. 6, and high compression coding is performed in other cases.

  In the second embodiment, the compression rate is set using the quantization parameter. However, the present invention is not limited to the above as long as the compression rate can be set. For example, the selected image may be encoded as an I picture in MVC (Multiview Video Coding), and the non-selected image may be encoded as a B picture.

[Fifth Embodiment]
In the fourth embodiment, it is determined whether the image is a selected image or a non-selected image, and encoding is performed by selecting one of quantization steps Q0 and Q1 (Q0 <Q1) at the time of encoding according to the determination result. Made. In addition to the fourth embodiment, in the fifth embodiment, an example will be described in which the free space of the storage medium managed by the external memory control unit 212 that is the final storage destination of the captured image is also taken into account. For this reason, in the fifth embodiment, a quantization step Q0 ′ for realizing the third compression ratio and a quantization step Q1 ′ for realizing the fourth compression ratio (Q0 ′ <Q1 ′) are provided. In terms of the relationship with the pair {Q0, Q1} shown above, there is a relationship of Q0 ≦ Q0 ′ and Q1 ≦ Q1 ′. That is, when Q0 ′ is used, the compression rate is higher than when Q0 is used, and the amount of generated encoded data can be reduced. When Q1 ′ is used, the compression rate is higher than when Q1 is used, which means that the amount of generated encoded data can be reduced.

  Hereinafter, the processing procedure of the CPU 201 when the shooting button 110 is pressed will be described with reference to the flowchart of FIG. This process is the process of S27 (S28) in FIG.

  In S51, Q0 is temporarily set as the quantization step for the selected image, and Q1 is temporarily set as the quantization step for the non-selected image. Next, the process proceeds to S52, and the free capacity of a removable storage medium (such as a compact flash (registered trademark) memory) connected to the external memory control unit 212, which is the storage destination of the captured image data, is equal to or greater than a preset threshold value. Judge whether there is. If the free space is equal to or greater than the threshold, the quantization step Q0 for the selected image and the quantization step Q1 for the non-selected image are determined. On the other hand, if the free space is less than the threshold value, the process proceeds to S53, where the quantization step Q0 is updated with Q0 ', and the quantization step Q1 is updated with Q1'.

  Thereafter, in S54, one of the 25 images taken is acquired. Then, in S55, it is determined whether or not the image is a selected image, in other words, whether or not the image is the image at the viewpoint position of the shaded portion in FIG. If it is determined that the image is a selected image, the image is encoded using a quantization step Q0 in S56. On the other hand, if it is determined that the image is a non-selected image (image at the viewpoint position indicated by a blank in FIG. 5), the image is encoded using the quantization step Q1 in S57.

  Then, in S58, it is determined whether or not encoding has been completed for all the images. If not, the processes in and after S57 are repeated. When it is determined that the encoding for all the images has been completed, the encoded data of the images at the respective viewpoint positions are multiplexed into a predetermined format and stored as one file in the storage medium.

  The above is the processing in S27 of FIG. 8, but the processing of S28 of FIG. 8 is different only in the determination of the selected image and the non-selected image according to the pattern of FIG. .

  In the above embodiment, the quantization steps {Q0, Q1} and {Q0 ′, Q1 ′} are switched depending on whether or not the free space of the storage medium is equal to or greater than the threshold. However, the free space is less than the threshold. In this case, Q1 may be changed to Q1 ′, and Q0 may not be changed. Further, when the capacity of the recording medium is small, the density of selected images, that is, the number of images may be adjusted. If the capacity of the recording medium is very small, unselected images may be deleted preferentially.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (13)

An image processing apparatus that has M × N imaging units, encodes images at different viewpoint positions obtained from each of the imaging units, and stores the images in a predetermined recording medium,
A setting means for setting whether to set a code amount suppression mode for suppressing a code amount or a non-code amount suppression mode;
When the code amount suppression mode for suppressing the code amount is set by the setting unit, a determination unit that determines whether the shooting mode at the time of shooting is a foreground shooting mode or a distant shooting mode;
When the non-code amount suppression mode is set by the setting unit, all images from the M × N imaging units are determined as storage targets,
When the determination unit determines that the camera is in the distant view shooting mode, among the images from the M × N imaging units, the image from the central imaging unit and all the images from the M × N imaging units Of these , images from I (I <M × N) imaging units including positions that can be the maximum parallax are determined as storage targets,
If it is determined that the near view imaging mode by the determining means, among the the M × N of the imaging unit of the image, the image from the imaging unit in the center, I-number of the storage target in the far Jing shooting mode And a determination unit that determines, as a storage target, an image of the image pickup unit and an image from J image pickup units located closer to the image pickup unit located in the central portion than the I image pickup units to be stored in the far-field shooting mode. When,
Encoding means for encoding an image in the imaging unit determined by the determining means as a storage target image;
An image processing apparatus comprising: storage means for storing encoded data obtained by encoding in a predetermined memory. In the far-field shooting mode, the I imaging units that can have the maximum parallax are a plurality of imaging units that are spaced apart by a maximum distance from the imaging unit located at the center. The image processing apparatus according to claim 1. The image from the imaging units located at the four corners of the M × N imaging units is determined as an image from the I imaging units that can be the maximum parallax in the distant view shooting mode. The image processing apparatus according to 1. The case of near-Jing shooting mode includes an image from the imaging unit is located in the central portion, an image processing apparatus according to claim 3, characterized by selecting the imaging unit located in a checkerboard pattern.   The image processing apparatus according to claim 1, wherein the determination unit determines from a shooting mode designated by a user.   The determination unit generates a depth image up to the subject from images of preset viewpoint positions in the M × N imaging units, calculates flatness of pixel values of the depth image, and calculates the calculated flatness The degree is compared with a preset threshold value, and when the flatness is less than or equal to the threshold value, it is determined as the distant view shooting mode, and when the flatness exceeds the threshold value, it is determined as the near view shooting mode. Item 8. The image processing apparatus according to Item 1.   The determination unit determines that the far-field shooting mode is selected when the focal length of the imaging unit is greater than or equal to a preset threshold value, and the near-field shooting mode when the focal length of the imaging unit is less than the preset threshold value. The image processing apparatus according to claim 1. An image processing apparatus that has M × N imaging units, encodes images at different viewpoint positions obtained from each of the imaging units, and stores the images in a predetermined recording medium,
A setting means for setting whether to set a code amount suppression mode for suppressing a code amount or a non-code amount suppression mode;
When the code amount suppression mode for suppressing the code amount is set by the setting unit, a determination unit that determines whether the shooting mode at the time of shooting is a foreground shooting mode or a distant shooting mode;
When the non-code amount suppression mode is set by the setting unit, all the images from the M × N imaging units are determined as encoding targets by the first compression rate,
When the determination unit determines that the camera is in the distant view shooting mode, among the images from the M × N imaging units , the image from the central imaging unit and all the images from the M × N imaging units Of these , images from the I (I <M × N) imaging units including the position where the maximum parallax can be obtained are the first compression rate, and other images are the second compression rate higher than the first compression rate. Determined to be encoded by
When the determination unit determines that the camera is in the foreground shooting mode, among the images from the M × N imaging units , the image from the imaging unit located at the center and the I images to be stored in the far-field shooting mode The first compression rate, and the image from the J imaging units closer to the imaging unit located in the central part than the I imaging units to be stored in the distant shooting mode , Determining means for determining other images as encoding targets by the second compression rate;
Encoding means for encoding each image from the M × N imaging units according to the compression rate determined by the determination means ;
An image processing apparatus comprising: storage means for storing encoded data obtained by encoding in a predetermined memory. Furthermore, it has a detection means for detecting the free space of the predetermined memory,
When the free space detected by the detection unit is less than a preset threshold, the determination unit replaces the first compression rate with a third compression rate higher than the first compression rate, instead of the second compression ratio, the fourth compression ratio higher than the second compression rate (however, the third compression ratio <fourth compression rate) according to claim 8, characterized by using Image processing apparatus. The program for functioning the said computer as each means of any one of Claim 1 thru | or 9 by reading and making it run with the computer which has a MxN image pick-up part. A computer-readable storage medium storing the program according to claim 10 . A control method for an image processing apparatus that includes M × N imaging units, encodes images at different viewpoint positions obtained from each of the imaging units, and stores the encoded images in a predetermined recording medium,
A setting step for setting whether the setting unit sets a code amount suppression mode for suppressing a code amount or a non-code amount suppression mode;
When the determination unit sets a code amount suppression mode that suppresses the code amount by the setting step, a determination step of determining whether the shooting mode at the time of shooting is a foreground shooting mode or a distant shooting mode;
The decision means
When the non-code amount suppression mode is set by the setting step, all the images from the M × N imaging units are determined as storage targets,
If it is determined in the determination step that the camera is in the distant view shooting mode, among the images from the M × N image capturing units, the image from the central image capturing unit and all the images from the M × N image capturing units Of these , images from I (I <M × N) imaging units including positions that can be the maximum parallax are determined as storage targets,
The determination step when it is determined that the near view imaging mode, the out of the M × N of the imaging unit images, the image from the imaging unit in the center, of the I-number of the storage target in the distant view image capturing mode A determination step of determining an image of the imaging unit and an image from J imaging units located closer to the imaging unit located in the central portion than the I imaging units to be stored in the distant shooting mode as a storage target ; ,
An encoding step in which an encoding unit encodes an image in the imaging unit determined in the determination step as a storage target image;
And a storage unit that stores the encoded data obtained by encoding in a predetermined memory. A control method for an image processing apparatus that includes M × N imaging units, encodes images at different viewpoint positions obtained from each of the imaging units, and stores the encoded images in a predetermined recording medium,
A setting step for setting whether the setting unit sets a code amount suppression mode for suppressing a code amount or a non-code amount suppression mode;
When the determination unit sets a code amount suppression mode that suppresses the code amount by the setting step, a determination step of determining whether the shooting mode at the time of shooting is a foreground shooting mode or a distant shooting mode;
The decision means
When the non-code amount suppression mode is set by the setting step, all the images from the M × N imaging units are determined as encoding targets by the first compression rate,
If it is determined in the determination step that the camera is in the distant view shooting mode, among the images from the M × N image capturing units , the image from the central image capturing unit and all the images from the M × N image capturing units Of these , images from the I (I <M × N) imaging units including the position where the maximum parallax can be obtained are the first compression rate, and other images are the second compression rate higher than the first compression rate. Determined to be encoded by
If it is determined in the determination step that the camera is in the foreground shooting mode, among the images from the M × N imaging units , the image from the imaging unit located at the center and the I images to be stored in the distant shooting mode The first compression rate, and the image from the J imaging units closer to the imaging unit located in the central part than the I imaging units to be stored in the distant shooting mode , A determination step of determining other images as encoding targets by the second compression rate;
An encoding step for encoding each of the images from the M × N imaging units according to the compression rate determined in the determination step ;
An image processing apparatus comprising: a storage unit that stores the encoded data obtained by encoding in a predetermined memory.

Images