JP6207202B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for processing image data based on information acquired from a plurality of viewpoint positions.

  Currently, as a method for confirming a captured image after image capturing in an image capturing apparatus, a method for displaying and confirming the captured image in a so-called reproduction mode is known.

  In recent years, an imaging apparatus using a technique called “Light Field Photography” has been proposed (Non-Patent Document 1, Patent Document 1). This imaging device is composed of an imaging lens, a microlens array, an imaging device, and an image processing unit, and the captured image data obtained from the imaging device is used in addition to the light intensity distribution on the light receiving surface. It also includes information on the direction of travel. In the image processing unit, observation images from a plurality of viewpoints and directions can be reconstructed.

  One of the reconstruction is focus adjustment processing after imaging (hereinafter referred to as refocusing) (for example, Patent Document 2), and an imaging device capable of refocusing after imaging (hereinafter referred to as light field camera) has been developed. Has been.

  In the refocus calculation processing to reconstruct observation images from multiple viewpoints and directions in the image processing unit, alignment processing for images from multiple viewpoints and directions is essential, and this alignment calculation amount (processing load) ) Is large.

  In addition, the refocus calculation process requires calculation for focusing on each area and calculation for blurring. In particular, for blurring, it is not necessary to uniformly blur areas other than the focus area, but visually natural blurring processing (for example, generation of captured image data with a shallow depth of focus) is essential. (Processing load) is large.

JP 2010-183316 A JP 2011-22796 A

R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, P. Hanrahan: "Light Field Photography with a Hand-Held Plenoptic Camera", Stanford Tech Report CTSR 2005-02 (2005)

  In the current image processing apparatus, when a refocus image is generated when an image is displayed, the refocus image is generated according to the user's operation, and it takes time to generate the refocus image, so that there is a waiting time.

An image processing apparatus according to the present invention includes: a determination unit that determines a rank of a target to be focused based on history information indicating an operation history; and a composite image that focuses on the target according to the rank determined by the determination unit Generation means for sequentially generating data from multi-viewpoint image data obtained by imaging data from a plurality of viewpoint positions ; and control means for displaying the generated composite image data, wherein the generation means is the display The composite image data is sequentially generated according to the order while no parameter is input from the user to the composite image data .

  According to the present invention, it is possible to predict a region where the user desires refocus display before the user performs an operation for designating a region to be refocused, and to start or complete the refocus calculation. Accordingly, it is possible to reduce the waiting time from when the user designates the refocus area until the refocus image is displayed.

It is a block diagram which shows an example of the hardware constitutions of the whole multiview camera which concerns on Example 1 of this invention. It is a flowchart which shows an example of the refocus process at the time of the reproduction | regeneration which concerns on Example 1 of this invention. It is a flowchart which shows an example of the parameter selection process which concerns on Example 1 of this invention. It is a flowchart which shows an example of the prediction method of the refocus calculation area | region which concerns on Example 1 of this invention. It is a figure which shows an example of the refocus log | history regarding the past person which concerns on Example 1 of this invention. It is a figure which shows an example of the display of the refocus image which concerns on Example 1 of this invention. It is a figure which shows an example of the temporary image format which preserve | saves the refocus image data based on 1st Example of this invention. It is a figure which shows an example of the image data format which preserve | saves the multi-viewpoint image which concerns on Example 1 of this invention. It is a flowchart which shows an example of the refocus process at the time of reproduction | regeneration which concerns on Example 2 of this invention. It is a flowchart which shows an example of the refocus process at the time of reproduction | regeneration which concerns on Example 2 of this invention. It is a figure which shows an example of the external appearance of the multiview camera imaging part which concerns on Example 3 of this invention. It is a figure which shows an example of the image data format which preserve | saves the multi-viewpoint image which concerns on Example 3 of this invention. It is a figure which shows an example of a function structure of the refocus calculating part which concerns on Example 3 of this invention, and an example of the flow of a process. It is a flowchart which shows an example of the flow of a process at the time of instruct | indicating the non-focus target based on Example 3 of this invention. It is a figure which shows an example of the relationship between the imaging device at the time of imaging the object which performs the image processing which concerns on Example 3 of this invention, the position of an imaging subject, a depth of focus, etc., and the display on a display. It is a figure which shows an example of the relationship between the imaging device at the time of imaging the object which performs the image processing which concerns on Example 3 of this invention, the position of an imaging subject, a depth of focus, etc., and the display on a display. It is a figure which shows an example of the relationship between the imaging device at the time of imaging the object which performs the image processing which concerns on Example 3 of this invention, the position of an imaging subject, a depth of focus, etc., and the display on a display. It is a figure which shows an example of the data format of the focusing / non-focusing object list which concerns on Example 4 of this invention. It is a figure which shows an example of the image data format which preserve | saves the multiview image which concerns on Example 4 of this invention. It is a figure which shows an example of the management structure of the focusing / non-focusing object list which concerns on Example 4 of this invention. It is a flowchart which shows an example of the addition / deletion process of the target object in the focusing / non-focusing target list which concerns on Example 4 of this invention. It is a flowchart which shows an example of the process which reflects the object addition in the focusing / non-focusing object list | wrist which concerns on Example 4 of this invention on the refocus position information of a multiview image. It is a flowchart which shows an example of the process which reflects the target object deletion in the focusing / non-focusing object list | wrist which concerns on Example 4 of this invention on the refocus position information of a multiview image. It is a flowchart which shows an example of the process which adds refocus position information to the said multiview image at the time of the multiview image imaging which concerns on Example 4 of this invention. It is a flowchart which shows an example of the process which reflects the information of the target object in the focusing / non-focusing object list which concerns on Example 4 of this invention on the refocus position information of a preservation | save multi-viewpoint image. It is a figure which shows an example of the data format of the focusing / non-focusing object list which concerns on Example 5 of this invention. An example of a user interface (UI) for updating recognition information of an object in the in-focus / non-focus object list according to Embodiment 5 of the present invention and deleting refocus position information generated based on the object FIG. It is a figure which shows an example of the system configuration | structure which performs focusing / non-focusing object list management, refocus position information generation, and refocus image generation according to Embodiment 6 of the present invention. It is a figure which shows an example of the relationship between the imaging device at the time of imaging the object which performs the image processing which concerns on Example 7 of this invention, the position of a to-be-photographed object, a focal depth, etc., and the display on a display. It is a figure which shows an example of the relationship between the imaging device at the time of imaging the object which performs the image processing which concerns on Example 7 of this invention, the position of a to-be-photographed object, a focal depth, etc., and the display on a display. It is a figure which shows an example of the relationship between the imaging device at the time of imaging the object which performs the image processing which concerns on Example 7 of this invention, the position of a to-be-photographed object, a focal depth, etc., and the display on a display. It is a figure which shows an example of the relationship between the imaging device at the time of imaging the object which performs image processing which concerns on Example 7 of this invention, the position of a to-be-photographed object, and a virtual focus surface, and the display on a display. It is a figure which shows an example of the relationship between the imaging device at the time of imaging the object which performs the image processing which concerns on Example 7 of this invention, the position of a to-be-photographed object, and a virtual focus surface, and the display on a display. It is a figure which shows an example of the relationship between the imaging device at the time of imaging the object which performs the image processing which concerns on Example 8 of this invention, the position of an imaging subject, and a virtual focus surface.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating an example of a hardware configuration of the image processing apparatus according to the first embodiment of the present invention. Hereinafter, FIG. 1 will be described in detail. The multi-lens camera imaging unit 101 is an aggregate of a plurality of mutually independent cameras having an optical system and an image sensor that are independent from each other. The multi-view camera imaging unit 101 includes an image sensor control unit and the like, and outputs a set of output data from a plurality of image sensors as multi-viewpoint image data.

  The RAM 102 is a memory used for temporary storage of multi-viewpoint image data captured by the multi-eye camera imaging unit 101, temporary storage of generated refocus image data, and other temporary data during calculation. The flash ROM 107 is a non-volatile memory, and functions as a history information storage unit that accumulates and stores operation histories related to images (type and position) and operations selected by the user in the past. The external memory 109 is an external memory such as an SD card. The external memory 109 is a non-volatile memory, and image data is saved even when the power is turned off. In the external memory 109, multi-viewpoint image data captured by the multi-view camera imaging unit 101 is stored as an image file in accordance with, for example, an image data format shown in FIG. Details of the image data format will be described later. The external memory 109 is also used as an area for temporarily storing image data being processed as a temporary image file.

  The memory control unit 110 includes a so-called bus system and a control unit of a memory and a device connected to the bus system, and controls reading and writing of data with respect to each memory of the RAM 102, the Flash ROM 107, and the external memory 109, for example.

  A user I / F (interface) 106 is an I / F (interface) for operating a device, selecting an image or region that a user desires to display refocused, or selecting a mode related to displaying a captured image. ). Specifically, it corresponds to the user I / F such as a touch panel installed on the display 105, a shutter button, and an operation dial.

  The overall control unit (CPU) 108 performs overall control of the device and calculation control such as selecting the most frequent area from the history information stored in the flash ROM 107 and instructing the refocus calculation unit 103.

  The refocus calculation unit 103 generates refocus image data from the multi-viewpoint image data in accordance with an instruction from the overall control unit 108. Here, an outline of generating refocus image data from multi-viewpoint image data will be described. In light field photography, the direction and intensity of light passing through each position (light field, hereinafter referred to as “LF”) is calculated from multiple viewpoint image data for a plurality of positions in space. Then, using the obtained LF information, an image when passing through the virtual optical system and forming an image on the virtual sensor is calculated. By appropriately setting such virtual optical systems and sensors, it is possible to generate refocus image data that is composite image data focused on a predetermined target. Note that the refocus processing is not the main focus of the present embodiment, and a method other than the method described above may be used. The generated refocus image data is stored in the RAM 102, the external memory 109, or the like. Details will be described later.

  The graphic processor 104 has a function of displaying the refocus image data generated by the refocus calculation unit 103 on the display 105. The network I / F 1110 performs network connection with the outside of the device, and performs data transfer with the external device on the network.

  The image analysis unit 112 detects the area of the subject in the image data and assigns an identification code to each area. The image analysis unit 112 stores the detected subject area information and identification code in the RAM 102. The subject in this embodiment is assumed to be an object including not only a person or animal in the foreground but also a landscape portion included in the background. The detection of the subject can be performed, for example, by utilizing some prominent features (two eyes, mouth, nose, etc.) and the inherent geometric positional relationship between the features. Alternatively, the detection of the subject can be performed by using a symmetrical feature of the face, a feature of the face color, template matching, a neural network, or the like. For the detection of the background, for example, the face of a person or animal in the image is detected by the method described above, and the position and size are calculated. Based on this calculation result, the image can be divided into a subject region including a person in the image and a background region other than that. For example, assuming that the subject area is a rectangle, the format of the area information is indicated by upper left and lower right coordinates. In addition, the subject identification target is three types, that is, a person (or face), an animal, and a landscape. However, the subject identification target is not limited to this, and finer classification is also possible. Note that the subject detection and identification method is not the main focus of the present embodiment, and methods other than those described above may be used. The object detection and recognition function in the image analysis unit 112 is performed immediately after the image is captured, and the result is stored in a non-volatile memory such as the flash ROM 107 or the external memory 109, for example. Therefore, in the description of the present embodiment, it is assumed that the area information of the subject and the identification code are already stored.

  The multi-viewpoint image data captured by the multiview camera imaging unit 101 is recorded in the external memory 109 as an image file according to the image data format of FIG.

  FIG. 8 is a diagram illustrating an image data format used when multi-viewpoint image data is stored as an image file in the present embodiment. The image file shown in FIG. 8 includes information 801 indicating image width, information 802 indicating image height, refocus position information 803, multi-view image header information 810, and multi-view image data 804.

  The refocus position information 803 includes a position information presence flag 805. The position information presence flag 805 is information indicating whether or not refocus position information exists. This position information presence flag is, for example, 0 (false) in the initial state (there is no history). In the case of 1 (true), the values of the subsequent position information 806 to 809 can be validated. That is, when the position information presence flag is true, information indicating the refocused position is included. Top_left_x 806 is information indicating the upper left X coordinate of the refocus area. Top_left_y 807 is information indicating the upper left Y coordinate of the refocus area. Bottom_right_x 808 is information indicating the lower right X coordinate of the refocus area. Bottom_right_y 809 is information indicating the lower right Y coordinate of the refocus area.

  The position information presence flag 805 is sufficient as 1-bit information indicating whether the image file is added or not, but is not limited to 1 bit. A multivalue indicating the format of the refocus position information format may be taken. For example, it is possible to adopt the following configuration. When the position information presence flag 805 is 0, there is no position information. When the position information presence flag 805 is 1, the position information is expressed as rectangular information as described above. When the position information presence flag 805 is 2, the position information is expressed by using the center coordinates and radius of a circle indicating the focus target as position information.

  The multi-viewpoint image header information 810 includes information necessary for performing the refocus processing and auxiliary imaging information. For example, the camera model name, lens configuration, imaging date, shutter speed, and exposure program. Further, the multi-view image header information 810 includes position information of each image data in the multi-view image data used for the refocus processing. That is, information indicating the position of each imaging unit is included. In the present embodiment, a configuration example in which the position information 806 to 809 exists regardless of whether the position information presence flag is true or false is shown. However, the present invention is not limited to this example. When the position information presence flag is false, the position information 806 to 809 entries themselves may not be included in the image data. Further, the refocus position information can be configured to indicate an arbitrary shape, and is not limited to the rectangular area described in the present embodiment.

  Note that the multi-viewpoint image data 804 may be RAW image data obtained from an image sensor, or after development subjected to development processing such as demosaicing processing, white balance processing, gamma processing, and noise reduction processing. It may be image data. Further, in this embodiment, the example using the multi-view camera imaging unit 101 has been described. However, the multi-viewpoint image data may be data from a plurality of viewpoint positions, for example, using image data obtained using a microlens array. May be.

  FIG. 2 is a flowchart from the transition to the playback mode to the display of the first refocus image. The reproduction mode is an operation mode in which control for displaying an image file stored in the external memory 109 on the display 105 is performed. The flow in FIG. 2 is executed by the overall control unit (CPU) 108 controlling the refocus calculation unit 103, the graphic processor 104, and the like.

  A program for the overall control unit 108 to execute the control shown in FIG. 2 is stored in, for example, the Flash ROM 107 or the like. Further, the refocusing program described above is also stored in the same manner. The image file in the flow shown in FIG. 2 is based on the premise that the multi-viewpoint image data captured by the multi-view camera imaging unit 101 is already recorded in the external memory 109 together with the header information as shown in FIG. Yes.

In step S201, the overall control unit 108 shifts the operation mode to the reproduction mode. The following three patterns can be considered as the case of moving to the processing in step S201.
1. 1. It is being activated in a mode other than the playback mode and then switched to the playback mode 2. Playback mode already at startup When another image is specified in playback mode

  In step S <b> 202, the overall control unit 108 acquires an image file from the external memory 109. In step S203, the overall control unit 108 acquires the playback refocus mode and history information, and the process proceeds to S204. The reproduction refocus mode is a mode in which, for example, a human mode, a landscape mode, an animal mode, or the like is selected when displaying image data that has been subjected to refocus calculation (hereinafter referred to as refocus image data). The history information is information obtained by accumulating the types and positions of image data selected by the user in the past and the history regarding operations. Details of the history information will be described later. In step S204, the header information of the image file read by the overall control unit 108 is analyzed. The header information corresponds to 801 to 803 in FIG. In this embodiment, the refocus position information 803 is particularly analyzed.

  In step S205, the overall control unit 108 determines whether there is a recommended parameter in the header information analyzed in step S204. The recommended parameter is a parameter indicating an image recommended for refocusing an image file, a display method, or the like. If the recommended parameter exists, the process proceeds to step S206. If the recommended parameter does not exist, the process proceeds to step S207. In the present embodiment, the position information presence flag 805 shown in FIG. 8 is treated as a 1-bit flag, and when the flag value is 1, the overall control unit 108 determines that a recommended parameter exists and proceeds to step S206. Transition. If the value of the flag is 0, the overall control unit 108 determines that there is no recommended parameter, and proceeds to step S207.

  In step S206, the overall control unit 108 controls the refocus calculation unit 103 using the recommended parameters obtained by analyzing the refocus position information 803 of the image file read in step S202, and acquires the refocus image data. Generate.

  Specifically, the refocus calculation unit 103 uses, from the multi-viewpoint image data 804 included in the reproduction file, a position where the average of the position information 806 and 808 is the X coordinate and the average of the position information 807 and 809 is the Y coordinate. Generate image data in focus. The refocus image data generated by the refocus calculation unit 103 is displayed on the display 105 via the graphic processor 104. The generated refocus image data is stored in the external memory 109 or the RAM 102 as a temporary image file in accordance with the temporary image file format shown in FIG. FIG. 7 is a diagram illustrating an example of a temporary image file format of an image file including refocus image data generated by the refocus calculation unit 103. The image file according to the temporary image file format includes refocus position information 702 and refocus image data 705. The refocus position information 702 includes information indicating a position where the focus is achieved when the refocus processing of the refocus image data 705 is performed. The position information 706 to 709 can be the same information as the position information 806 to 809 in FIG. The temporary image file shown in FIG. 7 is generated for each refocus image data, for example.

  If the process proceeds to step S206, the recommended parameter exists. Therefore, the temporary image file in which the information included in the recommended parameter is included in the refocus position information indicated by reference numeral 702 in FIG. 7 is stored. Note that the storage location of the temporary image file is not limited to the external memory 109 and the RAM 102, and may be stored in an arbitrary storage device on the cloud via the network I / F 111.

  Next, a process when there is no recommended parameter in the header information in step S205 will be described. The process in step S207 is a process when there is no recommended parameter in the image file read from the memory in step S202, that is, when the display is the first image file in the playback mode. In this embodiment, refocus image data is generated based on the history information and stored as a temporary image file. Also, history information is updated. The process of generating refocus image data based on the history information and the process of updating the history information will be described with reference to FIG.

  In step S208, the graphic processor 104 displays the refocused image generated in step S207 on the display 105. FIG. 6 is a diagram illustrating an example when the refocus image data generated in step S206 or step S207 is displayed. FIG. 6A shows an example of an image focused on a landscape (tree).

  In step S209, the graphic processor 104 additionally displays a parameter selection UI screen on the display 105. An example in which a parameter selection UI screen is added is shown in FIG. In the example of FIG. 6B, it is assumed that the user designates a point to be focused on the touch panel. That is, the graphic processor 104 displays a UI screen for allowing the user to input a position (parameter) to be focused. The parameter may represent two-dimensional position information in the image, or may represent a spatial distance (depth) in the multi-viewpoint image.

  In step S210, the overall control unit 108 updates the refocus position information 803 of the image file read in step S202. That is, when the recommended parameter does not exist in the header information, the refocus position information 803 is updated with the position information of the area for which the refocus position is determined in step S207. If the recommended parameter exists in the header information, the refocus position information 803 of the image file read in step S202 does not need to be updated. The above is the operation when the playback mode is activated, that is, the processing related to the refocus image displayed first when a certain image file is read.

  Next, the operation after displaying the first refocus image data (display of the second and subsequent refocus image data, the end of the operation, etc.) will be described with reference to FIG. That is, the processing after displaying the refocus image that is initially displayed when the image file described in the flow of FIG. 2 is read will be described with reference to FIG. The entire flow shown in FIG. 3 is controlled by the overall control unit 108 as in the flow of FIG.

  When the parameter selection UI shown in FIG. 6B is displayed in step S209 of FIG. 2, the user can select a parameter using the displayed UI. In step S301, the overall control unit 108 proceeds to step S304 when the user selects a parameter using the UI. In this embodiment, the user selects a parameter by designating one point on the screen. In step S304, the parameter selected by the user in step S301 is acquired. In step S304, the overall control unit 108 determines whether or not there is refocus image data including the parameter designated (selected) by the user in step S301 from the temporary image file group stored in the format of FIG. Determine whether. That is, it is determined whether or not there is a temporary image file including refocus position information 702 including parameters specified by the user. If there is a temporary image file including refocus image data that matches the parameter specified by the user in step S304, the overall control unit 108 shifts the processing to step S309. In step S309, information (position, display history information) of the temporary image including the refocus image is updated, and the process proceeds to step S306. Details will be described later.

  In step S306, the graphic processor 104 displays a temporary image including refocus image data matching the parameter specified by the user on the predetermined display 105.

  On the other hand, if there is no temporary image file including refocus image data matching the parameter specified by the user in step S304, the overall control unit 108 moves the process to step S305. In step S305, based on the parameter specified by the user in step S301, the refocus calculation unit 103 generates refocus image data focused on the area indicated by the parameter. In step S305, the generated refocus image data is stored as a temporary image file shown in FIG.

  When the process of step S305 is completed, the overall control unit 108 moves the process to step S309.

  In step S309, in step S305, the user updates information (position and display history information) related to the refocus image generated according to the parameter specified in step S301, and the process proceeds to step S306. Details will be described later.

  In step S306, for example, the refocus image data generated or selected according to the parameter specified in step S301 is read from the RAM 102 or the like and displayed on the display 105.

  Here, the information update in step S309 will be described in detail. As described above, the image analysis unit 112 detects the area of the subject in the captured multi-viewpoint image data, is assigned an identification code for each area, and the area information and the identification code are stored in the Flash ROM 107 or the external memory. 109 is stored in a non-volatile memory. Here, the stored area information is compared with the specified parameter, and if the specified parameter is included in the area information, the area information is updated as position information. That is, the area information classified as the subject and the position of the subject is specified is updated as the refocus position information 702 of the temporary image file. On the other hand, the stored area information is compared with the designated parameter, and if the designated parameter is not included in the area information, the designated parameter is updated as the refocus position information 702 of the temporary image file. .

Next, history information will be described with reference to FIG. In FIG. 5, the identification code is an identification code identified by the image analysis unit 112. The frequency refers to the frequency at which the user instructs the display of the focused image for each area indicated by the identification code (person, landscape, etc.) in the playback mode. This frequency may be a frequency for each image file, or may be a frequency specified by a user in the image processing apparatus regardless of the image file. FIG. 5A shows an example in which 501 is an identification code = scenery, 502 is an identification code = person, and 503 is an identification code = animal. Here, the frequency ranking for each identification code is as follows.
1st place: Landscape (501)
2nd place: People (502)
3rd place: Animals (503)

FIG. 5B shows an example in which the identification code is further subdivided for the identification code = person. The distinction between the daughter, son, and father can be determined by, for example, calculating the feature vector of each face image and comparing it with the feature vector of the region determined to be a person. Here, the frequency ranking for each identification code is as follows.
1st place: User's daughter (505)
2nd place: User's son (506)
3rd place: User's father (507)
4th: Unregistered person (508)
In the present embodiment, the history information indicates frequency information of each identification code.

  Next, update of history information will be described. As described above, when the designated parameter is included in the area information, the frequency information of the identification code associated with the area is incremented. On the other hand, when the designated parameter is not included in the area information, the useful information is only the coordinate information, and the history information is not updated.

  Next, a process when the user does not select a parameter in step S301 will be described. In a state where the already displayed image is continuously displayed, the process proceeds to step S302.

  In step S302, for all the region information extracted by the image analysis unit 112, the overall control unit 108 determines whether refocus image data in which each region information is focused is generated. Specifically, it is determined whether or not a temporary image file having refocus position information 702 including each area information exists for all area information.

  If it is determined in step S302 that no refocus image has been generated for all region information, the process proceeds to S303. On the other hand, if it is determined in step S302 that the generation of the refocus image has been completed for all the area information, the process proceeds to S307.

  If it is determined that no refocus image has been generated for all region information, refocus image data is generated based on the history information in step S303. Details will be described later. Next, after the generated refocus image is displayed in S306, the process proceeds to step S307.

In S307, the overall control unit 108 determines whether to end or continue the playback mode. The playback mode end conditions in S307 are, for example, the following three conditions.
1. 1. When switching to a mode other than the playback mode 2. When the power is turned off. When another image is designated in the playback mode, if it is determined in step S307 that the playback mode has ended, the process proceeds to step S308, and various processes necessary to end the playback mode are executed, and the playback mode ends. On the other hand, if it is determined in step S307 that the playback mode is to be continued, the process returns to step S301, and the operations from step S301 are repeated.

  Note that parameter selection is performed by the user pointing to one point on the screen, but it can also be specified as a closed region. For example, the presence / absence of history information can be determined by comparing the image including the point designated by the user and the image included in the designated closed region with the history of the flash ROM (history storage unit) 107.

  Next, details of the processing in S207 of FIG. 2 and S303 of FIG. 3 will be described using the flowchart of FIG. The process shown in the flowchart of FIG. 4 is executed by the overall control unit (CPU) 108 controlling the refocus calculation unit 103, the graphic processor 104, and the like.

  As described above, the image analysis unit 112 detects the area of the subject in the captured multi-viewpoint image data, and an identification code is assigned to each area, and the area information and the identification code are stored in the RAM 102 or externally. It is stored in a nonvolatile memory such as the memory 109. The history information is frequency information shown in FIG. That is, in the past history (frequency), it is assumed that a landscape image is ranked first in frequency, a person image ranked second, and an animal image ranked third. Further, it is assumed that the person's image has the first frequency as the daughter, the second as the son, the third as the father, and the fourth as an unregistered person.

  In step S401, the overall control unit 108 identifies an area where refocus image data has not been generated. First, the overall control unit 108 acquires all the pairs of identification codes and region information of the image analysis result regarding the target image data (that is, the image data indicated by the multi-viewpoint image data of the image file read in S202). In addition, position information of refocus image data included in all temporary image files is acquired. Next, the refocus position information 702 included in the temporary image file is compared with the area information from the acquired information. Then, the area information is excluded from the area information, and the area information for which no corresponding position information exists is extracted. This area information corresponds to an area where no refocus image data is generated.

  Next, in step S402, the overall control unit 108 selects, from among the identification codes associated with the area information extracted in step S401, the one that matches the identification code with the highest frequency of history information. If the frequency of the identification code corresponding to the landscape is the highest as a result of selection, the process proceeds to step S403. In step S403, it is determined whether the area corresponding to the landscape image is in the area information extracted in S401. If there is a landscape image, the process proceeds to step S404.

  In step S404, area information whose identification code is landscape is extracted, and when there is only one area, that area is determined as an area for generating next refocus image data. Further, when there are a plurality of regions having identification codes corresponding to the landscape, for example, the order can be determined by using the X coordinate and Y coordinate at the upper left of the region information. Specifically, the refocus calculation order can be determined in ascending order of the X coordinate value. If there are a plurality of the same X coordinates, the refocus calculation order can be determined in ascending order of the Y coordinates. Here, the region information with the highest rank is determined as the region for generating the next refocus image.

  Next, the process proceeds to step S405. In step S405, the refocus calculation unit 103 generates refocus image data in which the area information of the rank is focused based on the rank determined in step S404. Specifically, image data in which a focus is set to a position where the average of the upper left and lower right coordinates is taken as the X coordinate and the Y coordinate is generated from the multi-viewpoint image data.

  Next, the process proceeds to step S406. In step S406, along with the refocus image data generated in step S405, a temporary image file having the focused area information as the refocus position information is stored in the temporary image file format shown in FIG.

  Next, the process proceeds to step S407. In step S407, the history information is updated. Specifically, the frequency corresponding to the selected identification code is incremented. Then, the process ends.

  Next, returning to step S402, if the frequency of the identification code corresponding to the person is the highest as a result of selection, the process proceeds to step S409. In step S409, it is determined whether there is a human image. If there is no human image, the process returns to step S402. If there is a human image, the process proceeds to step S410. In step S410, it is determined whether there is a registrant identification code as shown in FIG. If there is a registrant identification code, the process proceeds to step S411. In step S411, it is determined whether or not there is a registrant whose existence has been confirmed in step S410 in the history information. If there is a registrant in the history information, the process proceeds to step S412. In step S412, the order of refocus image generation is determined according to the frequency of history information. In FIG. 5B, since the first rank is the daughter, the second rank is the son, and the third rank is the father, for example, if there is a daughter in the identification code, the daughter is the highest rank.

  If it is determined in step S411 that there is no registrant in the history information, the process proceeds to step S413. In step S410, if there is no registrant identification code, the process proceeds to step S414. Note that steps S413 and S414 perform the same processing as step S404.

  Next, returning to step S402, if the frequency of the identification code corresponding to the animal is the highest as a result of selection, the process proceeds to step S415. If it is determined in step S415 that there is an animal image, the process proceeds to step S416. In addition, about the process of step S416, the process similar to S404 can be performed.

  With the configuration as described above, it is possible to perform generation of refocus image data before instructing the user by using the past history. That is, when an image file including multi-viewpoint image data is first read and the recommended parameter is not included in the header information of the file, refocus image data is generated without user instruction using history information. be able to. Further, by using the history information, it is possible to generate refocus image data according to the frequency of the subject that the user has focused on in the past. By generating refocus image data in advance without user instruction, when the user specifies the desired focus area, the user can display the desired refocus image without waiting for the refocus calculation time. There is a possible effect.

  In addition, it is now possible to predict the next refocus image while displaying the refocus image and perform the refocus calculation, so that the use efficiency of calculation resources is improved and there is no hardware upgrade. The same effect as an improvement in processing speed can be expected.

  Furthermore, each time the refocus calculation is repeated, the user's preference can be determined more clearly, so that the prediction accuracy of the refocus area is improved every time it is used, and the switching speed of the refocus image can be expected to be increased.

  In this embodiment, parameter information for specifying a focus position last specified by the user is embedded in an image file including multi-viewpoint image data. As a result, when the image file is read and displayed next time, the image that was last displayed by the user can be displayed without resetting the parameters. As a result, it is possible to save the user from setting parameters and to improve the operability of the apparatus incorporating the present technology.

  In this embodiment, for the sake of simplicity, the example of FIG. 4 has been described with three types of options (people, scenery, and animals) as an example, but identifiable items may be added to the options. . Further, in the present embodiment, the image of a person has been described as three registrants and unregistered persons such as a daughter, son, father, and unregistered person, but the number of registrants may be four or more. Absent. Although the registered person is also a family, the present invention is not limited to this.

  Further, in this embodiment, an example in which refocus image data is sequentially generated based on history information when a recommended parameter does not exist in the header information of the image file has been described. However, even when the recommended parameter exists in the header information of the image file, the refocus image data can be generated based on the history information as in the process of step S209 in FIG. 2 (that is, the process shown in FIG. 3). Please note that.

  In the second embodiment, an example in which the current reproduction refocus mode and the history information of the reproduction refocus mode are added to the prediction factor will be described.

  In the description of step S207 in FIG. 2 (Example 1), the process when the recommended parameter does not exist in the image file stored in the memory, that is, when the display in the reproduction mode is the first time has been described. In the description of step S207 in the first embodiment, it has been described that the refocus area can be predicted based on the past history information related to the refocus image reproduction and the refocus calculation can be performed.

  The difference from the first embodiment is that the replay area prediction factor (ie, past history information) shown in step S207 of FIG. 2 and step S303 of FIG. The mode history information is added to the prediction factor. Playback refocus mode is a function that allows you to switch the refocus target according to the target you want to focus on.For example, you can select the human, landscape, or animal mode, and refocus calculation that suits each subject. This is a possible setting.

  The history information described in the first embodiment is general history information. The history information in the current playback refocus mode in this embodiment is information on the history in the playback refocus mode. Here, the history information described in the first embodiment may be referred to as first history information, and the history information in the current reproduction refocus mode may be referred to as second history information.

  In the first embodiment, an example in which only history information about past refocus calculation or display is used has been described. In the present embodiment, the information on the reproduction refocus mode is added to the history information as a prediction factor, so that the prediction accuracy of the refocus area and the calculation order is improved.

  First, the process up to the display of the first refocus image in this embodiment when there is no recommended parameter in the refocus image file stored in the memory will be described. In the example described with reference to FIG. 9, the history information (second history information) in the above-described reproduction refocus mode is not used when displaying the first refocus image. The second history information may also be used as in the process of 10.

  Steps S901 to S906 in FIG. 9 are the same as the processing in FIG. In step S910 in FIG. 9, the overall control unit 108 refers to the first history information, reads the frequency information of the past refocus calculation process, temporarily determines the refocus calculation order in descending order of frequency, The process proceeds to step S911. The provisionally determined process can be equivalent to the process shown in the flow shown in FIG.

  In step S911, the overall control unit 108 determines whether the setting in the camera or the like is a priority setting that prioritizes the current playback refocus mode or a setting that prioritizes past history information (for example, FIG. 5). To do. If it is determined in step S910 that the current reproduction refocus mode has priority, the process proceeds to step S913. If it is determined that the past history is given priority, the process proceeds to step S912.

  Generally, the user selects the current playback refocus mode setting or the past history information setting by a menu or the like displayed on a camera body or a viewer such as a PC. Also, which of the priority values is set as the initial value may be set in advance.

  If it is determined in step S911 that the playback refocus mode has priority, in step S913, the overall control unit 108 performs the following process. That is, it is determined whether or not there is a refocus target area including an image that conforms to the current reproduction refocus mode with respect to all the refocus target areas of the subject detected by the image analysis unit 112. If it is determined that an image suitable for the playback refocus mode is included, the process proceeds to step S914. On the other hand, if it is determined in step S913 that the refocus target area does not include an image suitable for the current reproduction refocus mode, the process proceeds to step S912. The subsequent processing is the same as the case where it is determined in step S911 that the current playback refocus mode is not prioritized.

  In step S914, the overall control unit 108 sets the refocus calculation order (temporarily determined in step S910) for the area determined in step S913 to include an image that matches the current playback refocus mode setting. I'll give you. It should be noted that raising one rank is merely an example, and there is no restriction on the fluctuation range of the rank. By the processing in step S913, the refocus calculation order provisionally determined in step S910 is changed. Then, the refocus calculation order of all the refocus target areas is determined, and the refocus calculation of each refocus calculation target area is performed according to the determined refocus calculation order to generate refocus image data.

  When the refocus calculation is completed in step S914, the generated refocus image data is stored in a memory such as the external memory 109 as a temporary storage file. In step S908, the refocus image data stored in the memory is read out and displayed on a predetermined display, and the process proceeds to step S909. The processing in steps S909 and S920 is the same as that in steps S209 and S210 in the first embodiment shown in FIG.

  Next, the second and subsequent refocus processing in the present embodiment will be described mainly with reference to FIG. That is, processing subsequent to step S909 in FIG. 9 will be described with reference to FIG. In FIG. 10, the second history information may be used.

  The case shown in FIG. 10 assumes a case where a refocus calculation for the second and subsequent sheets is performed. That is, it is assumed that there is a display history of refocus image data generated by performing at least one refocus calculation in the past, and a history of refocus position information regarding refocus calculation or display remains. When the recommended parameter is included in the header information of the image file, the recommended parameter can be handled as a history of the refocus position information. This operation is performed when the overall control unit 108 in FIG. 1 reads and executes a program stored in the RAM 102, the Flash ROM 107, the external memory 109, or the like.

  In step S1010, the overall control unit 108 determines whether or not the user has selected a parameter. If it is determined that the parameter has been selected, the process proceeds to step S1050.

  In step S1050, the overall control unit 108 searches for refocus image data matching the designated parameter from the temporary image file stored in a predetermined memory. That is, it tries to detect refocus image data associated with the refocus position information including the designated parameter. If suitable refocus image data is detected (that is, if the refocus image data has already been generated), the process proceeds to step S1052, and if not detected, the process proceeds to step S1051.

  In step S <b> 1051, the overall control unit 108 generates a refocus image data based on a parameter designated by the user, performing a refocus operation with the focus on an area corresponding to the parameter. Also, history information such as position information related to the refocus target and subject designation frequency is updated, and the process proceeds to step S1052.

  In step S1052, the overall control unit 108 reads out the image data detected in step S1050 or the refocus image data generated by the refocus calculation in step S1051 from a predetermined memory. And after displaying on a predetermined | prescribed display, a process transfers to step S1060.

  In step S1060, with respect to the refocus image playback device such as a camera or PC, three factors of the end operation of the playback mode, the transition operation to other modes except the playback mode, the refocus calculation of all refocus candidates, and the completion of display (Replay mode end factor) is determined.

  If there is no operation corresponding to the three determination factors in step S1060, the overall control unit 108 determines that the playback mode is to be continued, shifts the processing to step S1010, and repeats the above-described processing. If it is determined that there has been an operation corresponding to the three determination factors in step S1060, the overall control unit 108 moves the process to step S1070, and performs a playback mode end process.

  If it is determined in step S1010 that the user has not selected a parameter as described above, the overall control unit 108 proceeds to step S1011. In step S1011, it is determined whether or not the period during which no parameter is selected has exceeded a predetermined time. If it is determined that the period has not been exceeded, the process of this flow is terminated. If it is determined in step S1011 that no parameter is specified within a certain time, all the functions of the system may be terminated from the viewpoint of power saving or the like, or the sleep mode may be entered.

  If it is determined in step S1011 that the period without parameter selection is within a predetermined time, the overall control unit 108 proceeds to step S1020. The predetermined time is a time for waiting for the user to select a parameter, and is generated by predicting an image in a region where the user next desires a refocus display and performing a refocus operation on the predicted image. This is a period for storing image data in the memory.

  In step S1020, the overall control unit 108 determines whether or not a refocus image generation candidate (image or region) based on prediction exists in the refocus target image. That is, it is determined based on the first history information whether or not there is a subject candidate for which a region to be focused is specified. If it is determined that there is a candidate, the overall control unit 108 proceeds to step S1030. On the other hand, if it is determined in step S1020 that there is no candidate, the overall control unit 108 proceeds to step S1021.

  In step S1021, a candidate for a subject to be refocused that matches the current refocus playback mode set in the refocus image playback device such as the current camera or PC is selected, a refocus calculation is performed, and processing is performed. The process proceeds to step S1053.

  In step S1030, the overall control unit 108 reads the first history information used in the past refocus calculation, as described in step S910 of FIG. Then, in accordance with the frequency order, the order of subjects to be refocused is provisionally determined, and the process proceeds to step S1031. The provisionally determined process can be equivalent to the process shown in the flow shown in FIG.

  In step S1031, the overall control unit 108 determines whether or not the current setting of the refocus image playback apparatus has priority on the playback refocus mode. As modes to be used, there are a mode that prioritizes the playback refocus mode and a mode that prioritizes the past first history information. These settings may be set by the user via the user I / F 106 or may follow default settings. In step S1031, it is determined which of the set modes has priority.

  If it is determined in step S1031 that the playback refocus mode is prioritized, the overall control unit 108 proceeds to step S1033, and if it is determined that the playback refocus mode is not prioritized, the process proceeds to step S1032. The reproduction refocus mode priority in FIG. 10 is a mode for predicting a refocus area and its calculation order based on the current reproduction refocus mode or history information (second history information) of the reproduction refocus mode. is there.

  In step S1032, refocus calculation is performed by focusing on the area of the subject to be refocused according to the provisional refocus calculation order determined based on the first history information in step S1030, and refocus image data is generated. To do. In other words, the temporary refocus calculation order is used as the actual refocus calculation order. Then, the process proceeds to step S1053.

  In step S1033, in the refocus calculation target area, whether or not there is an image suitable for the current playback refocus mode, or the history information (second history information) of the playback refocus mode is displayed. It is determined whether there is a matching image among the candidates. If any exists, the process proceeds to step S1034. If none exists, the process proceeds to step S1032. That is, in step S1033, it is determined whether or not a refocus target area including a subject that is compatible with the playback refocus mode is included in the multi-viewpoint image data of the read image file.

  In step S1033, it is determined whether there is history information (second history information) in the playback refocus mode. When there is the second history information, the overall control unit 108 searches for an area including a subject that conforms to the reproduction refocus mode in order from the reproduction refocus mode with the highest frequency. In step S1033, an area that matches the current reproduction refocus mode, which is first in frequency, is searched in order from the area with the high calculation order temporarily determined in step S1030, and if an appropriate area is detected, the process is performed. The process proceeds to step S1034.

  Of course, when the search for the region suitable for the reproduction refocus mode with the first frequency is finished, it goes without saying that the search is performed in order of frequency so as to search for the region suitable for the reproduction refocus mode with the second highest frequency.

  As a result of the search, if a suitable image has not been detected, the process proceeds to step S1032. Then, as described above, the refocus calculation is performed according to the temporary refocus calculation order determined from the past first history information, and the overall control unit 108 shifts the processing to step S1053.

  As described above, the playback refocus mode includes the case of the current playback refocus mode set by the user and the case of using the second history information of the playback refocus mode. In the latter case, ranks corresponding to the frequencies are given to a plurality of types of reproduction refocus modes. Therefore, in step S1033, it can be said that the search is performed in the region of the order temporarily determined by the first history information in accordance with the order determined by the second history information.

  On the other hand, if a region suitable for the current reproduction refocus image is detected in step S1033, the refocus calculation order of the region including the image suitable for the reproduction refocus mode detected is changed (increased) in step S1034. ) Is executed.

  In step S1034, the overall control unit 108 raises the calculation order of the area suitable for the playback refocus mode detected in step S1033 by n to the calculation order temporarily determined in step S1030 (n is an arbitrary integer). . The value of n may be changed depending on the playback refocus mode. Specifically, if the playback refocus mode is person (family), person (acquaintance), person (other), landscape, or animal, the family is given the highest priority and the calculation order is automatically ranked first. The process of raising the rank and the friend by 2 ranks can be considered, but it is not particularly limited.

  When the change of the refocus calculation order of the region including the image conforming to the reproduction refocus mode in step S1034 and the refocus calculation according to the changed order are completed, the overall control unit 108 moves the process to step S1053.

  In step S1053, as described in the first embodiment, the parameters related to the refocus image (refocus image position information and the like) executed in step S1034 and the refocus image file are stored in the external memory 109 or the like. Then, the process again proceeds to step S1010.

  In step S1053, the same process is performed for the refocus image data generated in steps S1021 and S1032, and the process proceeds to step S1010.

  Various settings including the playback refocus mode can be set (selected) by the camera body, software on the PC, and the like. For example, a mechanical selection method such as a camera dial, a method of selecting from a display menu on a camera or a touch panel of a PC, and the like can be considered.

  Similar to the first history information (past frequency information) described in the first embodiment, the second history information in the reproduction refocus mode (for example, frequency information in the person mode and the landscape mode) has a history for each mode. It is updated (accumulated), and the stored contents of the history information storage unit 107 are also updated.

  By configuring as described above, the generation order of the refocus image specified by the user can be predicted, and the refocus calculation can be started before the user specifies the refocus area.

  This embodiment intends to further improve the user's preference and prediction accuracy of region selection by adding the current playback refocus mode or its second history information to the first history information of the first embodiment. It is a thing. In addition, using the period during which the user has not performed parameter selection, refocus calculation of the predicted area is performed and stored in the external memory 109 or the like, thereby shortening the time from parameter specification to refocus image display. Is also expected.

  The refocus image data predicted and generated by the refocus calculation as described above is stored in the external memory 109 or a storage device on the network in a format in which image position information is added as a header (see, for example, FIG. 7). Is done.

  Although the expression is omitted in the flowchart of FIG. 10, in an actual refocus playback apparatus, when a parameter is designated, it is ideal to interrupt the process at that point and forcibly shift the process to step S1051. .

  In the first and second embodiments, an example in which attention is focused on an object to be focused (hereinafter referred to as in-focus) when generating a refocus image has been described. However, when generating a refocus image, it may be possible to express the user's intention better when focusing on an object that is not focused (hereinafter referred to as non-focused).

  In the current image processing apparatus, when refocus image data is generated at the time of image display, there is no method for simply designating a place to be brought out of focus. For example, the technique disclosed in Patent Document 2 focuses on a plurality of subjects by pan focus when there are a plurality of subjects. However, when it is desired to remove only a specific subject from the focused state from a plurality of focused subjects, it has not been possible to accurately instruct some of the subjects to be out of focus. Therefore, there is a problem that an image in which a specific subject is out of focus cannot be easily generated and displayed.

  In the present embodiment, an example of generating an image in which a specific subject is out of focus based on designation of a place where the in-focus state is desired, and an example of a method for simply designating a place where the subject is desired to be out of focus. Show.

[Overview of hardware configuration of image processing device]
A hardware configuration of the image processing apparatus according to the third embodiment will be described in detail with reference to FIGS. 1, 11, 12, and 13. Since FIG. 1 is common to the first embodiment, a detailed description thereof will be omitted.

  FIG. 11 is a diagram illustrating an appearance of the multi-lens camera imaging unit 101.

As shown in FIG. 11, a plurality of imaging units 1101-1109 are arranged in front of the multiview camera imaging unit 101. Here, the imaging units 1101-1109 are equally arranged in a square lattice shape, and the vertical axis, the horizontal axis, and the optical axis of these imaging units are all in the same direction.

  In response to a user's imaging instruction, each of the imaging units 1101-1109 extracts an analog signal corresponding to the optical information of the subject imaged on the imaging device via an imaging lens, a diaphragm, or the like from the imaging element. Then, the analog signal is subjected to analog-digital conversion, and a set of image data subjected to image processing such as demosaicing is output as multi-viewpoint image data.

  By such a multi-lens camera imaging unit 101, it is possible to obtain an image data group obtained by imaging the same subject from a plurality of viewpoint positions. Here, an example in which the number of imaging units is nine is shown, but if there are a plurality of imaging units, the present embodiment can be applied, and the number of imaging units is arbitrary. Further, the arrangement of the imaging units does not need to be a uniform arrangement in a square lattice shape, and the arrangement of the imaging units is also arbitrary. For example, it may be arranged radially or linearly, or may be arranged at random.

[Image data format of multi-viewpoint image data]
In the example shown in FIG. 8 of the first embodiment, the position information presence flag 805 is mainly described as an example of 1-bit information indicating whether or not refocus position information exists in the image file. In this embodiment, the position information presence flag 805 may take a multi-value indicating whether the region of the position specified by the refocus position information is an in-focus object or an out-of-focus object. For example, a configuration as shown in FIG. In FIG. 12A, refocus position information 1203 corresponds to the refocus position information 803 in FIG. In FIG. 12A, the position information presence flag 1205 is composed of two field flags, chain information 1210 and attribute information 1220. The chain information 1210 indicates whether or not the refocus position information following the flag of the chain information 1210 is valid, and a series of refocus position information from 1210 to 1209 shown in FIG. Indicates whether to continue. For example, when the chain information 1210 is 1, the following series of attribute information 1220 and position information 1206 to 1209 are valid information. Then, it is shown that another group of refocus position information exists further behind the group of the series of refocus position information. The attribute information 1220 indicates the attributes of the position information indicated by the field information 1206 to 1209 following the attribute information 1220. For example, when the attribute information 1220 is 0, the field information of the position information 1206 to 1209 following the attribute information 1220 is handled as information indicating a position to be focused (in-focus position). On the other hand, when the attribute information 1220 is 1, the field information of the position information 1206 to 1209 following the attribute information 1220 is handled as information indicating a position (non-focus position) where an object to be blurred is present. Is called. That is, the attribute information 1220 can be said to be in-focus state data indicating the in-focus state.

  FIG. 12B shows a specific example when a multi-value flag is used for the position information presence flag 1205. FIG. 12B shows the extracted field when there are four fields of the refocus position information shown in FIG. 12A at the position of the refocus position information 1203 in FIG. Since the chain information 1210a to c is 1, the refocus position information of 1203a, 1203b, and 1203c is valid information. On the other hand, since the chain information 1210d is 0, the refocus position information of 1203d is invalid information. With this 1203d, the field of the refocus position information ends, and the multi-viewpoint image header information 810 shown in FIG. Indicates that it will continue. Further, since the attribute information 1220a and 1220b are 0, it indicates that 1203a and 1203b have position information about the focal point. In this case, it is shown that the object to be focused exists in two places. On the other hand, since the attribute information 1220c is 1, 1203c indicates position information regarding the out-of-focus position that is the position of the object to be blurred. That is, the example having the multi-value flag shown in FIG. 12B shows an example having three effective position information indicating two in-focus points and one non-in-focus point.

[Configuration Example of Refocus Operation Unit 103 and Process Flow]
A functional configuration example of the refocus calculation unit 103 and a flow of processing will be described with reference to the block diagram of FIG. In FIG. 13, a rectangular frame represents a processing module, and a rounded rectangle represents a data buffer constituted by a built-in SRAM or the like.

  In this embodiment, it is assumed that the data buffer is placed inside the refocus calculation unit, but in some cases, the RAM 102 or the like can be used via the data input unit / output unit. In this embodiment, it is assumed that the processing module in the refocus calculation unit 103 executes each process based on an instruction from the refocus calculation unit controller 1301. A control signal indicating an instruction from the refocus calculation unit controller 1301 is not shown. However, the entire processing described below can also be realized by executing an image processing program using one or a plurality of CPUs.

  In response to an instruction from the refocus calculation unit controller 1301, the data input unit 1302 captures imaging data and information about the device (for example, the multi-lens camera imaging unit 101) that captured the imaging data from the RAM 102 or the external memory 109. Enter accompanying information. Here, for example, the multi-viewpoint image header information 810 in FIG. 8 corresponds to the imaging information. In addition, multi-viewpoint image data 804 in FIG. 8 corresponds to imaging data. Of the input data, the imaging-accompanying information is stored in the buffer 1311 and the imaging data is stored in the buffer 1312. In this case, the imaging data includes a plurality of image data captured by a plurality of imaging units (for example, the imaging units 1101 to 1109 of the multi-lens camera imaging unit 101).

  Further, the imaging accompanying information stored in the buffer 1311 includes the imaging position (relative positional relationship) of each imaging unit in addition to the imaging information of each imaging unit as described above.

  Further, user instruction information is similarly input from the data input unit 1302 as necessary. This user instruction information includes contents that the user particularly wants to instruct when generating a refocused image, for example, information on a focus target to be focused or a non-focus target that is not desired to be focused. For example, the refocus position information 1203 in FIG. 12 corresponds to the user instruction information.

  When the acquisition of data necessary for processing is completed, the focus coordinate acquisition unit 1304 inputs reference image data from the buffer 1312 in accordance with the control of the refocus calculation unit controller 1301. The focus coordinate acquisition unit 1304 outputs focus coordinate information indicating a position to be focused on the display screen according to the user instruction information stored in the buffer 1313. The focus coordinate information is stored in the buffer 1315.

  Note that the reference image data is one of a plurality of pieces of image data that is imaging data stored in the buffer 1312 and may be any of the plurality of pieces of image data. In this embodiment, image data captured by the image capturing unit 1105 located in the center among the image capturing units 1101 to 1109 of the multi-lens camera image capturing unit 101 is used as reference image data.

  Next, the distance estimation unit 1303 inputs a plurality of image data from the buffer 1312 under the control of the refocus calculation unit controller 1301. The distance estimation unit 1303 inputs imaging incidental information from the buffer 1311. Then, the distance estimation unit 1303 generates the distance image data by estimating the depth value of the imaging scene by stereo matching based on the plurality of image data and the imaging accompanying information. The generated distance image data is stored in the buffer 1314.

  When the generation of the focus coordinate information and the distance image is completed, the virtual focus surface generation unit 1305 inputs the focus coordinate information, the distance image data, and the user instruction information under the control of the refocus calculation unit controller 1301. Generate focus plane information.

  The focus plane information is information for capturing in three dimensions the position of the camera and the subject to be focused at the time of imaging, such as the distance between the camera and the focus plane, the shape of the focus plane, and the depth of field. It consists of information. The focus plane information is stored in the buffer 1316.

  Finally, under the control of the refocus calculation unit controller 1301, the image composition unit 1306 receives a plurality of image data, imaging-accompanying information, distance image data, and focus plane information from the buffers 1312, 1311, 1314, and 1316, respectively. read out. Then, the image composition unit 1306 generates refocus image data.

The refocus image data is temporarily stored in the buffer 1317, then output from the refocus calculation unit 103 via the data output unit, and stored in the RAM 102 or the external memory 109. In some cases, the image is displayed on the display 105 via the graphic processor 104.

[Processing flow when an out-of-focus target is specified]
The flow of processing when an in-focus target is instructed will be described with reference to the flowcharts of FIGS. 1, 12, 13 and 14, and FIGS. The flowchart of FIG. 14 shows what processing instruction is given to each module including the refocus calculation unit 103 when the overall control unit 108 receives input of user instruction information.

  FIGS. 15 and 16 are diagrams illustrating the positional relationship between the imaging device and the imaging subject when an object to be subjected to image processing is imaged. In addition, it explains how the captured image is displayed on the display 105 and what instruction is given on the touch panel installed on the display 105.

  FIG. 15A shows a state where the positional relationship between the multi-lens camera imaging unit 101 and the imaging subject when the image shown in FIG. 15B is captured is looked down from above. The angle of view of the multi-camera imaging unit 101 is indicated by two lines 1510 and 1511 extending obliquely from the multi-camera imaging unit 101, and the subjects A, B, C, and D that are subjects within the range are displayed. Images are taken with mountains and trees. FIG. 15B is an initial image when the image displayed on the display 105 is captured.

  Here, first, the processing until the image of FIG. 15B is displayed will be described using steps S1401 to S1403 and steps S1414 and S1415 of FIG.

  In step S1401 of FIG. 14, when an image display is instructed, the overall control unit 108 determines whether or not the refocus position information 1203 of the stored multi-viewpoint image data is valid. If the overall control unit 108 determines that the refocus position information is valid, the overall control unit 108 proceeds to step S1402. In step S1402, the overall control unit 108 determines whether there is a temporary image file that matches the refocus position information. If the overall control unit 108 determines that a temporary image file exists, the overall control unit 108 proceeds to step S1403. In step S1403, the overall control unit 108 reads the image of the temporary image file from the external memory 109 or the like, sends it to the graphic processor 104, and displays it on the GUI screen of the display 105.

  If the overall control unit 108 determines in step S1402 that the temporary image file does not exist, the process proceeds to step S1415. In step S1415, under the control of the overall control unit 108, the refocus calculation unit 103 sets the virtual focus plane and the depth of field based on the refocus position information and generates image data. Next, in step S1403, the overall control unit 108 displays an image represented by the image data generated in step S1415.

  On the other hand, if the overall control unit 108 determines in step S1401 that the refocus position information is not valid, the process proceeds to step S1414. In step S1414, the overall control unit 108 first detects a target to be refocused as shown in FIG. 4 of the first embodiment, and adds a position including the target to the refocus position information. Next, the overall control unit 108 generates an image in step S1415, and displays the image in step S1403.

  If it is determined in step S1401 that the refocus position information is invalid, the overall control unit 108 is simply represented by reference image data without performing the processing shown in FIG. 4 of the first embodiment. You may perform the process which displays an image. In this case, in the example described below, it is assumed that the image represented by the reference image data is shown in FIG.

  The description on the case where the image on the GUI displayed in step S1403 is FIG. Here, an object indicated by a thick line in FIG. 15B indicates that it is a focusing target. In this case, the trees behind the persons A, B, C, D and C are displayed as in-focus objects. This is because the persons A, B, C, and D are designated as in-focus objects in the refocus position information of the multi-viewpoint image data. In this embodiment, as shown in FIG. 15A, the virtual focus plane is set at the position of a line segment 1501 passing through the persons A and B. Further, the setting is performed so that the depth of field is within the range indicated by the arrow 1502 so that the persons C and D fall within the in-focus range. A display image is generated based on the virtual focus plane and the depth of field. Therefore, as a result, an image focused on an object between the line segment 1503 far from the multi-view camera imaging unit 101 and the line segment 1504 near is displayed on the display 105. At this time, since the tree behind C is also within the range of the depth of field 1502, an image in focus is also displayed. Here, line segments 1501, 1503, and 1504 are represented by straight lines, but actually, a gentle arc is drawn according to the distance from the multi-view camera imaging unit 101. In this embodiment, this expression format is used for simple expression.

  Next, the steps of FIG. 14 relate to a method for generating image data in which a specific subject is out of focus based on designation of a place to be brought into the out-of-focus state, and a method for simply designating a place to be brought into the out-of-focus state. This will be described using S1404 and subsequent steps.

  When an image is displayed on the display 105 in step S1403, the overall control unit 108 waits for parameter selection by the user in step S1404. In the present embodiment, it is assumed that a touch panel installed on the display 105 is used as the user I / F 106. Next, in step S1405, the overall control unit 108 determines whether or not the user has touched a point on the screen with a finger to select a parameter (in this case, a position intended to change focus or non-focus). In step S1405, if the overall control unit 108 determines that the user has selected a parameter, the overall control unit 108 proceeds to step S1406. In step S1406, as a selection act in step S1405, the overall control unit 108 determines whether or not the time during which a specific point on the screen is pressed for selecting a parameter has continued for a certain period (long press). That is, it is determined whether the user has designated an area on the image for a certain period of time. In this embodiment, it is assumed that the act of long pressing is an instruction to make the selected point out of focus. For example, in this embodiment, it is determined that a long press is made when the button is continuously pressed for 3 seconds or more. In addition, the reference | standard of this time is an example, Time may be changed as needed. If the overall control unit 108 determines in step S1406 that the long press has been made, the process proceeds to step S1407. In step S1407, the overall control unit 108 determines whether or not the position designated by the long press is included in the refocus position information 1203 of the displayed multi-viewpoint image data. Here, whether or not the position specified by long press is included is determined by whether or not the specified position is inside the rectangle represented by the position information 1206 to 1209 of the valid refocus position information. The

  In step S1407, when the overall control unit 108 determines that the position designated by the long press is not included in the refocus position information, the process proceeds to step S1408. In step S1408, the overall control unit 108 creates an entry of refocus position information for newly expressing an area centered on the position selected by the user in step S1405. At this time, the field of the newly created attribute information 1220 is set to 1 which means an out-of-focus instruction.

  In step S <b> 1409, the overall control unit 108 performs area recognition for determining an out-of-focus range including the position designated by the user. This area recognition is performed by the image analysis unit 112 as described in the first embodiment. Based on the result of the region recognition, a range to be out of focus is set in the position information 1206 to 1209 of newly created refocus position information.

  In step S1407, if the overall control unit 108 determines that the position designated by the long press is included within the rectangle represented by the position information 1206 to 1209 of the valid refocus position information, the process proceeds to step S1418. To proceed. In step S1418, the overall control unit 108 sets the field of the attribute information 1220 of the refocus position information to 1, which means a non-focus instruction, as necessary.

  Here, an example is shown in which in-focus information and out-of-focus information and information on the refocus position are stored as part of an image data format for storing a multi-viewpoint image. On the other hand, in-focus and out-of-focus information at the same timing and image information cut out from the image based on information indicating the refocus position, etc. are combined and saved as a list independent of individual multi-viewpoint image data However, it can also be used. For example, it is conceivable to create and save a list of the format shown in FIG. Details and use of this list will be described in Example 4 described later.

  Next, in step S1410, the overall control unit 108 uses the multi-view image header information of the displayed multi-view image data as information for specifying the out-of-focus intensity according to the length of time that the button has been pressed for a long time. Add to 810. Details of the field of the multi-viewpoint image header information 810 are not described, but the information specifying the out-of-focus intensity includes the position of the field indicating the out-of-focus in the refocus position information and the intensity to be blurred. Saved in pairs. The strength to be blurred is set stronger as the time of long pressing is longer. However, when the maximum value of the strength of the blur is specified, the strength is not updated from the maximum value even if it is pressed longer.

  Next, in step S1411, the refocus calculation unit 103 changes the virtual focus plane and the depth of field based on the updated refocus position information. Then, image data is generated based on the virtual focus plane and the depth of field after the change. In step S1412, the overall control unit 108 displays the image generated in step S1411. An example of how the change of the virtual focus plane and the change of the depth of field for the out-of-focus state is performed will be described with reference to FIGS. 15, 16, and 17.

  In FIG. 15B, it is assumed that the person C is selected as an out-of-focus object, and the user continuously presses the position where the letter C is drawn on the display 105 with a finger for 3 seconds or longer. The overall control unit 108 monitors the situation, and passes data including new refocus position information to the refocus calculation unit 103. The refocus calculation unit 103 determines the virtual focus plane and the depth of field based on the flow shown in FIG. Similarly, the person D can be instructed as an out-of-focus target.

  For example, consider a case where the person C and the person D are excluded from the focus targets and it is required to create a picture in which only the persons A and B are in focus. In this embodiment, the persons A and B are on the virtual focus surface 1501 at the time of display in FIG. 15B, and the persons C and D are located slightly away from the virtual focus surface. In such a case, the person C and the person D can be excluded from the in-focus objects by adjusting the depth of field narrowly without changing the virtual focus plane. In this case, in this embodiment, as shown in FIG. 16A, the virtual focus plane does not move from the position of the line segment 1501 passing through the persons A and B, and the persons C and D are within the range of the depth of field. Setting is performed so that the depth of field is within a range indicated by an arrow 1602 so as to deviate. A display image is generated based on the virtual focus plane and the depth of field. That is, an image focused on the object between the dotted line 1603 and the dotted line 1604 (FIG. 16B) is generated and displayed on the display 105 in step S1412. In FIG. 16B, the thick dotted lines indicate that the persons C and D are out of focus.

  It is also conceivable that only the person C, not both the persons C and D, is excluded from the focus target, and it is required to make a picture that focuses on the persons A, B, and D. In this case, since the person C is closer to the multi-lens camera imaging unit 101 than the other persons A, B, and D to be focused, it is possible to cope by shifting the focus surface backward. Such a case will be described with reference to FIG. In this example, as shown in FIG. 17A, the virtual focus plane is set from the position of the line segment 1501 passing through the persons A and B to the position of the dotted line 1701 behind it. Further, the setting is performed so that the depth of field is in the range indicated by the arrow 1702 so that the person C is out of the range of the depth of field. A display image is generated based on the virtual focus plane and the depth of field. That is, the image figure 17 (b) focused on the object between the dotted line 1703 and the dotted line 1704 is generated and displayed on the display 105 in step S1412.

  In FIG. 17B, the thick dotted line indicates that the person C is out of focus. In this example, the tree behind the person D is newly included in the focus target. However, in the present embodiment, there is no particular restriction on the focus state with respect to objects other than the person, which is problematic. is not.

  Next, returning to FIG. 14, the description of the flowchart will be continued.

  When the image on the display 105 is updated in step S1412, in step S1413, the overall control unit 108 determines whether the user designation is further continued. The fact that the user designation continues further means that the long press is still continuing. In step S1413, if the overall control unit 108 determines that the long press is still continued, the overall control unit 108 returns to step S1410, changes the out-of-focus intensity according to the duration of the designated action, and if necessary. To recreate the image. If the overall control unit 108 determines in step S1413 that the long press has not continued, the process returns to step S1404 and waits for parameter selection by the user. If the overall control unit 108 determines in step S1405 that the user does not select a parameter, the process proceeds to step S1420. In step S1420, the overall control unit 108 determines whether there is an instruction to end the refocus display process. If the overall control unit 108 determines in step S1420 that there is no instruction to end the refocus display process, the overall control unit 108 returns to step S1404 and continues waiting for parameter selection by the user.

  If it is determined in step S1420 that there is an instruction to end the refocus display, the overall control unit 108 ends this display operation and ends a series of operations. The refocus display end instruction may be given by the user explicitly selecting a user IF such as a button or by an instruction based on a timeout when there is no instruction from the user for a certain period of time. Is possible.

  In step S1406, if the instruction from the user is on the touch panel but not a long press, the overall control unit 108 advances the process to step S1416. In step S <b> 1416, the overall control unit 108 determines whether or not the user's neighborhood is selected a plurality of times within a predetermined time. In this embodiment, when a location within a certain distance (for example, within a distance of 5 mm on the touch panel) is selected a plurality of times within a certain time (for example, within 2 seconds), the position is focused. It will be interpreted as an instruction to add. If it is determined in step S1416 that multiple selections have been made within a certain time, the overall control unit 108 adds the position selected by the user to the refocus position information as a focus target in S1417. The contents in step S1417 are almost the same as those in steps S1407 to S1409, and the main difference is that the value set in the field of attribute information 1220 is set to 0 which means a focus instruction. If it is not determined in step S1416 that a plurality of selections have been made within a certain period of time, parameters or images are not changed as shown in step S1419, and the overall control unit 108 The process proceeds to S1404.

  In the present embodiment, as a method for designating a target on which the user is not focused, a method of long pressing on a display image using a touch panel is used. However, designation by other methods is also possible. For example, a button for setting in-focus in advance may be provided on the touch panel screen, and a location pointed on the screen after touching the button may be recognized as a non-focus setting location. Further, when an object is specified on the touch panel screen, a menu may be displayed to instruct in-focus / in-focus.

  In addition, when a touch panel is not used, it may be possible to give an instruction with a mouse or the like. For example, it is possible to issue an instruction by linking a right click with an out-of-focus instruction in an application. In that case, it is also possible to give meaning by a long-press instruction for a right click or a plurality of instructions for a right click.

  By doing the above, it is possible to easily specify the target that is not focused in the system that can display the image with the focused position changed according to the user's preference after imaging It becomes. Further, it is possible to create an image in which a specific subject is out of focus based on a designation for not focusing.

Therefore, it is possible to eliminate the in-focus state unintended by the user and select only a specific subject to make it out of focus. As a result, it is possible to easily generate and display an image focused on only the target desired by the user.

  In the third embodiment, a method of generating a refocus image based on designation by designating an object to be focused and an object not to be focused is shown. By storing and managing the in-focus object and the out-of-focus object separately from the individual image data in the third embodiment, for example, if a refocus image focused on a specific person can be generated in a batch, the user's operation load Can be reduced. This is because the current image processing apparatus needs to perform operations according to the user's operation for each image when generating a focus plane when combining images taken from a plurality of viewpoints. Because there is. In this embodiment, an example of a technique for storing and managing a target to be focused and a target to be unfocused will be described. Hereinafter, an object to be focused is described as an in-focus object, and an object that is not focused is described as an out-of-focus object.

  Examples of the hardware configuration of the fourth embodiment are shown in FIGS. 1, 11, and 13. Since FIG. 1, FIG. 11 and FIG. 13 are common to the first embodiment and the third embodiment, detailed description thereof is omitted.

[Data format and data structure]
A data format for storing and managing in-focus objects and out-of-focus objects will be described in detail with reference to FIG. As described a little in the third embodiment, FIG. 18 shows a list format data format independent of individual multi-viewpoint image data. As for the data of the in-focus object or the out-of-focus object, data for each object is stored in a list format as indicated by 1801 to 1803. This list is hereinafter referred to as a focused / unfocused list or simply a list. Note that the data format is not limited to the list format as long as any stored object can be accessed.

  Data of each object in the list includes an identification code 1804, attribute information 1805, and recognition information 1806. The identification code 1804 is information for identifying the object, and is multi-value information indicating, for example, a person, an animal, a tree, or the like. Further, for example, the identification information may be classified in more detail such as a person, wife, eldest daughter, eldest son or the like. The identification code can be said to be identification data for identifying an object. The attribute information 1805 is 1-bit information indicating whether the object is an in-focus object or an out-of-focus object. One bit is sufficient to indicate in-focus / in-focus, but the number of bits of attribute information is not limited. The recognition information 1806 is information indicating the characteristics of the in-focus / non-in-focus object. For example, in the case of a person, the distance between feature points such as eyebrows and eyes, the area of a region surrounded by the feature points, and the luminance of pixels in the region. The recognition information 1806 is obtained by analysis by the image analysis unit 112. Based on the recognition information 1806, an in-focus object or an out-of-focus object is identified for the subject in the image. The recognition information can also be regarded as identification data for identifying an object.

  Next, the refocus position information 1203 field included in the image data format of this embodiment will be described in detail with reference to FIG. In the third embodiment, the position information presence flag 1205 has been described as an example of information including the chain information 1210 and the attribute information 1220. In the present embodiment, a configuration in which an identification code 1901 is further added is adopted. Here, the identification code 1901 is the same as the identification code 1804 in FIG. Other components are the same as those in the first embodiment, and thus the description thereof is omitted. The field of the refocus position information 1203 shown in FIG. 19 can have multi-value information as described in the third embodiment. That is, the position information of a plurality of in-focus objects or in-focus objects can be added to or deleted from the field of the refocus position information 1203.

  An example of the data structure for managing the in-focus / in-focus target list and the saved image will be described in detail with reference to FIG. The saved image is image data saved in the external memory 109 or the like, and is image data defined by the image data format of FIG. 8 including the refocus position information shown in FIG. The in-focus / in-focus target list and the image data shown in FIGS. 18 and 19 are classified and managed for each hierarchical group as shown in FIG. The in-focus / in-focus object list may be referred to as object definition information. The refocus position information addition processing to each image data based on the in-focus / non-focus target list described later is applied to the group in which the focus / non-focus target list exists and the image data in the group included in the group. It applies to. Details of the refocus position information addition processing will be described later.

  In FIG. 20, group 1 (family) 2005 is a group including group 1.1 (travel) 2006 and group 1.2 (athletic meet) 2007. The refocus position information addition processing based on the focus / non-focus target list 2002 in the group 1 (family) 2005 is applied to the next saved image. That is, the present invention is applied to a stored image 2004 in the group 1.1 (travel) and a stored image (not shown) existing in the group 1.2 (athletic meet) 2007 and its included group. On the other hand, the refocus position information addition process based on the in-focus / non-focus target list 2003 in the group 1.1 (travel) is applied only to the saved image 2004 in the group 1.1 (travel). Note that the global focus / non-focus target list 2001 is a focus / non-focus target list that adds refocus position information to all stored images without depending on a group. By including the focused / unfocused target list and the saved image in the group having such a hierarchical structure, the convenience for the user is improved. For example, in the focusing / non-focusing target list of group 1 (family) 2005, a family member is set as an object to be focused. On the other hand, in the in-focus / non-in-focus target list of group 1.1 (travel) 2005, for example, a companion person in the case of a tour trip is set as an object to be out of focus. By setting in this way, with respect to the stored images in the group 1.1, image data that is focused on a family member and is not focused on a tour companion person is generated.

  In the present embodiment, the above-described data structure is adopted, but the present invention can be implemented without being limited to this data structure.

[Processing flow for updating the focus / non-focus target list and refocus position information]
The processing flow for updating the in-focus / non-focus target list and adding refocus position information will be described in detail with reference to FIGS. 18, 19 and 21 to 25. The following explanation is the processing when updating the in-focus / non-focus target list, the processing when newly capturing an image, and the refocus position information based on the focus / non-focus target list at any timing This is divided into three processes for updating.

  First, the processing when the focus / non-focus target list is updated will be described. FIG. 21 to FIG. 23 show a processing flow when updating a list such as adding a new object or deleting an existing object with respect to one or a plurality of in-focus / in-focus object lists. In step S2101 in FIG. 21, the overall control unit 108 determines whether the user has designated a target in the stored image and instructed addition to the in-focus / in-focus target list. For example, an arbitrary stored image is displayed on the display 105, and it is determined whether or not there is an instruction to add to the in-focus / non-focus target list from the user via the user I / F. For example, when the user designates an arbitrary position in the saved image being displayed on the display 105 and selects an item to be added to the in-focus target list of the object at that position, the entire control is performed. The unit 108 determines that there is an addition instruction. Note that the in-focus / in-focus target list in step S2101 can be a list belonging to the same group as the saved image, as described with reference to FIG. Alternatively, the user may be allowed to select which group to add to the list. If the overall control unit 108 determines in step S2101 that there is no instruction to add a target to the list, the process proceeds to step S2107. If the overall control unit 108 determines in step S2101 that there is an instruction to add a target to the list, the process proceeds to step S2102.

  In step S2102, the overall control unit 108 assigns an identification code 1804 to the object instructed in step S2101 and stores it in the flash ROM 107 or the external memory 109 together with the object recognition information 1806.

Here, it is assumed that the user specifies a target from the stored image, and the image analysis unit 112 calculates and stores the recognition information of the target. However, recognition information such as general persons who are not specific individuals may be stored in advance in the Flash ROM 107 or the like. In this case, the recognition information 1806 stored in step S2102 may be a pointer indicating an address on the FlashROM 107 in which the recognition information is stored.

  Next, in step S2103, the overall control unit 108 determines whether the object to be added to the in-focus / in-focus target list instructed in step S2101 is an in-focus object or an out-of-focus object. The user can instruct whether to be in focus or out of focus by a method as shown in the third embodiment. If the object to be added to the list is a focus target, the overall control unit 108 proceeds to step S2104 and sets the bit of the attribute information 1805 to 0. If the object to be added to the list is an out-of-focus object, the overall control unit 108 advances the process in step S2105 and sets the bit of the attribute information 1805 to 1. In step S2106, the overall control unit 108 determines whether all the additional objects instructed in step S2101 have been processed. The overall control unit 108 repeats the processes from step S2102 to step S2105 until all the processes are processed.

  When all the targets to be added to the list have been processed, in step S2107, the overall control unit 108 determines whether or not there is an instruction to delete the target from the focused / non-focused target list. As for the deletion instruction, similarly to the addition instruction, for example, an arbitrary saved image is displayed on the display 105, and whether or not there is a deletion instruction from the in-focus / non-focus target list from the user via the user I / F. Determine whether. For example, when the user designates an arbitrary position in the stored image being displayed on the display 105 and selects an item to be deleted from the in-focus target list of the target object at that position, the entire control is performed. The unit 108 determines that there is a deletion instruction.

  If it is determined in step S2107 that there is no object deletion instruction from the list, the overall control unit 108 advances the process to step S2110. If it is determined in step S2107 that there is an object deletion instruction from the list, the overall control unit 108 advances the process to step S2108. In step S2108, the overall control unit 108 deletes the object instructed in step S2107 from the list. At this time, the list for deleting the object may be a list belonging to the same group as the saved image. Alternatively, the user may be allowed to select from which list the group belongs to be deleted. Or you may delete a target object from the list which belongs to all the groups. In step S2109, the overall control unit 108 determines whether all the objects instructed in step S2107 have been deleted from the list, and repeats the process of step S2108 until all the deletion objects are processed. When the process for all objects to be deleted from the list is finished, the overall control unit 108 advances the process to step S2110.

  In step S2110, the overall control unit 108 determines whether or not there is an instruction to update the refocus position information for the saved image when the in-focus / in-focus target list is updated. Here, the stored images can be all stored images included in the group to which the updated in-focus / in-focus target list belongs and the group included in the group. The refocus position information update is processing for adding or deleting a position information field as shown in FIG. 19 to the field of the refocus position information 803 in the format of FIG. As described above, the refocus position information 1203 shown in FIG. 19 can include values for a plurality of items. It should be noted that it is possible to allow the user to select whether or not to give a refocus position information update instruction at the time of list update in step S2110 by a user interface (hereinafter referred to as UI). Alternatively, the user may instruct when adding an object to the list in step S2101. Alternatively, it may be an instruction automatically set in advance by a user or a device. If it is determined in step S2110 that there is no refocus position information update instruction, the overall control unit 108 ends the process. If it is determined in step S2110 that there is a refocus position information update instruction, the overall control unit 108 proceeds to list addition reflection processing in step S2111.

  FIG. 22 shows details of the list addition reflection processing step S2111 in which the object to be added to the list is reflected in the refocus position information of the stored image, which will be described. First, in step S2201, the overall control unit 108 determines whether or not there is a process for adding an object to the in-focus / in-focus target list. If it is determined in step S2201 that there is no addition of an object to the list, the overall control unit 108 ends the list addition reflection process and advances the process to step S2112. If the overall control unit 108 determines in step S2201 that the object has been added to the in-focus / in-focus target list, the process proceeds to step S2202.

  In step S2202, the overall control unit 108 determines whether there is an additional object in the saved image. For example, the overall control unit 108 extracts one stored image from all stored images to which the focused / unfocused target list to which the target object is added is applied. Then, the overall control unit 108 determines whether there is an additional object in the saved image. This determination can be made based on whether the difference between the values is within a predetermined range by comparing the recognition information of the region in the stored image with the recognition information of the target object in the list by the image analysis unit 112. . If the overall control unit 108 determines in step S2202 that there is no additional object in the saved image, the process advances to step S2205. If the overall control unit 108 determines in step S2202 that there is an additional object in the saved image, the overall control unit 108 proceeds to step S2203. In step S2203, the overall control unit 108 updates the position information 1206 to 1209, the chain information 1210, the attribute information 1220, and the identification code 1901 in the refocus position information 1203 in FIG. . That is, the stored image is updated to the saved image having the refocus position information 1203 to which the items about the added object are added. Regardless of whether the added object is an in-focus object or not, the refocus position information 1203 is updated in this step. Since the position information 1206 to 1209, the chain information 1210, and the attribute information 1220 are the same as those in the first and third embodiments, the description thereof is omitted. For updating the identification code 1901, the identification code 1804 of the additional object in the list in-focus / non-in-focus object list is copied and stored.

  Next, in step S2204, the overall control unit 108 determines whether or not the refocus position information addition processing has been performed on all the stored images in the application range group with respect to one addition target to the list. If it is determined in step S2204 that there is an unprocessed saved image to be processed, the overall control unit 108 returns the process to step S2202 to perform a refocus position information addition process for the next saved image. If it is determined in step S2204 that there is no unprocessed saved image to be processed, the overall control unit 108 advances the process to step S2205. In step S2205, the overall control unit 108 determines whether or not refocus position information addition processing has been performed for all objects to be added to the list. If there is an unprocessed additional object, the overall control unit 108 returns the process to step S2202, and performs a process related to the next additional object. If it is determined in step S2205 that there are no unprocessed additional objects, the overall control unit 108 ends the list addition reflection process and advances the process to step S2112.

  FIG. 23 is a diagram showing details of the list deletion reflection processing step S2112 for reflecting the object to be deleted from the list on the refocus position information in the stored image data. First, in step S2301, the overall control unit 108 determines whether or not the object has been deleted from the in-focus / in-focus object list. When it is determined that there is no object to be deleted from the list, the overall control unit 108 ends the list deletion reflection process and the list update process is completed. If it is determined that there is an object to be deleted from the list, the overall control unit 108 advances the process to step S2302. In step S2302, the overall control unit 108 determines whether or not refocus position information for the deletion target exists in the saved image data regarding the deletion target. The determination can be performed by comparing the identification code 1804 of the object to be deleted with the identification code 1901 in the refocus position information added to the stored image data. If it is determined that there is no matching ID code, the overall control unit 108 advances the process to step S2304. If it is determined that there is a matching identification code, the overall control unit 108 advances the process to step S2303. In step S2303, the overall control unit 108 deletes the refocus position information having the identification code determined to match in step S2302. That is, the overall control unit 108 deletes the identification code 1901 shown in FIG. 19 and the chain information 1210, attribute information 1220, and position information 1206-1209 associated with the identification code 1901. Note that the refocus position information can have multiple values as described above. Therefore, when the refocus position information includes a plurality of fields (that is, information related to a plurality of objects), only information related to the identification code determined to match in step S2302 is deleted.

  In step S2304, the overall control unit 108 determines whether or not all the stored images in the application range group of the deletion target object from the list have been processed. If it is determined that there is an unprocessed stored image to be processed, the overall control unit 108 returns the process to step S2302 to process the next stored image. If it is determined that there is no unprocessed stored image to be processed, the overall control unit 108 advances the process to step S2305. In step S2305, the overall control unit 108 determines whether or not the refocus position information deletion process has been performed for all the objects that have been deleted from the list. If it is determined that there is an object to be deleted from the unprocessed list, the overall control unit 108 returns the process to step S2302 to perform a process related to the next object to be deleted. When it is determined that there is no object to be deleted from the unprocessed list, the overall control unit 108 ends the list deletion reflection process and the list update process is completed.

  Next, a processing flow when adding refocus position information based on a focused / unfocused target list to a captured image at the time of image capturing will be described in detail with reference to FIG. First, in step S2401, the overall control unit 108 determines a group to which the captured image belongs in accordance with a user instruction. For example, the overall control unit 108 displays a setting screen for selecting a group to which the captured image belongs on the display 105, for example, and determines a group to which the captured image belongs in accordance with an instruction from the user via the user I / F. Alternatively, it may be an instruction automatically set in advance by a user or a device.

  In step S2402, the overall control unit 108 determines whether there is a refocus position information addition instruction for the captured image. Similar to step S2401, the overall control unit 108 displays a setting screen for selecting whether or not to add refocus position information to the captured image on the display 105, for example. Then, it is determined whether or not there is a refocus position information addition instruction by an instruction from the user via the user I / F. Alternatively, it may be an instruction automatically set in advance by a user or a device. If it is determined that there is no refocus position information addition instruction, the overall control unit 108 ends the process. If it is determined in step S2402 that there is a refocus position information addition instruction, the overall control unit 108 advances the process to step S2403. The refocus position information addition instruction can be made to be selected by the user via the UI as to whether or not to instruct when the list is updated.

  In step S2403, the overall control unit 108 determines a focusing / non-focusing target list to be applied based on the group to which the captured image belongs. In step S2404, the overall control unit 108 performs general object recognition on the captured image and extracts an object. In step S2405, the overall control unit 108 extracts a predetermined number of objects extracted in step S2404. As an example, the overall control unit 108 performs face recognition processing as general object recognition in step S2404, and performs processing so as to extract a predetermined number of detected face areas in descending order of the area of the face area in S2405. Can do. The object in step S2404 is not limited to a face, and may be an animal, a tree, or a combination of these. In addition, the method of extracting a predetermined number of objects in step S2405 is not limited to the order in which the object area is large. For example, using the ranking by other methods, such as applying in order from the closest to the center of the image for the in-focus object, and ranking in ascending order from the center of the image for the in-focus object. Also good. By performing the processing in steps S2404 and S2405, the processing load in steps S2406 to S2408, which will be described later, can be reduced. However, step S2404 and step S2405 are not essential processes, and the present embodiment can be implemented without being included.

  In step S2406, the overall control unit 108 determines whether or not the object extracted in step S2405 is a target in the in-focus / in-focus target list. This determination is performed by comparing the object extracted in step S2405 with the recognition information in the in-focus / in-focus target list. If it is determined that the extracted object is not an object in the list, the overall control unit 108 advances the process to step S2408. If it is determined that the extracted object is a target in the list, the overall control unit 108 advances the process to step S2407. In step S2407, refocus position information addition processing is performed on the captured image. Step S2407 is the same processing as step S2203, and detailed description thereof is omitted.

  In step S <b> 2408, the overall control unit 108 determines whether or not the processing in steps S <b> 2406 and S <b> 2407 has been performed for all combinations of the target and extraction objects in the in-focus target list to be applied. If not completed for all combinations, the overall control unit 108 returns the process to step S2406 to perform the next combination process. If all combinations have been completed, the overall control unit 108 ends the overall processing during imaging.

  Next, a processing flow when updating the refocus position information of the stored image based on the focus / non-focus target list at that time at an arbitrary timing will be described with reference to FIG.

  First, in step S2501, the overall control unit 108 accepts selection of an image for which refocus position information is updated. The selection may be performed by the user through the UI, or may be automatically selected according to the device settings. In step S2502, the overall control unit 108 determines an in-focus / non-focus target list to be applied based on the group to which the refocus position information is updated as an application list.

  In step S2503, the overall control unit 108 determines whether or not the refocus position information related to the object in the application list determined in step S2502 already exists in the saved image selected in step S2501. This determination can be made by checking whether or not the refocus position information having the same identification code as the identification code of the object in the application list is included in the stored image data. If it is determined in step S2503 that the refocus position information regarding the object in the application list already exists, the overall control unit 108 advances the process to step S2510. If it is determined in step S2503 that there is no refocus position information regarding the object in the application list, the overall control unit 108 advances the process to step S2504.

  In step S2504, the overall control unit 108 determines whether or not the object in the list exists in the image selected in step S2501. The processing in step S2504 is performed based on whether the difference between the values is within a predetermined range by comparing the recognition information of the area in the stored image with the recognition information of the target object in the list by the image analysis unit 112. be able to. That is, in step S2503, the determination is made based on the presence or absence of the identification code, and in step S2504, the determination is made by comparing the recognition information. If the object in the list does not exist in the saved image selected in step S2501 in step S2504, the overall control unit 108 advances the process to step S2506. If an object in the list exists in the saved image selected in step S2501 in step S2504, the overall control unit 108 advances the process to step S2505. In step S2505, the overall control unit 108 adds refocus position information to the saved image. Since the processing in step S2505 is the same as that in step S2203, detailed description thereof is omitted.

  On the other hand, in step S2510, the overall control unit 108 includes attribute information of the object in the application list determined in step S2502, and attribute information included in the refocus position information determined to have the same identification code in step S2503. Compare If the attribute information matches, the overall control unit 108 determines that the attribute information has not been changed, and proceeds to step S2506. If the attribute information does not match, the overall control unit 108 determines that the attribute information has been changed, and proceeds to step S2511. In step S2511, the overall control unit 108 overwrites the attribute information included in the refocus position information with the attribute information of the object in the application list determined in step S2502. This is applied in the case where what has been a focus target until now is out of focus.

  In step S2506, the overall control unit 108 determines whether or not the refocus position information addition processing for the stored image has been completed for all the objects in the focused / non-focused object list to be applied. If there is an unprocessed object in the list, the overall control unit 108 returns the process to step S2503 to process the next object. If there is no unprocessed object in the list, the overall control unit 108 advances the process to step S2507.

  In step S2507, the overall control unit 108 determines whether the refocus position information added to the saved image selected in step S2501 relates to an object that does not exist in the application list determined in step S2502. Determine whether or not. This determination can be made by checking whether there is refocus position information having an identification code that does not correspond to any of the identification codes of the objects in the application list. If there is refocus information regarding an object that does not exist in the application list, the overall control unit 108 advances the process to step S2508 to delete the refocus information added to the saved image. If there is no refocus position information regarding the object that does not exist in the application list, the overall control unit 108 advances the process to step S2509.

  In step S2509, the overall control unit 108 determines whether or not refocus position information update processing has been performed on all the stored images selected in step S2501. If there is an unprocessed saved image, the overall control unit 108 returns the process to step S2502 for processing the next image. If there is no unprocessed saved image, the overall control unit 108 ends the process.

  By performing processing as shown in FIG. 25, the update of the in-focus / non-in-focus target list can be applied to any saved image at any timing.

  21, 24, and 25, the process of updating the list and updating the refocus position information of the stored image has been described. As a subsequent process, a process for generating image data in which a focus surface is newly formed as described in the third embodiment is performed on the saved image to which the refocus position information is added or deleted. This stored image update processing may be performed after each processing of FIGS. 21, 24, and 25, or may be performed in accordance with an instruction from the user.

  By storing and managing in-focus objects and in-focus objects separately from individual image data as described above, images that reflect the user's intention can be quickly displayed when combining images captured from multiple viewpoints. It can be created and displayed.

  In addition, it is possible to collectively instruct refocus images belonging to a certain group so as to focus on a specific target, and it is possible to reduce the load of the user's operation.

  The present embodiment will be described in detail with reference to FIGS. 1, 11, 12, and 18 to 27. FIG. Since FIG. 1, FIG. 11, FIG. 12, and FIG. 18 to FIG.

  In the fourth embodiment, an example is shown in which a focusing / non-focusing target list is stored and managed, and refocus position information is collectively updated to image data based on this list. In this embodiment, an example of a method for easily updating the recognition information 1806 of the in-focus / non-in-focus object specified by the user specifying the object from the stored image to a more discriminating one is shown. . In addition, an example of a technique in which the user determines what is different from the user's intention among the refocus position information generated based on the in-focus / non-in-focus target list, and makes a simple correction will be shown.

  In this embodiment, the data format of the in-focus / non-focus object list in Embodiment 4 is as shown in FIG. In FIG. 26, in addition to the data format of FIG. 18, an object cut-out image 2601 obtained by cutting out the area of the object for each object is added. An object cut-out image (hereinafter referred to as a cut-out image) 2601 is obtained by cutting out a recognized object region when a user designates an object to be added to the in-focus / non-focus object list from the image. It is done.

  FIG. 27 shows an example of a UI for the user to update the recognition information of the object in the in-focus / non-focus object list and correct the refocus position information different from the user's intention.

  In FIG. 27A, 2701 to 2703 display images corresponding to the cut-out images 2601 of the respective in-focus objects in the in-focus / in-focus object list. Similarly, 2704 and 2705 display cut-out images of unfocused objects. Hereinafter, the selection operation using the UI will be described as an operation on the touch panel. However, a selection method by moving the cursor with a mouse or a selection method by button operation may be used, and the operation method is not limited. Further, a tap operation described below refers to a series of operations in which a certain range on the touch panel is pressed for a certain period of time and then released.

  Here, a case is considered where the user operates recognition information and refocus position information for the person A who is the object in the list. First, the user taps an image 2701 on FIG. A thick frame surrounding 2701 represents selection. Next, by tapping the recognition data update button 2719, the display is switched from FIG. 27 (a) to FIG. 27 (b).

  Reference numerals 2706 and 2707 display object cutout images 2601 in the in-focus / in-focus object list to which an identification code indicating the person A is assigned. In the example of FIG. 27, the same identification code 1804 indicating the person A is assigned to the object data 2706 and 2707. In this way, a plurality of object data having the same identification code may be included in the list. When there are a plurality of data having the same identification code, the cutout image displayed in FIG. 27A described above may be any one of the cutout images for the object having the same identification code. . Further, when there are a plurality of data having the same identification code, the comparison of the recognition information described in the previous embodiment can be performed for each stored image using the plurality of identification codes.

  Reference numerals 2708 to 2714 in FIG. 27B are cut-out images obtained from image data including refocus position information to which the same identification code as the object 2706 or 2707 in the in-focus / non-focus object list is assigned. is there. This cut-out image can be obtained by cutting out a rectangular area indicated by the coordinates of the position information 1206 to 1209 in the refocus position information 1203 included in each image data.

  Here, 2708 to 2714 are all desirably person A, but in the example of FIG. 27B, person F2709 and person G2714 are displayed due to misrecognition. At this time, the user can select 2709 and 2714 by tapping and select an application cancel button 2718 to delete the refocus position information regarding 2709 and 2714 from the corresponding image data. That is, the field of the refocus position information including the identification code indicating the person A is deleted from the image data from which the images 2709 and 2714 are cut out. As a result, when the recognition data is updated next time, the identification code indicating the person A is not included, so that it is not extracted as an application target in FIG.

  In addition, it is desirable that the refocus position information for focusing on the person A is generated for all the persons A in the application target saved image. However, depending on the recognition information of the person A included in the in-focus / non-focus target list, it is conceivable that the refocus position information is not necessarily generated for all the persons A.

  In order to improve this, the user first taps and selects an image determined to be more suitable for identification from 2708 to 2714. Next, by tapping an add to list button 2717, the image can be added to the in-focus / non-in-focus object list as an in-focus object having the identification code of the person A.

  Furthermore, the objects in the lists shown in 2706 and 2707 can be deleted by tapping them and selecting a delete button 2715.

  When a list reapply button 2716 is tapped after a series of list update processing, all the refocus position information to which the identification code of person A is assigned is deleted from the image data of the stored image. Thereafter, the refocus position information is added to the applied storage image again by the method of the fourth embodiment using the recognition information of the person A after the list update. The result is reflected on the UI of FIG. Here, the recognition information of the additional object to be updated in the list is obtained by the image analysis unit 112 processing the extracted image or the original image of the additional object.

  FIG. 27C shows a screen as a result of adding the object 2711 to the list in FIG. 27B and re-applying the list. The added image is shown at 2720 as an object in the list. Reference numerals 2701 to 2714 and 2721 denote cut-out images from the saved image to which the refocus position information is added based on the objects 2706, 2707, and 2720 in the list. By adding the object 2720 to the list, a cutout image 2721 is displayed in addition to the seven cases shown in FIG. This indicates that refocus position information associated with an identification code indicating the person A is added to the cut-out original image 2721.

  By performing the processing as described above, it is possible to update the recognition information of the list to one having higher discriminability while confirming the cut-out image targeted by the user for focusing and non-focusing. It is also possible to easily delete unnecessary information from the generated refocus position information. In addition, based on the list update, it is possible to instruct in-focus / in-focus for refocused images belonging to a certain group, reducing the user's operation load. Become.

  This embodiment will be described in detail with reference to FIG. The description common to the above-described embodiment is omitted.

  The system configuration in this embodiment is shown in FIG. The system in this embodiment includes a digital camera 2801 having the configuration shown in FIG. 1, an image display terminal 2802 such as a PC, a mobile phone, and a tablet, and a server 2804. These have a communication function for connecting to a network. A digital camera 2801, an image display terminal 2802 such as a PC, a mobile phone, and a tablet, a server 2804, and the like are connected to the network 2803.

  The operation of the in-focus / non-in-focus target list shown in the fourth and fifth embodiments can be performed more efficiently by cooperation through a network of devices as shown in FIG.

  The server 2804 has identification information for identifying a general object that a user, such as a person, an animal, a power pole, or a tree, does not individually register in the in-focus / in-focus object list. Further, it has a function of generating refocus position information based on the recognition information and the information on the in-focus state. Further, the server 2804 may have a function of generating a refocus image based on the refocus position information.

  When a refocus image is generated on the server 2804, the refocus image is transferred on the image display terminal 2802 by transferring the refocus image to the image display terminal 2802 such as a PC, a mobile phone, or a tablet via the network. You can also browse.

  Further, it is assumed that the image display terminal 2802 can update the in-focus / non-focus target list as shown in the fifth embodiment using the display screen. In this case, the focus / non-focus target list may be downloaded from the server 2804 to the image display terminal 2802, or there may be a mechanism capable of directly changing data on the server 2804. When downloaded to the image display terminal 2802, the list is uploaded to the server after the focus / unfocus target list update process. Thereafter, the refocus position information can be generated on the server side in accordance with the focus / non-focus target list updated as necessary.

  The digital camera 2801 communicates with the server 2804 to obtain information about a focused / unfocused object that can be processed by the server. This information acquisition can be automatically performed when communication between the server 2804 and the digital camera 2801 is performed after the focus / non-focus target list on the server is updated. . It is also possible to notify the user of the contents of the update through the UI. When the setting is made so as to notify the user when the update is made, the user selects an in-focus / non-in-focus object and saves it in the digital camera 2801. In this way, the digital camera 2801 can optimize the display of an image captured thereafter based on the updated in-focus / in-focus target list.

  Image data captured and stored in the digital camera 2801 and in-focus / non-focus object information to be processed by the server selected and stored by the user are transferred to the server 2804 via the network 2803. The image data transferred to the server 2804 includes refocus position information added at that time. The server 2804 performs refocus position information generation processing for each image data based on in-focus / non-focus object information set by the user. The generated refocus position information is transferred again to the digital camera 2801 via the network 2803, and the refocus position information of the corresponding image in the digital camera 2801 is updated. At this time, the refocus image data generated by the server 2804 based on the refocus position information may be transferred to the digital camera 2801 at the same time.

  With this configuration, the processing load of refocus position information generation / refocus image generation on the digital camera 2801 can be reduced, and the data amount of the recognition information stored on the digital camera 2801 can be reduced. It becomes possible to do.

  In the fourth embodiment, how to change the basic virtual focus plane and the depth of field corresponding to the in-focus / in-focus instructions has been described. In this embodiment, a method for changing the virtual focus plane in response to a more complicated request will be described. This will be described with reference to FIGS. 15 and 29 to 33.

  In FIG. 15 (b), it is assumed that the person C is long pressed on the screen and an instruction to make it out of focus is issued. At this time, it is assumed that the user has set a policy of “change the appearance of things other than the designated target as much as possible” as a policy for changing the virtual focus plane. In this case, it is possible to blur the person C by reducing the depth of field only for the part related to the person C on the angle of view without changing the virtual focus plane. FIG. 29 (a) shows an example of such a case. Although the virtual focus plane 1501 is not changed, the depth of field is set shallow as indicated by an arrow 2902 only in the portion related to the angle of view of the person C. Therefore, the in-focus range is between the dotted line 2903 and the dotted line 2904, and the person C is out of the in-focus range. FIG. 29B shows a display image at that time.

  With reference to FIGS. 30 and 31, a method for changing the focus surface when one of the persons on the same virtual focus surface is desired to be focused will be described. In FIG. 30A, the persons A, B, and E are in the focused state assuming that they exist on the same virtual focus plane 3001. Since the person D is not an object to be focused at this time, it is assumed that the depth of field 3002 is determined so as to remove the person D from the focusing range. Therefore, in FIG. 30, the focusing range is between the straight line 3003 and the straight line 3004. A display image on the display 105 at that time is shown in FIG.

  Assume that on the screen of FIG. 30B, the person E is long pressed on the screen, and an instruction to make it out of focus is issued. At this time, since the persons A, B, and E exist on the same virtual focus plane, the person E cannot be brought out of focus only by adjusting the depth of field. In such a case, it is possible to cope by shifting a part of the virtual focus surface. Such a case is shown in FIG. In FIG. 31A, the virtual focus surface 3101 is bent near the person E toward the side far from the multi-lens camera imaging unit 101. Since the depth of field has not been changed, the line indicating the depth of field range is also bent as indicated by dotted lines 3103 and 3104 in accordance with the change of the virtual focus plane. By this operation, as shown in FIG. 31B, an image focused on only the persons A and B is displayed on the display 105.

  An example in which the virtual focus surface is defined by a curved surface will be described with reference to FIGS. FIG. 32A shows a case where images focused on persons A, B, C, and D are combined. In this case, the virtual focus surface 3201 is defined as a curved surface passing through the persons A, B, C, and D. Therefore, as shown in FIG. 32 (b), the persons A, B, C, and D are each in focus and the depth of field is shallow (the degree of blur is large except where the subject is in focus). An image can be displayed on the display 105.

  Here, it is assumed that the C image on the display 105 is long pressed to designate the person C as a non-focus target. FIG. 33 shows the resulting image and the situation of the virtual focus plane. FIG. 33A shows a state in which the virtual focus surface 3301 is changed to pass through the place of the tree behind the person C, not the person C. As described above, since the depth of field of this image is shallow, the person C becomes a blurred image as shown in FIG.

  The flow of processing for synthesizing an image by defining the virtual focus surface as a curved surface has been described above with reference to FIG. Here, a characteristic phenomenon in defining as a curved surface will be described with reference to FIG. 13 again.

  The virtual focus surface generation unit 1305 generates the shape of the focus surface according to the instructed number and position of objects to be focused. For example, in the case of FIG. 32, an instruction is given to focus on the persons A, B, C, and D. At this time, the virtual focus plane generation unit 1305 determines the two-dimensional coordinates (XA) corresponding to the display screen of the persons A, B, C, and D (here, the image in FIG. 33B) from the focus coordinate information in the buffer 1315. , YA) (XB, YB) (XC, YC) (XD, YD). Further, from the distance image in the buffer 1314, the three-dimensional spatial positions (xA, yA, zA), (xB, yB, zB), (xC) when the persons A, B, C, and D are imaged using the two-dimensional coordinates. , yC, zC), (xD, yD, zD). In addition, the virtual focus plane generation unit defines the order of the curve to be a base for generating the focus plane based on the number of objects to be focused. In this embodiment, “the number of objects to be focused—1” is the order. In this example, since there are four subjects A, B, C, and D to be focused, the number of targets to be focused is 4, and the order of the curve is defined as 3. Note that this order determination method is not uniquely determined by this method, and if there are many objects to be focused, an upper limit may be set. Further, in the actual positional relationship of objects to be focused, the order of the curve may be lower than the order shown above.

Next, the virtual focus surface generation unit 1305 defines an actual curved surface. FIG. 32A shows a state in which the three-dimensional space where the subject is present is observed from the y-axis direction. That is, the depth direction in FIG. 32A is the z-axis, which is the distance between the imaging unit and each subject. Now, since the degree of the curve is defined as 3, the polynomial corresponding to the curve is as follows.
z = ax ^ 3 + bx ^ 2 + cx + d

  The three-dimensional spatial positions (xA, yA, zA), (xB, yB, zB), (xC, yC, zC), (xD, Solves simultaneous equations with values corresponding to x and z of yD, zD). As a result, a curve on the xz plane can be obtained. There are various methods for obtaining such as using a matrix operation and obtaining an approximate curve by the method of least squares, but this is omitted here. The degree and coefficient of the polynomial obtained here become the focal plane information expressing the shape of the focal plane stored in the buffer 1316.

  When the focal plane information is obtained, the image composition unit 1306 expands the curve indicated by the obtained polynomial in the y-axis direction, and generates a virtual focal plane. Then, the image data of the buffer 1312 is weighted for each individual eye and added to synthesize an image. At that time, for the positions corresponding to the persons A, B, C, and D on the virtual focus plane, an image focused on the virtual focus plane is synthesized. For people other than people A, B, C, and D, the blur condition is calculated from the distance between the three-dimensional space position where the imaging target actually exists and the position where the virtual focus plane exists, Synthesize an image with a corresponding blur effect. The calculation of the degree of blur is performed in consideration of the set depth of field. By using the curved focus surface in this way, it is possible to selectively defocus only the designated target as shown in FIGS.

  By doing as described above, it is possible to create an image in which only one of the two objects at the same distance from the multi-view camera imaging unit 101 is selectively blurred. Therefore, the user can easily select and instruct the object displayed on the screen without focusing on the distance or depth of field, and create an intended image. It becomes possible.

  In the embodiments so far, images that are simultaneously captured from different viewpoints as a plurality of images using the multi-camera imaging unit 101 have been used. For example, it is also possible to use images captured at successive times. For example, at two or more consecutive times, it can be considered that the camera position is changed slightly and the image is taken while the camera lens focal position is changed slightly. In either case, if the subject does not move and the change in the position of the camera or the change in the focal position of the lens is known, it is possible to create a distance image by performing distance estimation using a plurality of images.

  FIG. 34 shows an example in which continuous imaging is performed by slightly changing the focal position of the camera lens. In FIG. 34, reference numeral 3401 denotes a camera imaging unit capable of continuous shooting. The angle of view of the camera imaging unit 3401 is indicated by two lines 3410 and 3411 extending obliquely from 3401, and the subjects A, B, C, and D that are subjects within the range are imaged together with mountains and trees. ing.

  Here, it is assumed that an image in which the virtual focus plane 3402 is set so as to focus on the person C at time t1 is captured. Assume that an image in which the virtual focus plane 3403 is set so that the persons A and B are focused at time t2 immediately after that is captured. Furthermore, it is assumed that an image in which the virtual focus surface 3404 is set so as to focus on the person D at time t3 is captured.

  Here, it is assumed that the intervals between the times t1, t2, and t3 are sufficiently short so that the movement of the subject can be ignored. In that case, it is possible to make a distance image by estimating the distance from the images captured by the three virtual focus planes, the focal length information at that time, and the captured image information. If a distance image is created, it is possible to create an image in which the focal position is changed along the processing flow described with reference to FIG. 13 in the third embodiment. Therefore, it is possible for the user to designate an area that is not desired to be focused on the display image, generate a focus plane so that the designated area becomes an out-of-focus area, and combine and display the image.

<Other examples>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

A determination means for determining a target order to be focused based on history information indicating an operation history;
Generating means for sequentially generating composite image data focused on the object from a plurality of viewpoint positions according to the order determined by the determining means, from multi-view image data obtained;
Control means for displaying the generated composite image data,
The image processing apparatus is characterized in that the generation means sequentially generates the composite image data according to the order while there is no parameter input from the user for the displayed composite image data. When there is an input of a parameter from the user for the displayed composite image data, the control means
Determining whether composite image data focused on a target corresponding to the input parameter is generated by the generation unit;
If it is determined that the generating means has generated, the generated composite image data is displayed,
If it is determined that the generation unit has not generated, the generation unit generates composite image data focused on a target corresponding to the parameter,
The image processing apparatus according to claim 1. Recognizing means for recognizing an object included in the image indicated by the multi-viewpoint image data;
The image processing apparatus according to claim 1, wherein the determination unit extracts an object recognized by the recognition unit as the target.   2. A history information storage unit that stores the number of times that a target including an area corresponding to a parameter input by a user is selected as the history information with respect to the displayed composite image data. 4. The image processing apparatus according to any one of items 1 to 3. A determination means for determining whether or not a mode indicating an object to be focused is preferentially set;
5. The determination unit according to claim 1, wherein, when the determination unit determines that the mode is preferentially set, the determination unit changes the order of objects to be adapted to the mode. Image processing apparatus.   The image processing apparatus according to claim 5, wherein the determination unit changes the order by using second history information indicating an operation history in the mode. A determination step for determining a target order to be focused based on history information indicating an operation history;
A generation step for sequentially generating composite image data focused on the object from multi-viewpoint image data obtained by imaging from a plurality of viewpoint positions according to the order determined in the determination step;
A control step of displaying the generated composite image data,
In the generation step, the composite image data is sequentially generated in accordance with the order while no parameter is input from the user to the displayed composite image data. Program for functioning as an image processing apparatus according to computer to any one of claims 1 to 6.

Images