JP2018107553A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which, in an imaging apparatus for wipe-displaying as sub-screens images photographed by a plurality of separate cameras to an image photographed by a main camera, if an image with a high degree of similarity to the image being captured by the main camera is output as a sub-screen, there is no difference between a main-screen and a sub-screen and meaningless images even if viewed at the same time are wipe-displayed.SOLUTION: For each image of a separate camera shooting the same subject as the subject shot by a main camera, a degree of difference indicating a degree of a difference from the image photographed by the main camera is obtained. From the degree of difference obtained for each image of a separate camera, an image of a camera calculated as having the greatest difference from the image of the main camera is selected for a sub screen.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an imaging apparatus and a control method in the imaging apparatus.

  Conventionally, to display the same scene from multiple perspectives, the parent image captured by the main camera can be displayed on the image display device as a sub-screen. There is an imaging device having a function of recording an image in a displayed state. For example, in Patent Document 1, in order to enable shooting while comparing the state of the same subject in real time, a child screen is wiped out on the parent screen of the image display device, and an image to be displayed as the child screen image is displayed. There has been proposed an imaging apparatus that extracts and selects an image having a high similarity to the parent screen from recorded images.

Japanese Patent No. 4173379

  For example, in an imaging device that displays or records a combination of a main screen and a sub-screen, when there are multiple images that are sub-screens, such as when there are a plurality of cameras different from the main camera, a space for displaying the sub-screen It is necessary to narrow down and select an image as a child screen from a plurality of images due to the problem of visibility of the wipe display and the like.

  However, if an image having a high degree of similarity to the image captured by the main camera is output as a child screen, there may be no difference between the parent screen and the child screen. For this reason, images that are meaningless to be viewed at the same time are displayed as wipes, and the meaning of image display or image recording by photographing with a plurality of cameras may be lost.

  Therefore, the degree of difference indicating the degree of difference between the image captured by the main camera and the image captured by the main camera is obtained for each of the images captured by another camera in which the same subject is captured by the main camera. Based on the degree of difference obtained for each image of another camera, the image of the camera that is calculated to have the largest difference from the image of the main camera is selected as the sub-screen.

  Even if the same subject is shot on both the main screen and the sub screen, the sub screen differs from the main screen by selecting an image with a large image difference from the in-camera image as the sub screen. The image is captured from the viewpoint. By recording images from different viewpoints in combination, even if the same subject is captured, the effect of displaying a small screen by photographing with a plurality of cameras can be exhibited due to the difference in the viewpoints of the images.

It is a figure which shows the digital camera which shows the example of a structure of this invention. It is a flowchart which shows a present Example. It is a figure which shows the example which calculates | requires the difference degree of a present Example. It is a graph which shows the example which calculates | requires the difference degree of a present Example. It is a figure which shows the example which calculates | requires the difference degree of a present Example. It is a graph which shows the example which calculates | requires the difference degree of a present Example. It is a figure which shows the example which calculates | requires the difference degree of a present Example. It is a graph which shows the example which calculates | requires the difference degree of a present Example. It is a figure which shows the example which calculates | requires the difference degree of a present Example. It is a graph which shows the example which calculates | requires the difference degree of a present Example. It is a figure which shows the example which calculates | requires the difference degree of a present Example. It is a graph which shows the example which calculates | requires the difference degree of a present Example. It is a figure which shows the example of the image made into the subscreen by which the image image | photographed with the main camera was selected.

  The present invention will be described using a digital camera as an example. Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing a configuration of an embodiment of the present invention. In FIG. 1, reference numeral 100 denotes an image processing apparatus. The imaging apparatus 100 includes two imaging units, a main camera and a sub camera. Reference numeral 10-1 denotes a main camera 10-2, an imaging lens of a sub camera, 12-1 denotes a main camera 12-2, a mechanical shutter having a sub camera aperture function, and 14-1 denotes a main camera 14-2. Imaging elements 16-1 and 16-2 for converting an optical image into an electrical signal are A / D converters for converting an analog signal output corresponding to the imaging elements 14-1 and 14-2 into a digital signal.

  Reference numerals 18-1 and 18-2 denote timing generation circuits for supplying clock signals and control signals to the corresponding imaging devices 14-1 and 14-2 and corresponding A / D converters 16-1 and 16-2. Control is performed by the control circuit 22 and the system control circuit 50. In addition to the mechanical shutters 12-1 and 12-2, the storage time can be controlled as an electronic shutter by controlling the reset timing of the image sensor 18-1 or 18-2. Can be used.

  Reference numerals 20-1 and 20-2 denote image processing circuits, and pixel interpolation processing for image enlargement / reduction with respect to data from the A / D converter 16-1 or 16-2 or data from the memory control circuit 22 And color conversion processing, noise removal processing and edge enhancement processing. Further, the image processing circuits 20-1 and 20-2 also perform face detection processing for detecting a face area on the image data from the memory control circuit 22, generation of face feature data for face authentication, and the like.

  In addition, the image processing circuits 20-1 and 20-2 perform predetermined arithmetic processing using image data captured for TTL AWB (auto white balance) processing, and the obtained arithmetic results are displayed. Calculation is performed as a WB (white balance) evaluation value, or color conversion of image data is performed based on the calculated WB evaluation value. Further, in the image processing circuits 20-1 and 20-2, captured images are used to calculate AF evaluation values, AE evaluation values, and EF evaluation values for performing TTL AF processing, AE processing, and EF processing. Predetermined arithmetic processing is performed using the data.

  Based on the obtained AF evaluation value, AE evaluation value, and EF evaluation value, the system control circuit 50 applies to the exposure control means 40-1 and 40-2 and the distance measurement control means 42-1 and 42-2 according to the algorithm. Take control. A memory control circuit 22 controls the A / D converters 16-1 and 16-2, the timing generation circuits 18-1 and 18-2, the image processing circuit 20, the memory 30, and the compression / decompression circuit 32. The data of the A / D converter 16 is written into the memory 30 via the image processing circuit 20 and the memory control circuit 22, or the data of the A / D converter 16 is directly written via the memory control circuit 22.

  Reference numeral 28 denotes an image display unit composed of a TFT LCD or the like. Display image data written in the memory 30 is displayed by the image display unit 28 via the memory control circuit 22. The image display unit 28 can also display a plurality of images written in the memory 30 at the same time. If the image data captured using the image display unit 28 is sequentially displayed, the electronic viewfinder function can be realized.

  The image display unit 28 can arbitrarily turn on / off the display according to an instruction from the system control circuit 50. When the display is turned off, the power consumption of the image processing apparatus 100 can be greatly reduced. I can do it. Reference numeral 30 denotes a memory for storing captured still images and moving images, and has a sufficient storage capacity to store a predetermined number of still images and a predetermined time of moving images. This makes it possible to write a large amount of images to the memory 30 at a high speed even when handling a plurality of still images or shooting a moving image.

  The memory 30 can also be used as an area for temporarily storing feature data for authentication and a work area for the system control circuit 50. Reference numeral 31 denotes a non-volatile memory composed of a flash ROM or the like. The program code executed by the system control circuit 50 is written in the nonvolatile memory 31, and the program code is executed while being read sequentially. In addition, the non-volatile memory has an area for storing facial feature data for authentication as dictionary data, an area for storing system information, and an area for storing user setting information. It can be read and restored.

  Reference numeral 32 denotes a compression / decompression circuit that compresses and decompresses image data by adaptive discrete cosine transform (ADCT) or the like, reads an image stored in the memory 30, performs compression processing or decompression processing, and stores the processed data in the memory 30. Write to. The main camera 40-2 is an exposure control means for a sub camera, and controls the shutter 12-1 or 12-2 having an aperture function. 42-1 is a distance measuring control means for the main camera 42-2 and controls the focusing of the photographing lens 10-1 or 10-2, and 44-1 is a zoom for the sub camera. The zooming of the taking lens 10-1 or 10-2 is controlled by the control means.

  A flash 48 has an AF auxiliary light projecting function and a flash light control function. The exposure control means 40-1 and 40-2 and the distance measurement control means 42-1 and 42-2 are controlled using the TTL system, and the captured image data is calculated by the image processing circuit 20-1 or 20-2. Based on the calculated result, the system control circuit 50 controls the exposure control means 40 and the distance measurement control means 42-1 or 42-2.

  A system control circuit 50 controls the entire image processing apparatus 100. Reference numerals 60, 62, 64, 66, 70, and 72 denote operation means for inputting various operation instructions of the system control circuit 50. A single unit such as a switch, a dial, a touch panel, pointing by line-of-sight detection, a voice recognition device, or the like Consists of multiple combinations.

  Here, a specific description of these operating means will be given. Reference numeral 60 denotes a mode dial switch which can be set to switch between power-off, automatic shooting mode, shooting mode, panoramic shooting mode, movie shooting mode, simultaneous shooting mode of multiple cameras, playback mode, PC connection mode, and the like. . Reference numeral 62 denotes a shutter switch SW1, which is turned ON during the operation of the shutter button, and performs AF (autofocus) processing for the main camera or sub camera, or both the main camera and sub camera according to the order set by the mode dial switch 60. , AE (automatic exposure) processing, AWB (auto white balance) processing and the like start operation.

  A shutter switch SW2 64 is turned on when the operation of the shutter button is completed. In the case of flash photography, after performing EF (flash pre-emission) processing, at least one of the image sensors 14-1 and 14-2 is exposed for the exposure time determined by the AE processing. Light is emitted during the period, and at least one of the exposure control means 40-1 and 40-2 is shielded from light simultaneously with the end of the exposure period, thereby completing the exposure to at least one of the image sensor 14-1 or 14-2.

  Read processing for writing the signal read from the image sensor 14-1 or 14-2 to the memory 30 via the A / D converter 16-1 or 16-2 and the memory control circuit 22, and the image processing circuit 20-1 Alternatively, a series of development processing using calculation in 20-2 or the memory control circuit 22, reading processing of image data from the memory 30, compression by the compression / decompression circuit 32, and recording processing for writing image data to the recording medium 200. The operation start of the process is instructed. Further, an image is acquired via the communication device 110 described later in accordance with the mode set by the mode dial switch 60 by the activation of the 64 shutter switch SW2.

  Reference numeral 66 denotes a display changeover switch, which can change the display of the image display unit 28. With this function, when photographing is performed using the optical viewfinder 104, it is possible to save power by cutting off the current supply to the image display unit including a TFT LCD or the like. Reference numeral 70 denotes an operation unit including various buttons, a touch panel, a rotary dial, and the like, which include a menu button, a set button, a macro button, a multi-screen playback page break button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, and the like. . Menu move + (plus) button, menu move-(minus) button, playback image move + (plus) button, playback image-(minus) button, shooting quality selection button, exposure compensation button, date / time setting button, etc. is there.

  Reference numeral 72-1 or 72-2 denotes a zoom switch unit as a zoom operation means for a user to give a magnification change instruction for a captured image. Hereinafter, it is also referred to as zoom switch 72-1 or 72-2. The zoom switch 72-1 or 72-2 includes a tele switch that changes the imaging field angle to the telephoto side and a wide switch that changes the wide angle side. By using the zoom switch 72, the zoom control means 44-1 or 44-2 is controlled to give an instruction to change the imaging angle of view of the photographing lens 10-1 or 10-2 and serve as a trigger for performing an optical zoom operation. In addition, the image processing circuit 20-1 or 20-2 also serves as a trigger for image zooming or electronic zooming change of the imaging angle of view by pixel interpolation processing.

  Reference numeral 86 denotes a power supply means including a primary battery of an alkaline battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li ion battery, an AC adapter, or the like. Reference numeral 90 denotes an interface with a recording medium such as a memory card or hard disk, and reference numeral 92 denotes a connector for connecting to a recording medium such as a memory card or hard disk. Reference numeral 104 denotes an optical viewfinder, which can shoot using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28, and can simultaneously display the main screen and the sub-screen display. Is possible.

  Reference numeral 110 denotes a communication device using wireless communication, which can transmit and receive image data and metadata by communicating with an external camera, a smartphone, and the like. The received image data and metadata are stored in the system control 50. It can be temporarily stored in the memory 30 via the memory control 22 according to the instruction. Reference numeral 200 denotes a recording medium such as a memory card or a hard disk. The recording medium 200 includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, or the like, an interface 204 with the image processing apparatus 100, and a connector 206 for connecting to the image processing apparatus 100. Reference numeral 210 denotes a recording medium such as a memory card or a hard disk.

  An embodiment according to the present invention will be described with reference to the flowchart of FIG. At the start of the flow of FIG. 2, the image of the main camera is already stored in the memory 30, and the image of the main camera is recorded as the image of the main screen. The flow in FIG. 2 is a flow for selecting an image to be a small screen.

  In S <b> 201, an image of another camera other than the main camera is acquired and temporarily stored in the memory 30. The image other than the main camera is an image acquired from a sub camera of the imaging device 100 or an image received from an external camera or a smartphone via the communication device 110. The communication device 110 can receive images in synchronization with a plurality of external devices.

  In S202, it is determined whether or not the same person as the main camera is photographed and photographed with respect to the image obtained in S201. Whether or not the same person is photographed is generated by using the image processing circuit 20-1 or 20-2 to generate feature information for the main camera image temporarily stored in the memory 30 and each image acquired in S201. Judgment is performed by performing personal authentication. When a plurality of persons are captured in the main camera image and other images, it is determined whether or not the person who has become the main subject is captured by the main camera.

  Also, in the authentication method, metadata including feature data associated with image data is generated by an external camera, acquired as an image via the communication device 110 and used for authentication, or information representing a person identified by the external camera You may use the one with embedded metadata. If the same person as the main camera is photographed in S202, it is left as a candidate for the small screen, and if it is determined that the same person is not photographed, the image is removed from the small screen candidate and the process proceeds to S203. In S203, a comprehensive difference evaluation value Diff_All with the image of the name camera is calculated for each image that is a sub-screen candidate. An example of calculating the degree of difference will be described with reference to FIGS. 3 to 11 until the wrong evaluation value Diff_All is obtained.

  In order to obtain the overall difference evaluation value Diff_All, Diff01, Diff02, Diff03, Diff04, and Diff05 are obtained. First, in order to obtain Diff01 as a side face angle difference evaluation value, the lateral angle of the eyelid of the subject identified as the same person is calculated in S202, and the difference from the eyelid angle of the image of the main camera is calculated. Here, the main camera is the main camera of the imaging apparatus 100, the camera (a) is a sub camera of the imaging apparatus 100, and the camera (b) is an external camera that can acquire images via the communication apparatus 110.

  For example, when the face of the person in the main camera is a front face as shown in FIG. When the angle of the person photographed by the camera (a) is 45 degrees to the right, the difference in the recumbent angle of the camera (a) is 45 degrees. If the angle of the person photographed by the camera (b) is 90 degrees to the left, the difference in the recumbent angle is 90 degrees. When the difference of the recumbent angle is obtained, Diff01 is obtained as the profile evaluation value of the profile angle using FIG. 4 as an example. The horizontal axis of the graph of FIG. 4 is the difference angle of the recumbent angle, and the vertical axis is the difference evaluation value. In light of FIG. 4, the difference evaluation value of the camera (a) is 5, and the difference evaluation value of the camera (b) is 10.

  Next, how to obtain the difference evaluation value Diff02 depending on the size of the face will be described with reference to FIGS. FIG. 5 shows the size of the face of the same person photographed by each camera, and the dotted square indicates the size of the face of the person. Based on the dotted line, the number of pixels corresponding to the size of the eyelid taken by each camera is obtained. When the resolution of each camera is different, the resolution is calculated by converting it to the equivalent of the main camera.

  The absolute value of the difference in the number of pixels from the main camera is obtained from the size of the face photographed by each camera, and the difference evaluation value Diff02 according to the size of the face is obtained from the graph shown in FIG. In the case of FIG. 4, if the difference evaluation value Diff02 of the camera (a) is larger, or if the difference in face size between the main camera and the main camera exceeds 50 pixels, Diff02 is the camera (a). And camera (b) both become 10.

  Next, how to obtain the difference evaluation value Diff03 based on the face position will be described with reference to FIGS. The position of the same person of each camera is specified as shown in FIG. Using the center of the face as the center of gravity of the dotted frame, the difference between the face positions of the main camera, the camera (a), and the camera (b) is obtained. When the resolution and aspect are different between the main camera and another camera, coordinate conversion is performed on the resolution and aspect of the main camera to obtain a difference in face position.

  When the face position difference is obtained, the difference evaluation value Diff03 is obtained using FIG. The horizontal axis is the difference in face position when the diagonal of the angle of view of the main image is 100%, and the vertical axis is the difference evaluation value Diff03. In the case of FIG. 7, the difference evaluation value Diff03 is larger for the camera (b).

  A method for obtaining the difference evaluation value Diff04 for another subject will be described with reference to FIGS. First, the number of subjects that are not captured in the main camera is obtained for each camera. Only a person may be counted as a subject that is not captured in the main camera, or the number of objects and objects that are detected by object detection may be used. In the case of FIG. 9, the number of subjects that are not shown in the main camera is 2 in the camera (a), and the camera (b) is 0 because both of them are shown in the main camera. When the difference evaluation value Diff04 for another subject is obtained with reference to FIG. 10, Diff04 becomes higher for camera (a) than for camera (b).

  A method for obtaining the difference evaluation value Diff05 based on the luminance difference of the background will be described with reference to FIGS. First, the luminance average obtained by removing the face area of the same person subject as that of the main camera from the images taken by the respective cameras is obtained. An example taken with each camera is shown in FIG. The face area is set as an area indicated by a dotted line, and the brightness value photographed by each camera is obtained, and the difference from the brightness value of the main camera is obtained. Using FIG. 12, the difference evaluation value Diff05 based on the difference in luminance of the background is obtained from the difference from the luminance value of the main camera.

  In this embodiment, the difference evaluation value based on the background luminance difference is obtained, but the difference evaluation value based on the background color difference may be calculated. A total difference evaluation value Diff_All is obtained using the difference evaluation values Diff01 to Diff05 obtained so far. As an example of how to obtain Diff_All, it is assumed that Diff_All is multiplied by Diff01 to Diff05. When the difference evaluation value Diff_All is obtained for each image other than the main image, the process proceeds to S204. In S204, an image to be used as a child screen is selected based on the difference evaluation value Diff_All. The image that is the child screen is the image that has the highest difference evaluation value Diff_All.

  When the sub-screen is determined, the process proceeds to S205, and the image taken as the selected sub-screen is wiped for the image taken by the main camera, and the image taken as the selected sub-screen is simultaneously recorded. Or, as with the display, the image captured by the main camera is recorded in a superimposed state with the selected sub-screen image. FIG. 13 shows an example of an image taken as a sub-screen with an image selected by the main camera.

  The description in the present embodiment is merely an example, and not only the above-described multiplication but also adding weights to Diff01 to Diff05, etc., when obtaining the difference evaluation value Diff_All using Diff01 to Diff05. You can go. Further, in order to obtain the difference evaluation value Diff_All, it may be calculated and used using another difference criterion.

  Further, the combination of the parent screen and the child screen may be used for a still image or a moving image. In addition, when used for a moving image, a flow for determining each frame as a child screen may be used, or a frame for selecting a child screen may be thinned out. You may use. When the flow determined as the child screen is not continuous, images used for the child screen are images taken by the same camera until switching.

  Depending on the imaging timing and communication status of the main camera and the sub camera, an image used as a sub-screen may not be acquired in synchronization with the main camera. In that case, the flow to determine the child screen update timing as a child screen according to any timing of the sub camera, or the flow to determine the child screen as an image of the camera that has established communication The child screen may be updated by executing.

  If there is a large difference between the main camera and sub camera, update the sub-screen according to the difference, such as delaying the update timing of the sub camera image and controlling the timing to show the difference between the main camera and sub camera. It doesn't matter. If there is no sub-camera image that contains the same person as the main camera image in S202 of FIG. 2, whether or not a person registered in the authentication dictionary of the main camera appears in place of the same person. May be determined in S202, and the process may proceed to S203 to determine an image to be used as a child screen from the difference determination.

  In S204 of FIG. 2, the image as the child screen is the image having the highest difference evaluation value Diff_All. For example, when the difference evaluation value Diff_All is provided with a threshold value and does not satisfy the reference value, the child screen may be omitted. If the above condition is satisfied, there may be a plurality of sub-screens.

  The configuration of the imaging device 100 in this embodiment is a configuration of a main camera and a sub camera. However, the imaging device 100 includes only the main camera, and all images of the sub camera are externally transmitted using the communication device 110. You may acquire it from other cameras.

10-1: Main camera photographing lens 12-1: Main camera shutter 14-1: Main camera imaging device
16-1: Main camera A / D converter 18-1: Main camera timing generation circuit
20-1: Main camera image processing circuit
10-2: Sub camera photographing lens 12-2: Sub camera shutter 14-2: Sub camera imaging device 16-2: Sub camera A / D converter 18-2: Sub camera timing generation circuit
20-2: Sub-camera image processing circuit
22: Memory control circuit 28: Image display unit 30: Memory 32: Image compression / decompression circuit 40-1: Main camera exposure control means 42-1: Main camera distance measurement control means
44-1: Main camera zoom control means 48: Flash
40-2: Sub camera exposure control means 42-2: Sub camera distance measurement control means 44-2: Sub camera zoom control means 50: System control circuit 60: Mode dial switch 62: Shutter switch SW1 64: Shutter switch SW2
66: Display changeover switch 70: Operation unit 72: Zoom switch 90: Interface 92: Connector 100: Image processing apparatus 102: Protection means 104: Optical viewfinder 106: Gyro sensor 110: GPS
200: Recording medium 202: Recording unit 204: Interface 206: Connector

Claims (9)

  The difference between the image of the main camera and the image of the sub camera in the imaging device that has the function of displaying or recording the image captured by the main camera as the main screen and the images captured by multiple sub cameras other than the main camera as the sub screen A means for calculating the evaluation value of the degree, and selecting an image to be a sub-screen based on the evaluation value of the degree of difference from images taken by the sub camera and showing the same subject as the main camera An imaging device that is characterized.   2. The imaging apparatus according to claim 1, wherein a person is detected from the images of the main camera and the sub camera, and whether or not the same subject is photographed is determined according to whether or not the person is authenticated.   The evaluation value for the degree of difference is the difference between the size and orientation of the authenticated person's face or the screen position in the images of the main camera and the sub camera, and the number of detected human faces other than the authenticated person's face. The imaging apparatus according to claim 2, wherein the imaging device is generated from difference information of any one of the difference between the background and the brightness or color difference of the background other than the face of the person.   2. The imaging apparatus according to claim 1, wherein the sub-camera image having an evaluation value having a high degree of difference from the image of the main camera is selected as a sub-screen.   The imaging apparatus according to claim 1, wherein the sub-screen image is updated at predetermined time intervals, and the update timing of the sub-screen image is changed according to an evaluation value of a difference between the main camera and the sub camera. Features. The imaging apparatus according to claim 1, wherein when the evaluation value of the degree of difference between the main camera and the sub camera is less than a reference, no child screen is displayed.   If there is no sub camera image that contains the same subject as the main camera image, the sub camera image that contains the person registered in the dictionary data for authenticating the person with the main camera is given priority. The imaging apparatus according to claim 1, wherein the imaging apparatus is selected as an image to be a screen.   The imaging apparatus according to claim 1, wherein the image of the sub camera includes an image obtained by communication with an external camera.   The difference between the image of the main camera and the image of the sub camera in the imaging device that has the function of displaying or recording the image captured by the main camera as the main screen and the images captured by multiple sub cameras other than the main camera as the sub screen Has a means to calculate the evaluation value of the degree, it is determined whether or not to display the small screen based on the evaluation value of the degree of difference, and when displaying the small screen, the degree of difference is evaluated from the images taken by the sub camera A control method for an imaging apparatus, wherein an image to be a sub-screen is selected based on a value.