JP2015159405A - image processing apparatus, imaging device, control method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which allows for automatic selection of a sub-video suitable of the user from a plurality of sub-videos associated with a main video, and to provide an imaging device, a control method, and a program.SOLUTION: An imaging device includes a face detector 125, an evaluation value by expression calculation unit 126, and a system control unit 150. The face detector 125 detects the face of a person from the sub-videos recorded in a recording medium 125. The evaluation value by expression calculation unit 126 calculates an evaluation value representing to what expression the face of the person corresponds, based on the area of the person thus detected, for each expression. The system control unit 150 discriminates the expression of the sub-video based on the calculation result from the expression calculation unit 126, selects a predetermined number of sub-videos from a plurality of sub-videos recorded in a recording medium 104, and displays the main video recorded in a recording medium 104 and the sub-videos thus selected on a display 128.

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, a control method, a program, and a storage medium that automatically select and display a predetermined sub-video from a plurality of sub-videos recorded in association with a captured main video. .

  2. Description of the Related Art Conventionally, there is an imaging apparatus including two camera units, a first camera unit and a second camera unit (see, for example, Patent Document 1). In the imaging apparatus described in Patent Document 1, it is possible to simultaneously photograph using the first camera unit and the second camera unit in the photographing mode. According to this, an image of the subject can be obtained by photographing with the first camera unit, and an image of the photographer can be obtained by photographing with the second camera unit. Further, by synthesizing and displaying the respective images obtained by photographing with the first camera unit and the second camera unit, it is possible to display an image in which the subject and the photographer are integrated.

Japanese Patent Laid-Open No. 2005-94741

  In the imaging apparatus including the first camera unit and the second camera unit described in Patent Document 1, each time the main video captured by the first camera unit is reproduced and displayed, the viewer of the main video is displayed by the second camera unit. It is conceivable to shoot as a sub video. In this case, sub-videos of a plurality of viewers are recorded with respect to one main video by the imaging device, and a video in which a subject and a plurality of viewers who viewed the subject are associated with each other is obtained.

  However, Patent Document 1 does not mention a method of selecting and displaying a predetermined sub video from a plurality of sub videos. In addition, since the video playback area for displaying the captured video is limited in the imaging apparatus, it is necessary to select the sub video to be displayed in the video playback area when there are many sub videos recorded by the imaging apparatus. In this case, it is desirable to automatically select and display a sub-video suitable for the user by the control unit of the imaging apparatus.

  An object of the present invention is to provide an image processing apparatus, an imaging apparatus, a control method, a program, and a storage medium that can automatically select a sub video suitable for a user from a plurality of sub videos associated with a main video. There is.

  In order to achieve the above object, the present invention provides a recording means for recording a main video and a plurality of sub-videos associated with the main video, and a detection for detecting a human face from the sub-video recorded in the recording means. And an evaluation value calculation means for each expression that calculates an evaluation value for each expression based on the facial area of the person detected by the detection means, and an evaluation value representing what expression the person's face corresponds to, Based on the evaluation value calculated by the expression-specific evaluation value calculation means, the selection means for selecting a sub video from the plurality of sub videos recorded in the recording means, and the main video recorded in the recording means together with the main video And a control means for displaying the sub video selected by the selection means.

  According to the present invention, the evaluation value representing what expression the person's face corresponds to is calculated for each expression based on the area of the person's face detected from the sub video recorded in the recording means. Furthermore, based on the calculated evaluation value, the selected sub video is displayed together with the main video recorded in the recording means. Thereby, it is possible to automatically select a sub video suitable for the user from a plurality of sub videos associated with the main video.

1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. 4 is a flowchart illustrating processing in a playback mode of the imaging apparatus according to the first embodiment. It is a flowchart which shows the process at the time of the reproduction | regeneration mode of the imaging device which concerns on 2nd Embodiment. 14 is a flowchart illustrating processing in a playback mode of the imaging apparatus according to the third embodiment. It is a figure which shows the viewer's face with respect to a main image | video, the evaluation value classified by expression, and a facial expression discrimination | determination result. It is a figure which shows the example of a display of the some sub video linked | related with the main video and the main video. It is a figure explaining the process which selects a viewer from the sub image | video of the imaging device which concerns on 4th Embodiment. It is a figure explaining the process which selects a viewer from the sub image | video of the imaging device which concerns on 5th Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing the configuration of the imaging apparatus according to the first embodiment of the present invention. In FIG. 1, an imaging apparatus 100 is an example for realizing the image processing apparatus of the present invention, and is configured as a digital camera including two imaging units, a main imaging unit 190 and a sub imaging unit 191.

  The main imaging unit 190 and the sub imaging unit 191 include a shutter 101, a barrier 102, a photographing lens 103, an imaging unit 122, and an A / D converter 123, respectively. The main imaging unit 190 and the sub imaging unit 191 have structures that can be directed in different directions. The main imaging unit 190 is used for shooting a subject (main video). The sub imaging unit 191 is used for shooting a viewer (sub video) browsing the subject (main video) captured by the main imaging unit 190.

  The photographing lens 103 is a lens group including a zoom lens and a focus lens. The shutter 101 has a diaphragm function. The imaging unit 122 includes a CCD or CMOS element that converts an optical image of a subject into an electrical signal. The A / D converter 123 converts the analog signal output from the imaging unit 122 into a digital signal. The barrier 102 covers the photographing lens 103 and the like, thereby preventing the imaging system including the photographing lens 103, the shutter 101, and the imaging unit 122 from being soiled or damaged.

  The image processing unit 124 performs various processes including a process of generating still image data from an image signal subjected to image quality adjustment processing. That is, resize processing such as predetermined pixel interpolation and reduction, and color conversion processing are performed on the data output from the A / D converter 123 or the data output from the memory control unit 115. In addition, the image processing unit 124 performs predetermined calculation processing using the captured image data. Based on the obtained calculation result, the system control unit 150 performs exposure control and distance measurement control. Thereby, AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing of the TTL (through-the-lens) method are performed.

  Further, the image processing unit 124 performs predetermined calculation processing using the captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained calculation result. Data output from the A / D converter 123 is written into the memory 132 via the image processing unit 124 and the memory control unit 115 or via the memory control unit 115.

  The face detection unit 125 performs predetermined face detection processing for detecting the face of a person photographed by the imaging apparatus 100 on the data output from the A / D converter 123 or the data output from the memory control unit 115. The face detection result is written into the memory 132 via the memory control unit 115. The face detection unit 125 functions as detection means of the present invention.

  The facial expression-specific evaluation value calculation unit 126 calculates an evaluation value for each facial expression of a person photographed by the imaging apparatus 100 for the data output from the A / D converter 123 or the data output from the memory control unit 115. An evaluation value calculation process is performed. The evaluation value for each facial expression is a numerical value representing what facial expression (smile, crying face, angry face) the person's face corresponds to, as shown in FIGS. 5 (a) and 5 (b) described later. It is. The expression-specific evaluation value calculation unit 126 functions as the expression-specific evaluation value calculation means of the present invention.

  The memory 132 is image data obtained by the imaging unit 122 and converted into digital data by the A / D converter 123, image data for display on the display unit 128, detection result of a human face area by the face detection unit 125, and the like. Is stored. The memory 132 has a storage capacity sufficient to store a predetermined number of still images and a predetermined time of moving images and audio. The memory 132 also serves as an image display memory (video memory). The D / A converter 113 converts the display image data stored in the memory 132 into an analog signal and supplies the analog signal to the display unit 128. Thereby, an image is displayed on the display unit 128.

  The display unit 128 performs display (still image, moving image) according to the analog signal from the D / A converter 113 on a display device such as an LCD. A digital signal (image data) once A / D converted by the A / D converter 123 and stored in the memory 132 is converted into an analog signal by the D / A converter 113 and sequentially transferred to the display unit 128 for display. Thereby, the display unit 128 functions as an electronic viewfinder, and can perform through image display.

  The nonvolatile memory 156 is an electrically erasable / recordable memory, and for example, a FROM or the like is used. The non-volatile memory 156 stores constants for operation of the system control unit 150, statistical data for each facial expression used to calculate an evaluation value for each facial expression photographed by the imaging apparatus 100, a program, and the like. The program referred to here is a program for executing processing shown in each flowchart described later. Further, the statistical data for each facial expression is data in which a tendency of the facial feature amount to change for each facial expression of a person such as a smile, an angry face, and a crying face is recorded in advance.

  The system control unit 150 controls the entire imaging apparatus, and implements processing of each embodiment described later by executing a program recorded in the nonvolatile memory 156. The system control unit 150 also performs display control by controlling the memory 132, the D / A converter 113, the display unit 128, and the like. The system control unit 150 executes processing shown in each flowchart described later based on a program stored in the nonvolatile memory 156.

  The system control unit 150 functions as a determination unit, a selection unit, a control unit, a facial expression ratio calculation unit, and a frame selection unit of the present invention. The system memory 152 uses a RAM and develops constants and variables for operation of the system control unit 150, a program read from the nonvolatile memory 156, and the like.

  The mode switch 160, the shutter button 161, the first shutter switch 162, the second shutter switch 163, and the operation unit 170 are operation means for inputting various operation instructions to the system control unit 150. The mode changeover switch 160 sets the operation mode of the system control unit 150 to a still image recording mode for recording a still image on the recording medium 104, a moving image recording mode for recording a moving image on the recording medium 104, and a still image / moving image on the display unit 128. Switch to one of the playback modes to display.

  The first shutter switch 162 is turned on when the shutter button 161 is half-pressed (shooting preparation instruction) during operation of the shutter button 161, and generates a first shutter switch signal SW1. The system control unit 150 starts operations such as AF processing, AE processing, AWB processing, and EF processing in response to the signal SW1. The second shutter switch 163 is turned on when the operation of the shutter button 161 is completed, that is, when it is fully pressed (shooting instruction), and generates a second shutter switch signal SW2. Based on the signal SW2, the system control unit 150 starts a series of photographing processing operations from reading a signal from the imaging unit 122 to writing image data in the recording medium 104.

  Each operation member of the operation unit 170 is appropriately assigned a function for each scene by selecting and operating various function icons displayed on the display unit 128, and functions as various function buttons. Examples of the function buttons include an end button, a return button, an image advance button, a jump button, a narrowing button, and an attribute change button. For example, when a menu button is pressed, various setting menu screens are displayed on the display unit 128. The user can make various settings intuitively using the menu screen displayed on the display unit 128 and the four-way button or the SET button.

  The power control unit 180 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches a block to be energized, and the like, and detects whether or not a battery is attached, the type of battery, and the remaining battery level. The power supply control unit 180 controls the DC-DC converter based on the detection result and an instruction from the system control unit 150, and supplies necessary voltages to each unit including the recording medium 104 for a necessary period. The power supply unit 130 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. The recording medium I / F 118 serves as an interface with the recording medium 104.

  The recording medium 104 records a video (such as a plurality of sub-videos associated with the main video and the main video) taken by the main imaging unit 190 and the sub-imaging unit 191, and is recorded from a semiconductor memory or a magnetic disk. It is configured. Here, the main video is a video obtained by shooting the subject with the main imaging unit 190. The sub video is a video obtained by photographing the viewer of the main video by the sub imaging unit 191 when the main video is reproduced and displayed on the display unit 128.

  The imaging apparatus shown in FIG. 1 is a configuration example when the present invention is implemented in a digital camera, and is not limited to the configuration shown in FIG. 1 as long as the operation described below can be executed. In other words, the present invention can be implemented in the same manner as the imaging apparatus even in a smartphone, a personal computer, or the like as long as it has a recording medium on which a main video and a plurality of sub-videos associated with the main video are recorded.

  Next, processing in the playback mode of the imaging apparatus according to the present embodiment will be described in detail with reference to the flowchart of FIG.

  In view of the object of the present invention (a suitable sub video can be automatically selected from a plurality of sub videos associated with the main video), the main video is shot by the imaging apparatus of FIG. A case where reproduction and sub-video shooting are executed a plurality of times will be described as an example. Accordingly, it is assumed that a main video and a plurality of sub-videos associated with the main video are recorded in the recording medium 104 (recording unit) in advance.

  FIG. 2 is a flowchart showing processing in the playback mode of the imaging apparatus according to the present embodiment. In FIG. 2, when starting the playback mode, the system control unit 150 selects one main video from a plurality of main videos recorded on the recording medium 104 based on a user input from the operation unit 170 (step S201). Next, the system control unit 150 displays the selected main video on the display unit 128 (step S202). Next, the system control unit 150 performs face detection processing for detecting a human face in the sub video by the face detection unit 125 for one sub video among the plurality of sub videos recorded in the recording medium 104 ( Step S203).

  Next, the system control unit 150 performs facial expression evaluation value calculation for the facial expression evaluation value calculation unit 126 to calculate an evaluation value for each facial expression with respect to one of the facial expressions of the sub-video person subjected to the face detection processing ( Step S204). As an evaluation value calculation method for each facial expression, for example, as disclosed in JP-A-2005-31566, feature points of a person's face in image data are detected, and the smile level of the person is detected from the detected feature points. A technique for estimating

  The expression-specific evaluation value calculation of this embodiment is performed based on the detection result of the person's face area written in the memory 132 and the expression-specific statistical data stored in the nonvolatile memory 156 in advance. Here, the expression-specific statistical data is data representing how facial feature points change for each facial expression of a person, and is calculated by a statistical method.

  Next, the system control unit 150 determines whether or not the evaluation value for each facial expression has been calculated for all facial expressions for the one sub-video (step S205). If the facial expression evaluation values are not calculated for all facial expressions, the system control unit 150 returns to step S204 and repeats the processing. When the evaluation value for each facial expression is calculated for all facial expressions, the system control unit 150 performs the facial expression with the maximum evaluation value for each facial expression among all the evaluation values for each facial expression calculated for each facial expression. Is determined as the facial expression of the sub-picture (step S206).

  Next, the system control unit 150 determines whether or not the facial expression-based evaluation value calculation and facial expression determination have been performed on all sub-videos (step S207). If the evaluation value by facial expression and the facial expression determination are not performed for all the sub-videos, the system control unit 150 returns to step S203 and repeats a series of processes. When facial expression-based evaluation values are calculated and facial expressions are determined for all sub-videos, the system control unit 150 has high facial expression-based evaluation values from a plurality of sub-videos whose facial expression-specific evaluation values are higher than a predetermined threshold. A predetermined number of sub-videos are preferentially selected in order (step S208).

  Next, the system control unit 150 causes the display unit 128 to display the sub-images selected in descending order of the evaluation value by facial expression (step S209). Next, the system control unit 150 captures an image of the person who is operating the image capturing device with the sub image capturing unit 191 and records the image as a sub image on the recording medium 104 (step S210), and ends the present process.

  As described above, by performing the procedure shown in FIG. 2 under the control of the system control unit 150 of the imaging apparatus, a person with a high facial expression evaluation value (viewer of the main video) is preferentially displayed on the display unit 128. Can do.

  FIG. 5A illustrates an example of processing performed based on the above-described procedure, and is a diagram illustrating a viewer's face, facial expression evaluation value, and facial expression discrimination result for the main video. In FIG. 5 (a), with respect to six persons from person A to person F who are viewers of the main video, the evaluation value for each expression calculated for each expression of smile, crying face, and angry face, three smiling faces, two crying faces 1 shows a facial expression discrimination result determined by one person as an angry face. In this case, assuming that there are three sub-images displayed on the display unit 128, a person A, a person C, and a person D with high facial expression evaluation values are selected and displayed.

  FIG. 6 is a diagram illustrating a display example of a main video and a plurality of sub-videos associated with the main video. In FIG. 6, a video display area 600 of the display unit 128 is an area for displaying a main video and several sub-videos associated with the main video. The video display area 600 includes a main video display area 601, a first sub video display area 602, a second sub video display area 603, and a third sub video display area 604. However, more than three sub-videos associated with the main video are recorded on the recording medium 104, and the three displayed sub-videos are automatically selected from the recording medium 104.

  As described above, according to this embodiment, the evaluation value for each facial expression is calculated from a plurality of sub-videos, the facial expression of the sub-video is determined based on the evaluation value for each facial expression, and the corresponding sub-video is selected based on the facial expression of the sub-video. And display. Thereby, it is possible to automatically select a sub video suitable for the user from a plurality of sub videos associated with the main video.

[Second Embodiment]
The second embodiment of the present invention differs from the first embodiment in the points described below. Other elements of the present embodiment are the same as the corresponding ones of the first embodiment (FIG. 1), and thus description thereof is omitted.

  Next, processing in the playback mode of the imaging apparatus according to the present embodiment will be described in detail with reference to the flowchart of FIG.

  FIG. 3 is a flowchart showing processing in the reproduction mode of the imaging apparatus according to the present embodiment. In FIG. 3, the processing from step S301 to step S307 is the same as the processing from step S201 to step S207 in FIG. After the processes in steps S301 to S307, the system control unit 150 calculates the expression-specific ratios for all the sub videos based on the facial expression discrimination results for all the sub videos (step S308). Here, the expression-specific ratio is the ratio of the sub-video determined to be a specific facial expression to all the sub-videos.

  For example, if there are six sub-videos recorded on the recording medium 104 corresponding to a certain main image, and the facial expression discrimination results are three smiles, two crying faces, and one angry face, the ratio by facial expression is as follows: It is calculated as follows. That is, the expression-specific ratios are 50% smile (= 3/6), 33% crying face (≈2 / 6), and 16% angry face (≈1 / 6).

  Next, the system control unit 150 preferentially selects a predetermined number of sub-images to be displayed on the display unit 128 from the facial expressions having the highest expression-specific ratio (step S309). In this case, sub-pictures are selected in descending order of evaluation value for each facial expression within the same facial expression. Since the processes in steps S310 to S311 below are the same as the processes in steps S209 to S210 in FIG.

  As described above, by executing the procedure shown in FIG. 3 under the control of the system control unit 150 of the imaging apparatus, a person (viewer) who has a facial expression related to the main video can be displayed on the display unit 128.

  FIG. 5B illustrates an example of processing performed based on the above-described procedure, and is a diagram illustrating an evaluation value for each face, facial expression, and facial expression discrimination result for the main video. In FIG. 5 (b), the evaluation value according to expression calculated for each expression of smiles, crying faces, and angry faces for six persons A to F who are viewers of the main video, five persons are smiling, one person is angry An expression discrimination result determined as a face is shown. In this case, assuming that there are three sub-images to be displayed on the display unit 128, the sub-images of the person A, person B, and person C with the highest evaluation value by expression are selected and displayed from the five smiling faces with the highest ratio by expression. The

  As described above, according to the present embodiment, it is possible to automatically select a sub video suitable for the user from a plurality of sub videos associated with the main video.

[Third Embodiment]
The third embodiment of the present invention differs from the first embodiment in the points described below. Other elements of the present embodiment are the same as the corresponding ones of the first embodiment (FIG. 1), and thus description thereof is omitted.

  Next, processing in the playback mode of the imaging apparatus according to the present embodiment will be described in detail with reference to the flowchart of FIG.

  FIG. 4 is a flowchart illustrating processing in the playback mode of the imaging apparatus according to the present embodiment. In FIG. 4, the processing from step S401 to step S408 is the same as the processing from step S301 to step S308 in FIG. After the processing in steps S401 to S408, the system control unit 150 determines whether the facial expression ratio exceeds a predetermined threshold for facial expressions having the largest facial expression ratio (step S409). When the facial expression ratio exceeds a predetermined threshold, the system control unit 150 displays a predetermined number of sub-videos in descending order of the facial expression evaluation value from a plurality of sub-videos whose facial expression evaluation values are higher than the threshold. Select preferentially (step S410).

  When the facial expression ratio does not exceed a predetermined threshold, the system control unit 150 preferentially selects a predetermined number of sub-images to be displayed from a plurality of facial expressions whose facial expression evaluation value is higher than the threshold. Step S411). In this case, sub-images are selected in descending order of the evaluation value for each facial expression within a plurality of facial expressions. Since the processes in steps S412 to S413 below are the same as the processes in steps S310 to S311 in FIG.

  In step S410, it is considered that the viewer's facial expression for the main video is uniquely determined from the determination result in step S409. In step S411, it is considered that the viewer's facial expression for the main video is not uniquely determined. Therefore, the sub video selection method is switched by the above-described method. That is, switching between preferentially selecting a sub-video with a high expression-specific evaluation value or preferentially selecting a sub-video with a high expression-specific ratio from a plurality of sub-videos associated with the main video. Thereby, a more preferable viewer can be selected and displayed according to the main video.

  As described above, according to the present embodiment, it is possible to automatically select a sub video suitable for the user from a plurality of sub videos associated with the main video.

[Fourth Embodiment]
The fourth embodiment of the present invention is different from the first embodiment in the points described below. Other elements of the present embodiment are the same as the corresponding ones of the first embodiment (FIG. 1), and thus description thereof is omitted.

  In addition to the process of generating still image data from the image signal subjected to the image quality adjustment process described in the first embodiment, the image processing unit 124 of the imaging apparatus according to the present embodiment further includes a plurality of frames subjected to the image quality adjustment process. A process of generating moving image data from the image signal is performed. Here, the image processing unit 124 may generate moving image data that is compression-coded by intra-frame encoding each frame of the moving image data. Moreover, you may produce | generate the moving image data compression-coded using the difference between several flame | frames of moving image data, a motion estimation, etc. For example, it is possible to generate moving image data of various known compression encoding methods such as Motion JPEG, MPEG, and H.264 (MPEG4-Part10 AVC).

  In general, frame image data subjected to intra-frame coding is called an I picture. In addition, image data inter-frame encoded using a difference from the previous frame is called a P picture. In addition, image data that is inter-frame encoded using a difference between the front and rear frames is called a B picture. Note that these compression methods use known compression methods and are not related to the characteristics of the present invention, and thus the description thereof is omitted.

  The system control unit 150 synthesizes the moving image data and audio data (not shown) to form a data stream, and writes the data stream to the recording medium 104 as one moving image file. On the other hand, in the playback mode, the system control unit 150 reads out to the memory 132 a still image file composed of a compressed image signal recorded on the recording medium 104 or a moving image file composed of a compressed image signal and a compressed audio signal.

  The system control unit 150 sends the read compressed image signal and compressed audio signal to the image processing unit 124 and an audio processing unit (not shown). The image processing unit 124 decodes the compressed image signal from the memory 132 in which the compressed image signal is temporarily stored by a known predetermined procedure. Then, the memory control unit 115 transmits the decoded image signal to the face detection unit 125 and the facial expression evaluation value calculation unit 126. Thereby, the evaluation value calculation process is executed for each facial expression.

  Next, processing in the playback mode of the imaging apparatus according to the present embodiment will be described in detail with reference to FIG.

  In the first to third embodiments, the case where the sub video is still image data has been described. On the other hand, in the present embodiment, a case where the main video and the sub video are moving image data will be described. That is, at least one frame (specifically, a plurality of frames at predetermined time intervals) is selected from the moving image data of the sub video recorded on the recording medium 104, the evaluation value for each facial expression is calculated, and the facial expression is discriminated. A process for selecting a viewer will be described.

  FIG. 7A and FIG. 7B are diagrams illustrating processing for selecting a viewer from the sub video of the imaging apparatus according to the present embodiment. FIG. 7A shows moving image data of a viewer (person A) corresponding to the main video. FIG. 7B shows moving image data of a viewer (person B) corresponding to the main video. 7A and 7B, the horizontal axis represents the elapsed time of the moving image data, and the vertical axis represents the evaluation value for each expression at a certain time. t1 to t5 indicate a certain time when the facial expression evaluation value is acquired. For example, the facial expression evaluation value of the moving image data is acquired every minute. This time interval may be fixed or may be changed depending on the recording time of the moving image.

  The expression-specific evaluation value for a smile obtained for an image signal obtained by decoding one frame of the moving image data of the person A at time t1 (= 0) is e1a. Further, the evaluation value classified by facial expression for the smile obtained for the person B at time t1 (= 0) is e1b. In this case, the evaluation value for each facial expression is acquired for each of the other facial expressions. Acquisition of the evaluation value for each facial expression at time t1 (= 0) is executed for all of the records in which a plurality of sub-videos for the main video are recorded in advance.

  For these facial expression evaluation values, a sub-video to be displayed on the display unit 128 is selected using the method described in the first to third embodiments. Then, the plurality of selected sub-videos are displayed on the display unit 128 as shown in FIG. 6 as a display example. In this case, as an image to be displayed on the display unit 128, a viewer's moving image corresponding to the main video may be displayed, or a still image obtained by obtaining an expression-specific evaluation value may be displayed.

  The selection result of the sub video to be displayed on the display unit 128 is maintained until the next time t2. That is, the facial expression evaluation value is valid until the next time t2. At time t2, as in the processing at time t1, for each of a plurality of sub-videos with respect to the main video, a sub-video to be acquired and displayed is selected and displayed, and the selected sub-video is displayed on the display unit 128. . The same processing is repeated after time t3.

  By switching the sub video selection method by the above-described method, a more preferable viewer can be selected and displayed according to the main video even when the main video and the sub video are moving image data.

  As described above, according to the present embodiment, it is possible to automatically select a sub video suitable for the user from a plurality of sub videos associated with the main video.

[Fifth Embodiment]
The fifth embodiment of the present invention differs from the first embodiment in the points described below. Other elements of the present embodiment are the same as the corresponding ones of the first embodiment (FIG. 1), and thus description thereof is omitted.

  Next, processing in the playback mode of the imaging apparatus according to the present embodiment will be described in detail with reference to FIG. In the present embodiment, since the process of selecting a viewer is different from that in the fourth embodiment, the contents thereof will be described.

  FIG. 8A and FIG. 8B are diagrams illustrating processing for selecting a viewer from the sub video of the imaging apparatus according to the present embodiment. FIG. 8A shows moving image data of a viewer (person A) corresponding to the main video. FIG. 8B shows moving image data of a viewer (person B) corresponding to the main video. 8 (a) and 8 (b), t1 to t5 indicate certain times at which facial expression evaluation values are acquired, and facial expression evaluation values at all preset times are acquired in advance. And all the acquired evaluation values classified by expression are compared, and the maximum value is calculated.

  In FIG. 8A, the maximum value of a plurality of expression-specific evaluation values for a smile acquired for an image signal obtained by decoding one frame of the moving image data of the person A is eamax (time t2). Further, in FIG. 8B, the maximum value of the plurality of facial expression evaluation values for a smile acquired for an image signal obtained by decoding one frame of the moving image data of the person B is ebmax (time t4).

  For the maximum evaluation value for each sub-image, the sub-image to be displayed on the display unit 128 is selected using the method shown in the first to third embodiments. In other words, the sub-image having the maximum evaluation value by facial expression is selected from the plurality of evaluation values by facial expression calculated by the expression-specific evaluation value calculation unit 126 from the plurality of sub-videos recorded on the recording medium 104.

  For example, if the viewer for the main video has the highest evaluation value for each expression of “smiling”, “crying face”, and “angry face”, the “smile” is the maximum value, and “smile” is selected as the sub video To do. Specifically, in the case of the person C in FIG. 5A, the evaluation value by expression is “smile” 100, “crying face” 25, and “angry face” 0, so the evaluation value by expression becomes the maximum value. “Smile” is the facial expression discrimination result (selected as a sub video). Since the subsequent steps are the same as those in the above embodiments, description thereof is omitted.

  As described above, according to the present embodiment, it is possible to automatically select a sub video suitable for the user from a plurality of sub videos associated with the main video.

[Sixth Embodiment]
The sixth embodiment of the present invention is different from the first embodiment in the points described below. Other elements of the present embodiment are the same as the corresponding ones of the first embodiment (FIG. 1), and thus description thereof is omitted.

  Next, processing in the playback mode of the imaging apparatus according to the present embodiment will be described in detail. In the present embodiment, since the process of selecting a viewer is different from those in the fourth and fifth embodiments, the contents thereof will be described.

  In the fourth and fifth embodiments, since it is necessary to acquire facial expression evaluation values for a plurality of sub-videos at each set time, the processing is complicated, and the actual processing load such as decoding of moving image data is increased. large. In the present embodiment, contents to select and display a more preferable viewer according to the main video while reducing the processing load will be described. Specifically, an example in which one frame is selected for each of a plurality of sub-videos at a time corresponding to one frame of the main video recorded on the recording medium 104 will be described.

  As a technology related to video content, a technology has been proposed in which a specific image included in the video content is detected, a highlight scene (high importance scene) is extracted, and highlight scene information is added to the video content ( JP, 2006-014085, A).

  In the present embodiment, the time information of the highlight scene is acquired from the highlight scene information of the main video data using a technique as described in the above publication. Let this time be tm. One frame at time tm corresponding to time tm of the highlight scene in a plurality of sub-videos associated with the main video is decoded, and an evaluation value for each expression is acquired.

  For the facial expression evaluation values related to the acquired plurality of sub-videos, the sub-video to be displayed on the display unit 128 is selected using the method described in the first to third embodiments. Since the subsequent steps are the same as those in the above embodiments, description thereof is omitted. By selecting the sub video by the above-described method, even when the main video and the sub video are moving image data, it is possible to select and display a more preferable viewer according to the main video while reducing the processing load. Can do.

  As described above, according to the present embodiment, it is possible to automatically select a sub video suitable for the user from a plurality of sub videos associated with the main video.

[Other Embodiments]
In the first to sixth embodiments, preferred embodiments of the present invention have been described. However, the present invention is not limited to these first to sixth embodiments, and various modifications and changes can be made within the scope of the gist. Is possible.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed. The program of the present invention has computer-readable program code for causing a computer to execute the control method of the image processing apparatus of the present invention, and is stored in a storage medium.

DESCRIPTION OF SYMBOLS 100 Imaging device 104 Recording medium 125 Face detection part 126 Evaluation value calculation part 128 classified by expression Display part 150 System control part

Claims (15)

Recording means for recording a main video and a plurality of sub-videos associated with the main video;
Detecting means for detecting a human face from the sub-video recorded in the recording means;
Based on the facial area of the person detected by the detection means, an evaluation value calculation means for each expression that calculates an evaluation value for each facial expression representing what expression the person's face corresponds to;
Based on the evaluation value calculated by the expression-specific evaluation value calculation means, a selection means for selecting a sub video from the plurality of sub videos recorded in the recording means;
Control means for displaying the sub video selected by the selection means together with the main video recorded in the recording means;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the selection unit selects a predetermined number of sub-images. Based on the result calculated by the facial expression-based evaluation value calculation means, further comprising a determination means for determining the expression of the sub-image,
The selection means selects a sub video from the plurality of sub videos recorded in the recording means based on the evaluation value calculated by the expression-specific evaluation value calculation means and the facial expression determined by the determination means. The image processing apparatus according to claim 1, wherein:   The said selection means preferentially selects the sub video | video with high evaluation value according to the expression of the determined facial expression from the said some sub video | video recorded on the said recording means. Image processing device.   The image processing apparatus according to claim 3, wherein the selection unit preferentially selects an expression having the same facial expression determined by the determination unit and having a high evaluation value of the facial expression. Further comprising a facial expression ratio calculation means for calculating a facial expression ratio indicating a ratio of all the sub videos of the sub video determined to be a specific facial expression by the determination means;
The selection means preferentially selects a sub-video with a high expression-specific ratio of the identified facial expression from the plurality of sub-videos recorded in the recording means based on the result calculated by the expression-specific ratio calculation means. The image processing apparatus according to claim 1, wherein the image processing apparatus is selected.   The selection means preferentially selects a sub-video with a high evaluation value for each facial expression from the plurality of sub-videos recorded in the recording means based on the result calculated by the expression-specific ratio calculation means, The image processing apparatus according to claim 6, wherein the sub-video having a high expression-specific ratio is preferentially selected. Frame selection means for selecting at least one frame from the sub-video recorded in the recording means;
The image processing apparatus according to claim 1, wherein the detection unit detects a human face from the frame selected by the frame selection unit.   The image processing apparatus according to claim 8, wherein the frame selection unit selects a plurality of frames at predetermined time intervals from the sub video recorded in the recording unit.   9. The image processing apparatus according to claim 8, wherein the frame selection unit selects one frame for each of a plurality of sub-videos at a time corresponding to one frame of the main video recorded in the recording unit. .   The main video is a video obtained by photographing a subject, and the sub video is a video obtained by photographing a viewer of the main video when the main video is displayed. The image processing apparatus according to any one of 1 to 10.   An imaging apparatus comprising the image processing apparatus according to claim 1. A control method for an image processing apparatus, comprising:
A detection step of detecting a person's face from the sub video recorded in the recording means for recording the main video and a plurality of sub videos associated with the main video;
Based on the facial area of the person detected by the detection step, an evaluation value for each facial expression that calculates an evaluation value for each facial expression representing what facial expression the person's face corresponds to;
Based on the evaluation value calculated by the expression-specific evaluation value calculation step, a selection step of selecting a sub image from the plurality of sub images recorded in the recording unit;
A control step of displaying the sub video selected by the selection step together with the main video recorded in the recording means;
An image processing apparatus control method comprising: A computer-readable program having program code for causing a computer to execute the control method of the image processing apparatus according to claim 13, wherein the control method includes:
A detection step of detecting a person's face from the sub video recorded in the recording means for recording the main video and a plurality of sub videos associated with the main video;
Based on the facial area of the person detected by the detection step, an evaluation value for each facial expression that calculates an evaluation value for each facial expression representing what facial expression the person's face corresponds to;
Based on the evaluation value calculated by the expression-specific evaluation value calculation step, a selection step of selecting a sub image from the plurality of sub images recorded in the recording unit;
A control step of displaying the sub video selected by the selection step together with the main video recorded in the recording means;
A program characterized by having.   A computer-readable storage medium storing the program according to claim 14.

Images