JP2013191963A - Image processing device and control method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device that generates a video on the basis of a time suitable for recognition.SOLUTION: A plurality of materials that are a moving image or a still image are selected, recognition times for the respective selected materials are set, and the selected materials are connected based on the set recognition times to generate a video.

Description

  The present invention relates to an image processing apparatus for editing an image including a moving image and a still image, and a control method thereof.

In recent years, in addition to the popularization of home digital video cameras, the number of digital devices for shooting moving images has increased, such as the addition of moving image shooting functions to digital SLR cameras.
Conventionally, when collecting and editing material videos such as captured videos and still images, the resulting videos are monotonous in videos created by people with little video production experience. This is because it does not follow the video production and editing rules called video techniques or video grammar used by professionals, but it is cumbersome for non-professionals to use such video techniques and video grammars. This is one of the factors that raise the threshold.
The video technique is a rule of camera work for a unit of material called a cut, how to connect the materials, and the length of the material, so that professional video producers can deliberately or unconsciously I make videos using these techniques. Non-Patent Document 1 analyzes professional video techniques and automates shooting and editing based on the analysis results. Yes. In addition, the time length of the material is automatically set using the size of the ratio of the subject to the screen called the shot size.
In Patent Document 1, the reproduction time of the material is set according to the number of subjects included in the material and the position of the subject. Patent Document 2 discloses that the size of a shot size is obtained from an image.

JP 2006-245646 A JP 2007-13478 A

Kobe University Doctoral Dissertation 2008-01, Research on Shooting / Editing Support Technology Based on Video Grammar and Automatic Shooting / Generation Technology for Video Content

However, even if the time length of the material is set based on the shot size as in Non-Patent Document 1, or the time length is set according to the number and position of subjects as in Patent Document 1, the video generated by editing The recognition time of the subject in the video is not constant due to the appearance interval of the subject. For this reason, when the time length specified by the shot size is shorter than the recognition time, the recognition is insufficient and the video is uncomfortable.
Therefore, an object of the present invention is to perform image editing based on a time suitable for recognition.

  In order to achieve the above object, an image processing apparatus according to an aspect of the present invention has the following arrangement. That is, a selection means for selecting a plurality of images, a setting means for setting a recognition time for recognizing what each of the selected images is captured, and the selection based on the recognition time by the setting means. And generating means for generating a video by connecting the images.

  According to the present invention, since the reproduction time is set from a time suitable for recognizing an image, it is possible to easily perform image editing.

The figure which shows the outline of the hardware of an image processing apparatus Diagram showing processing block of video editing application Flowchart of the entire process for setting the recognition time Flow chart of processing for determining subject size Flow chart of processing to set recognition time Figure showing an example of the user interface of an automatic editing application Figure showing an example of the editing result of the automatic editing application Process flowchart for setting recognition time Process flowchart for setting recognition time

[Embodiment 1]
Hereinafter, the present invention will be specifically described with reference to the drawings.

  FIG. 1 is a diagram showing an outline of hardware of an image processing apparatus that implements the present embodiment.

  The image processing apparatus 100 is a PC (personal computer) that performs image processing, executes various programs, and connects external devices.

  The CPU 101 is involved in all the processes of each configuration, reads the instructions stored in the ROM 102 and the RAM 103 in order, interprets them, and executes the processes according to the results. The CPU 101 also performs each process of a flow to be described later. The ROM 102 and RAM 103 provide the CPU 101 with programs, data, work areas, and the like necessary for the processing.

  The recording device 104 stores large-capacity data such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive), and is used to store editing materials and editing results. .

  An input / output interface (I / F) 105 is an interface necessary for connection to an external device, and is connected to the display 106, the keyboard 107, the mouse 108, the network I / F 109, and the memory card I / F 110.

  The display 106 is an output device for displaying a processing screen. A keyboard 107 and a mouse 108 are input devices for reflecting user operations. The network I / F 109 is an interface for connecting to other devices and exchanging data through the network. The memory card I / F 110 is an interface for reading / writing data from / to an external recording device.

  In addition, although the component of an apparatus exists other than the above, since it is not the main point of this invention, description is abbreviate | omitted.

  In the present embodiment, an automatic editing application that runs on a PC will be described as an example of the image processing apparatus.

  The processing block diagram of FIG. 2 is a diagram showing main processing blocks of the video editing application. The processing blocks will be described in order with reference to FIG. The material group F201 is a set of captured moving images, still images, and music materials. The used material selection unit F202 is a processing block that selects a plurality of moving images or still images used from the material group F201. Since the selection condition and the selection method are not the main point of the present invention, the description is omitted.

  The selected material placement unit F203 is a processing block that places a plurality of materials selected in F202 on the video editing time axis. Since the arrangement on the time axis is not the main point of the present invention, the description thereof is omitted. The arrangement on the time axis here refers to arranging the edit video from the start to the end in order. The recognition time calculation unit F204 is a processing block that sets the length of recognition time of each material using the order information of each material arranged by the selected material arrangement unit F203. Here, the recognition time is used to mean a time during which it can be recognized what kind of material is reflected in the material. The video processing unit F205 is a processing block that performs video processing called an effect between materials. The output unit F206 outputs the material edited in the video processing unit F205 in a specific video format.

  In the present embodiment, a processing block for setting a material recognition time in the recognition time calculation unit F204 will be described below.

  FIG. 3 is a diagram showing an overall flowchart of processing for setting a material recognition time in the present embodiment.

  In step S301, order information arranged on the time axis of the material used in the generated video is acquired.

  In step S302, an area including the target subject is specified from the material group, the subject is recognized from the image data of the specified target subject, and information for recognizing (discriminating) the target subject is output. The target subject area and the target subject designation method are assumed to be performed while the user looks at the screen. However, the result of analysis by another device may be used, and information that recognizes the target subject may be acquired. . The processing after the next step is performed on each material in the order of arrangement time.

  In step S303, subject recognition processing for discriminating the subject is performed, and the subject of interest included in the material is recognized using the subject of interest information acquired in step S302. Record the recognition result in the history database. The details of the algorithm for the person recognition processing are not the main point of the present invention, but a known method is used, and thus detailed description thereof is omitted. Further, when the recognition information is acquired before this step, the acquired recognition information may be used without performing the recognition process in this step.

  In step S304, subject size detection processing is performed on the selected material. Here, face detection processing is used to detect the subject area. As in the subject recognition process in step S303, a known technique is used as the algorithm technique for the face detection process. In calculating the subject size, face detection processing is performed, and the size and position of the face area included in the material are acquired. Details of the processing for calculating the subject size from the face area will be described later using the processing flow of FIG. If subject size information and face area information have been acquired before this processing step, the acquired information may be used without performing face detection processing in this processing block.

  Step S305 is a process of setting the material recognition time using the noticed subject information, the calculated subject size information, and subject recognition information.

  Details of the processing will be described later with reference to the flowchart of FIG.

  In step S306, it is determined whether or not the processing for all the materials has been completed. If the processing has not been completed, the process proceeds to step S307. If the processing for all the materials has been completed, the processing for adjusting the time length of the material is completed. It becomes.

  Here, when the material is a moving image, the subject information of a predetermined frame of the material may be used, or the subject information obtained from a plurality of frames may be averaged.

  This is the end of the description of the process for setting the material recognition time.

  Next, a process for classifying the size when face detection is used as the subject size information will be described using the flowchart of FIG. This is a detailed processing flow of step S304 in FIG.

  In step S401, information on the size and position of the face area is acquired. When the cutout or enlargement / reduction processing is performed on the material, the size and position after the processing are calculated. At this time, the coordinates of the center position of the face are calculated from the size of the face area. The size of the face here is calculated using information on area, height, width, and slant. Although the center coordinates of the face were used for the face position, it is not limited to the center, and it is only necessary to be able to determine where the face area is on the screen, so the coordinates for calculating the position are the center. It is not necessary.

  In step S402, the width and height of the material image are acquired. When changing the resolution of the material, the width and height after the resolution change are acquired. The resolution change includes a case where an area to be used is cut out from the screen, and the area to be cut out is used as the width and height of the image.

  In step S403, using the image width and height information acquired in step S402 and the center position coordinates of the face acquired in step S401, the coordinates of the face are determined. It is determined whether the face position coordinates are above the half of the height of the image. If the position coordinates are above, the process proceeds to step S404. If not, the process proceeds to step S408.

  In step S404, the ratio of the height of the face area to the height is calculated from the size of the face area acquired in step S401 and the height of the image acquired in step S402. If the ratio of the height of the face area is larger than the threshold, the process proceeds to step S405, and if smaller than the threshold, the process proceeds to step S406. The threshold is set to 1/8 because the average head of a Japanese adult is about 7, so if the subject is another race or child, change it to an appropriate threshold. .

  In step S405, the size of the face area occupying the screen is calculated from the size of the face area and the width and height of the material screen, and it is determined whether or not the face area is greater than or equal to the threshold value. If it is smaller than the threshold value, the process proceeds to step S407, and if it is greater, the process proceeds to step S408. Here, the threshold is assumed to be 2/3 of the face area height relative to the screen height or the face area width relative to the screen width.

  Step S406, step S407, and step S408 are processes for determining the divided sizes. S406 is determined to be a small size, S407 is a standard size shot, and S408 is determined to be a large size.

  This is the end of the description of the process of classifying the subject size based on the size information of the face area included in the material.

  Next, each processing in the processing flow for setting a recognition time suitable for each material from the target subject information, subject recognition information, and subject size information will be described using the flowchart of FIG.

  In step S501, target subject information is acquired. In step S502, the subject recognition information is used to compare the subject included in the material with the target subject information. In step S503, it is determined whether the subject included in the material includes the target subject as a result of the comparison in step S502. If it is determined in step S503 that the target subject is included, the process proceeds to step S504, and it is determined that the target subject is not included. If so, the process proceeds to step S409.

  In step S504, the subject size is acquired. Here, information obtained by classifying the subject size into a standard size, a large size, or a small size is acquired. In step S505, it is determined whether the acquired size is a standard size. If it is not the standard size, the process proceeds to step S506, and if it is a standard size, the process proceeds to step S509.

  In step S506, it is determined whether or not the size is large. If the size is large, the process proceeds to step S507. If the size is small, the process proceeds to step S508.

  In step S507, the recognition time is set shorter than the standard reproduction time. In step S508, the recognition time is set longer than the standard reproduction time. In step S509, the recognition time is set as the standard reproduction time. In this embodiment, the recognition time is set in three stages, but the time actually required for recognition may be set. In addition, although the material recognition time is the material reproduction time, the reproduction time different from the material recognition time may be associated with the material recognition time in advance.

  The playback time is the time for which the same still image is displayed in the case of a still image. In the case of a movie, the playback time may be played from the beginning of the selected movie. You may make it reproduce | regenerate for reproduction | regeneration time from the position judged from the content of this.

  This is the end of the description of the processing flow for setting the material recognition time from the noticed subject information, subject recognition information, and subject size.

  In the present embodiment, an example of an application for automatic editing has been described. As an example of the automatic editing application, as shown in FIG. 6, there are means (602, 603, 604) for selecting the material to be used, the theme of the video to be edited, and the conditions of the video to be generated, and means for starting creation (605).

  As an editing result, the material is arranged on the time axis called the timeline as shown in FIG. 7, and the time length of the material is set like the video material 706. Confirmation of reproduction is performed through the reproduction preview screen 702 by operating the reproduction control unit 704 or the instruction unit 705. Note that the user interface screens of FIGS. 6 and 7 are examples, and the arrangement and components are not limited because they are different from the main points of the present invention.

  In this embodiment, the example of the application that automatically edits has been described, but the present invention can also be applied to an application that manually edits. If FIG. 7 is a manual edit screen, the optimal material time length can be set automatically when the material is placed on the timeline.

  By performing the processing control described above, the length of the video material is automatically set by obtaining the appropriate time for recognizing the video material from the information of the subject of the video material in video editing. Easy video editing is possible.

[Embodiment 2]
In the first embodiment, the material recognition time is set based on the size of the subject size using the subject size of the target subject. In the second embodiment, the material recognition time is set using the appearance interval of the target subject. Adjust the length.

  The entire processing flow is the processing flow of FIG. 3 of the first embodiment, and the processing flow of FIG. 5 which is the material recognition time setting processing according to the size in step S305 is replaced with the processing flow of FIG.

  A process of setting the material recognition time using the appearance interval of the subject of interest will be described using the processing flow of FIG.

  In step S801, target subject information is acquired. In step S802, it is determined whether or not the subject of interest has appeared in the previous material in time using the subject recognition information. At this time, an appearance history database for each subject may be created and used for the presence or absence of appearance. If the subject of interest exists in the previous material, the process proceeds to step S803, and if not, the process proceeds to step S805.

  In step S803, the appearance interval with the material in which the subject of interest has previously appeared is calculated. In step S804, it is determined whether the calculated appearance interval is smaller than the threshold value. If it is smaller, the process proceeds to step S806, and if larger, the process proceeds to step S805. In step S805, the recognition time is lengthened. In step S806, the playback time is lengthened. In step S807, the appearance history is updated.

  Above, description of the processing flow which sets the recognition time of a raw material with the appearance interval of FIG. 8 is complete | finished.

  In the second embodiment, since the length of the video material is automatically set by obtaining a time suitable for recognizing the video material from the appearance interval of the subject of interest, video editing that is easy to recognize is possible. Become.

[Embodiment 3]
In the third embodiment, the number of other characters in the appearance interval of the subject of interest is used in the determination of the material recognition time. The overall processing flow uses the processing flow of FIG. 3 of the first embodiment, and replaces the processing flow of FIG. 5 which is the processing of step S305 with the processing flow of FIG.

  Next, the processing flow of FIG. 9 will be described.

  Steps S901 to S904 are the same as the processing from step S801 to step S804 in FIG. The determination process in step S904 is the same as that in step S804 in FIG. 8, but the transition destination when the appearance interval is small is changed to step S905.

  In step S905, the number of other characters appearing within the appearance interval determined in step S903 is determined, and it is determined whether the number of characters is less than a predetermined threshold using the subject recognition information. If it is less than the threshold value, the process proceeds to step S908, and if it is greater, the process proceeds to step S906.

  In step S906, the recognition time is lengthened. In step S907, the recognition time is shortened. In step S908, the appearance history of the subject of interest is updated. Above, description of Embodiment 3 is complete | finished.

  In the third embodiment, the processing for adjusting the time length of the material using the number of other subjects appearing during the appearance interval of the subject of interest has been described.

  As described above, the length of the video material is automatically set by obtaining a time suitable for recognizing the video material from the number of other subjects appearing during the appearance interval of the subject of interest. Video editing that is easy to recognize is possible.

  The component of the present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.). The present invention may be applied to a facsimile machine or the like. An object of the present invention is to supply a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and the system reads and executes the program codes stored in the storage medium. Is also achieved. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. As a storage medium for supplying the program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a DVD, a magnetic tape, a nonvolatile data storage unit, a ROM, or the like is used. I can do it. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on the instruction of the program code. Perform some of the actual processing. Furthermore, it is needless to say that the case where the functions of the above-described embodiments are realized by the processing. Further, the program code read from the storage medium is written to a data storage unit provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing, and the processing of the above-described embodiment is realized by the processing. Needless to say.

Claims (8)

A selection means for selecting a plurality of images;
For each of the selected images, setting means for setting a recognition time for recognizing what is captured,
An image processing apparatus comprising: generating means for connecting the selected images based on a recognition time by the setting means to generate a video. A designation means for designating a subject;
Obtaining means for obtaining subject information relating to the designated subject from each of the selected images;
The image processing apparatus according to claim 1, wherein the setting unit sets a recognition time of an image corresponding to the subject information from the acquired subject information.   The image processing apparatus according to claim 1, wherein the subject information is at least one of a face size, a face position, and a ratio of the face size to the image size.   The image processing apparatus according to claim 1, further comprising an arrangement unit that sequentially arranges the images selected by the selection unit.   The image processing according to claim 4, wherein the setting unit sets a recognition time of the selected image based on an appearance interval of the designated subject of the image arranged by the arrangement unit. apparatus.   The image processing apparatus according to claim 4, wherein the setting unit sets a recognition time of the selected image based on the number of characters appearing within the appearance interval of the designated subject. A selection step in which the selection means selects a plurality of images that are moving images or still images; and
A setting step in which a setting unit sets a recognition time for each of the selected images;
A generation method in which the generation unit generates a video by joining the selected images based on a recognition time by the setting unit.   The program for making a computer perform each process of the control method of the image processing apparatus of Claim 7.

Images