JP4192703B2 - Content processing apparatus, content processing method, and program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a content processing apparatus, a content processing method, and a program.
[0002]
[Prior art]
In recent years, information has been made multimedia, and the amount of information of multimedia contents including video information, audio information, text information, and the like has been rapidly increasing. By storing these pieces of information and recalling them later as necessary, they can be used more effectively.
[0003]
For this reason, there is a content processing apparatus that can effectively use such multimedia content.
This content processing apparatus performs voice recognition based on the voice information added to the video information, and structures the text information and the video information as a voice recognition result.
[0004]
As an example of a system for associating a plurality of types of information, an optimal third text is generated from a first text obtained by character recognition of characters on a manuscript and a second text obtained by voice recognition of voice information. There is an electronic text creation system that obtains text information of digital data (see, for example, Patent Document 1).
[0005]
In addition, there is a system that performs voice recognition based on voice information added to video information and structures text information and video information as a voice recognition result (see, for example, Patent Document 2).
[0006]
Furthermore, there is a multimedia information processing apparatus that digitizes video information, audio information, text information, etc., and stores the information and temporal information indicating the temporal relationship between the information (for example, Patent Document 3 and Patents). Reference 4).
[0007]
[Patent Document 1]
Japanese Patent Laid-Open No. 2001-28279 (page 3-8, FIG. 1)
[Patent Document 2]
JP 2002-189728 A (page 3, FIG. 1)
[Patent Document 3]
JP-A-8-253209 (page 3-8, FIG. 5)
[Patent Document 4]
JP 2002-278974 (page 4-page 6, figure 1)
[0008]
[Problems to be solved by the invention]
However, since the conventional content processing apparatus described in Patent Document 1 performs matching processing between the manuscript and the text of the speech recognition processing result, more accurate text information can be automatically created. However, since Patent Literature 1 does not disclose the association with other information such as video information, the technique described in Patent Literature 1 is applied to the structuring processing of video information, audio information, and text information. In order to do this, an artificially structured process must be performed.
[0009]
Further, in the conventional content processing apparatus described in Patent Document 2, it is possible to structure video information and audio information added to the video information. There is no disclosure regarding conversion.
[0010]
Furthermore, in the conventional content processing apparatus described in Patent Document 3, structuring processing between pieces of information such as video information, audio information, and text information is performed using time information at an input time. The time information needs to be added when creating video information, audio information, text information, and the like. Therefore, when time information is not added, text information, video information, and audio information cannot be automatically associated with each other, and manpower is required for the association.
[0011]
In the conventional content processing apparatus described in Patent Document 4, assuming that time information is described in a text, movie, drama, or play script, and comparing it with the elapsed time of video information, The text information and the video information are associated with each other. However, the time information is often not described in the script. Even when time information is described, there are many cases where the script and the video are shifted in time due to actual shooting or editing. In particular, when a scene is switched in one second or less as in a TV drama, there is a high possibility that these shifts greatly affect the association.
[0012]
In addition, any conventional content processing apparatus relies heavily on voice recognition, but in actual TV programs, even if a person appears, a scene without speech or a voice environment is bad and voice recognition is difficult. There are many cases. In addition, the content of the statement is greatly different from the script due to the ad lib of the performer, and the voice information and the text information may not be associated with each other.
[0013]
If it is difficult to associate audio information with text information, it takes time to generate usable content, and there is a limit to using the content.
[0014]
The present invention has been made in view of such conventional problems, and an object thereof is to provide a content processing apparatus, a content processing method, and a program that can easily use content.
[0015]
[Means for Solving the Problems]
In order to achieve this object, a content processing apparatus according to the first aspect of the present invention provides:
Video information and audio information News In a content processing apparatus for associating content to be included and script information expressing the content scenario with character data,
Recognizing the content, characterizing each scene included in the content with character data Image recognition character information and voice recognition characters A recognition unit that generates information;
Generated by the recognition unit Image recognition character information and voice recognition characters The characteristic part of the information Respectively A time code generation unit that generates a time code indicating a start time and an end time of each divided part,
Acquiring the characteristic part of the script information, dividing the script information for each scene based on the acquired characteristic part, The image recognition character information generated by the recognition unit and the voice recognition character information When The , Dividing at the position indicated by the time code generated by the time code generation unit, Even if the divided image recognition character information and the voice recognition character information do not match, it is determined that each is correct, Each scene of the script information Divided image recognition character information and voice recognition character information With each scene of the script information Divided image recognition character information and voice recognition character information A mapping unit for generating correspondence information indicating a correspondence relationship with
Based on the correspondence information generated by the mapping unit and the time code generated by the time code generation unit, structured information indicating the relationship between each scene of the script information and the time code of each scene is obtained as necessary content. And a structured processing unit that is generated as information for searching.
[0018]
A scene prediction unit that predicts the screen configuration of the script information, adds the predicted screen configuration to the script information, and outputs it to the mapping unit may be provided.
[0019]
You may provide the data storage part for storing the said script information, the said content, the time code which the said time code generation part produced | generated, and the structured information which the said structured process part produced | generated.
[0020]
The data storage unit
A content storage unit for storing the content;
A text file storage unit for storing the script information and the structured information;
A time code storage unit for storing the time code.
[0021]
The structuring processing unit generates a script information file storing the script information and a time code file storing the time code in the text file storage unit, and a start address and an end address of each segment of the script information file And generating management information indicating the start address and end address of each time code in the time code file,
The data storage unit may include a management information storage unit that stores the management information.
[0022]
The data storage unit
A markup language file storage unit that stores the script information as a markup language file may be provided.
[0023]
A synchronization data output unit may be provided that outputs the script information and the content stored in the data storage unit in synchronization based on an input search condition.
[0024]
The synchronous data output unit
An input / output unit that inputs search conditions for extracting necessary scenes from the script information and content, and outputs the script information and content of scenes corresponding to the search conditions;
A search control unit that identifies a scene in the script information corresponding to the search condition input to the input / output unit, and extracts a time code corresponding to the identified scene;
A synchronization processing unit that identifies the scene of the content corresponding to the extracted time code, and synchronizes the content of the identified scene and the script information corresponding to the search condition;
A synchronization processing unit that outputs content and script information corresponding to the scene synchronized by the synchronization processing unit to the input / output unit may be provided.
[0025]
A content processing method according to a second aspect of the present invention includes:
Video information and audio information News A content processing method for associating content to be included and script information in which a scenario of the content is expressed by character data,
Recognizing the content, the feature part of each scene included in the content is represented by character data Image recognition character information and voice recognition characters Generating information;
The generated Image recognition character information and voice recognition characters The characteristic part of the information Respectively Extracting and separating, generating a time code indicating a start time and an end time of each separated part;
Acquiring the characteristic part of the script information, dividing the script information for each scene based on the acquired characteristic part, Generated image recognition character information and voice recognition character information When The Generated Divide at the position indicated by the time code, Even if the divided image recognition character information and the voice recognition character information do not match, it is determined that each is correct, Each scene of the script information The divided image recognition character information and voice recognition character information With each scene of the script information Divided image recognition character information and voice recognition character information Generating correspondence information indicating a correspondence relationship with
Based on the generated correspondence information and the generated time code, structured information indicating a relationship between each scene of the script information and the time code of each scene is generated as information for searching for necessary content. Steps,
Storing the script information, content, time code, and structured information.
[0026]
The program according to the third aspect of the present invention is:
On the computer,
The content Information and audio information contained in The feature part of each scene included in the content is expressed as character data. Image recognition character information and voice recognition characters Procedures for generating information,
The generated Image recognition character information and voice recognition characters The characteristic part of the information Respectively A procedure to extract and delimit, and generate a time code indicating the start time and end time of each delimited part,
Above Representing the content scenario as text data Acquiring the characteristic part of the script information, dividing the script information for each scene based on the acquired characteristic part, The generated image recognition character information and the voice recognition character information When The Generated Divide at the position indicated by the time code, Even if the divided image recognition character information and the voice recognition character information do not match, it is determined that each is correct, Each scene of the script information The divided image recognition character information and voice recognition character information With each scene of the script information Divided image recognition character information and voice recognition character information Generating correspondence information indicating a correspondence relationship with
Based on the generated correspondence information and the generated time code, structured information indicating the relationship between each scene of the script information and the time code of each scene is generated as information for searching for necessary content. procedure,
A procedure for storing the script information, content, time code, structured information;
Is to execute.
[0027]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, a content processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 shows the configuration of the content processing apparatus according to the present embodiment.
The content processing apparatus includes a synchronization data generation unit 1, a data storage unit 2, and a synchronization data output unit 3.
[0028]
Note that the content holder 4 is a holder that holds a video medium 5 in which video information and audio information are recorded, and a text medium 6 in which a document related to the video information is described.
[0029]
The video media 5 and the text media 6 are external recording media, and magnetic recording tapes and DVDs (Digital Versatile Disks) are used. Although the recording capacity is small, a flexible disk, MD (Mini Disc; registered trademark), MO (Magneto-Optic), CD-ROM (Compact Disk Read-Only Memory) or the like is used for the video media 5 and the text media 6. You can also
[0030]
In the case of a drama program in television broadcasting, the text media 6 records drama program script data as text information. This text information expresses a scenario of content with character data. The video media 5 records video information of the drama program and audio information such as the voice of the actor uttered according to the dialogue.
[0031]
Data recorded on the video media 5 and the text media 6 taken out from the content holder 4 is input to the synchronous data generation unit 1.
[0032]
The synchronous data generation unit 1 extracts data recorded in the video media 5 and the text media 6, performs data processing, and generates synchronized data.
[0033]
The synchronization data generation unit 1 includes a text input unit 11, a scene prediction unit 12, a video / audio input unit 13, an image recognition unit 14, an audio recognition unit 15, a time code generation unit 16, a mapping unit 17, And a structured processing unit 18.
[0034]
The text input unit 11 reads text information from the text media 6. When the text media 6 is a recording medium on which analog data is recorded, the text input unit 11 reads analog data from the text media 6, for example, an OCR (optical character reading) device and a digital-analog conversion. (Not shown). The text input unit 11 outputs the text information read from the text media 6 to the scene prediction unit 12.
[0035]
The scene prediction unit 12 predicts the screen configuration and audio configuration of each scene based on the text information output from the text input unit 11. The screen configuration refers to the names and movements of persons and objects shown on the screen. In addition, the voice configuration refers to a person's voice, sound, music, and sound effect in the scene.
[0036]
In order to predict the screen configuration and the voice configuration, the scene predicting unit 12 omits a rule, for example, a rule that the face of the person who entered the script appears in the script unless there is an instruction for camera work, and the subject. In this case, rules such as using the subject of the previous scene are stored in advance.
[0037]
Then, the scene prediction unit 12 predicts the screen configuration and audio configuration of each scene described in the script based on the accumulated rules and the text information output from the text input unit 11. For the prediction of the screen configuration and audio configuration of each scene, it is sufficient to list the contents described in the script for each scene.
[0038]
When the description is ambiguous, for example, in the script of a TV drama, there is a description that “person A goes out of the room”, the scene prediction unit 12 does not specify that the person A disappears on the screen. From the current scene setting and the action of the person, it is estimated that this person A has disappeared from the screen.
The scene prediction unit 12 adds the predicted screen configuration and audio configuration of each scene to the text information and outputs the text information to the mapping unit 17.
[0039]
The video / audio input unit 13 extracts video information and audio information recorded on an audio track from the video medium 5. Note that the video / audio input unit 13 includes a video capture or the like in order to read analog data from the video media 5 when the video media 5 is a recording medium on which analog data is recorded. The video / audio input unit 13 reads analog video information and analog audio information from the video media 5 and converts them into digital video information and digital audio information in AVI format or MPEG format using video capture or the like.
The video / audio input unit 13 outputs the extracted video information and audio information to the image recognition unit 14 and the audio recognition unit 15, respectively.
[0040]
The image recognition unit 14 performs image recognition processing on the video information output from the video / audio input unit 13 and generates image recognition character information as an image recognition result. This image recognition character information is a character data representing a characteristic part of each scene included in video information. For example, when the face of criminal A is shown as a character in the video, the image recognition unit 14 generates image recognition character information “face of criminal A” as an image recognition result.
[0041]
In order to perform such image recognition processing, the image recognition unit 14 performs cut, camera work, lighting environment, background, characters, facial expressions, face orientation, age, gender, hairstyle, makeup, and human actions from video information. Extract all or some of the features of the object in the screen and the motion of the object. Then, the image recognition unit 14 divides and divides the image based on the extracted feature portion, and performs image recognition processing.
[0042]
Further, the image recognition unit 14 outputs the video information to the time code generation unit 16 and outputs information indicating the division position of the video information to the time code generation unit 16.
[0043]
The voice recognition unit 15 performs voice recognition processing on the voice information output from the video / audio input unit 13 and generates voice recognition character information as a voice recognition result. The voice recognition character information is a character data representing a characteristic part of each scene included in the voice information. For example, when the voice includes criminal A's voice as a character, the voice recognition unit 15 generates voice recognition character information “voice of criminal A” as a voice recognition result.
[0044]
In order to perform such a speech recognition process, the speech recognition unit 15 extracts all or a part of the feature portions of the content of a utterance, a speaker, music, sound effects, other non-language speech, and a silent section.
[0045]
Further, the voice recognition unit 15 performs a word segmentation process on the generated voice recognition character information, and divides the voice information into each word. Then, the voice recognition unit 15 outputs information indicating the division position when the voice information is divided into words together with the voice information to the time code generation unit 16.
[0046]
The time code generation unit 16 generates a time code related to images and sounds based on information output from the image recognition unit 14 and the voice recognition unit 15. The time code indicates time information of start and end of the division position of the video information and audio information.
[0047]
The time code generation unit 16 generates the following time code based on the feature portion included in the image.
(1) Cut time code: Indicates the cut start time and end time in the image information.
(2) Camera work time code: Indicates the start time and end time of camera work.
(3) Lighting environment time code: Indicates the start time and end time of the lighting environment.
(4) Background time code: Indicates the start time and end time of the background.
(5) Character time code: Indicates the appearance start time and end time of the character.
(6) Facial feature time code: indicates the start time and end time of facial features such as the face direction, facial expression, and makeup of a person.
(7) Person feature time code: Indicates the start time and end time of person features such as age, gender, hairstyle, makeup, clothes, and appearance.
(8) Person action time code: Indicates the start time and end time of a person action.
(9) Object time code: Indicates the appearance start time and end time of an object.
(10) Object motion time code: Indicates the start time and end time of the motion of the object.
[0048]
Further, the time code generation unit 16 generates the following time code based on the feature portion included in the voice.
(11) Word time code: indicates the start time and end time of each word in the audio information.
(12) Speaker time code: Indicates the start time and end time of each speaker in the voice information.
(13) Non-verbal voice time code: Indicates the start time and end time of non-language voice in voice information such as screams, laughter, and sighs.
(14) Music time code: Indicates the start time and end time of music in the audio information.
(15) Sound effect time code: Indicates the start time and end time of the sound effect.
(16) Silent section time code: Indicates the start time and end time of the silent section.
[0049]
The time code generation unit 16 outputs all or part of the generated time codes related to the image and sound to the image recognition unit 14 and the sound recognition unit 15, respectively.
[0050]
The above-described image recognition unit 14 adds all or a part of the time code output from the time code generation unit 16 to the image recognition character information, and converts the image recognition character information with the time code into video. It outputs to the mapping part 17 with information.
[0051]
The voice recognition unit 15 assigns each time code output from the time code generation unit 16. That is, the voice recognition unit 15 assigns a word time code to each word. The voice recognition unit 15 sets the speaker time code to each utterance, the non-language voice time code to non-language voice, the music time code to music, the sound effect time code to sound effect, and the silence interval time code to silence interval. Assign.
[0052]
The voice recognition unit 15 adds all or part of the time code output from the time code generation unit 16 to the voice recognition character information, and converts the voice recognition character information with the time code into the voice code. It outputs to the mapping part 17 with information.
[0053]
The mapping unit 17 divides the text information output from the scene prediction unit 12 for each predicted scene (scene), and the divided text information and the divided images output from the image recognition unit 14 and the voice recognition unit 15 respectively. The recognition character information and the voice recognition character information are associated with each other.
[0054]
Specifically, the mapping unit 17 uses the text information output from the scene prediction unit 12 to switch speaker, blank lines, line breaks, etc. by line breaks, indents, person names, proper names, place names, etc. Detected as script breakpoint. When the mapping unit 17 detects the delimiter, the mapping unit 17 delimits the text information at the delimiter position so that it can be easily searched.
[0055]
Next, the mapping unit 17 compares the text information with the divided image recognition character information and speech recognition character information, respectively.
[0056]
For comparison, for example, DP matching is used. DP matching is also called DTW (Dynamic Time Warping), and time normalization is performed using dynamic programming (Dymanic Programming) so that the same phonemes in a word correspond to each other. This is a method for obtaining.
[0057]
Next, the mapping unit 17 associates these pieces of information to generate correspondence information. As shown in FIG. 2, the correspondence information is information indicating a one-to-one relationship between the divided image recognition character information and voice recognition character information and text information.
[0058]
Even if the text information, the image recognition character information, and the voice recognition character information do not match, the mapping unit 17 determines that each recognition result is correct and performs the association.
[0059]
Then, the mapping unit 17 outputs the divided image recognition character information, voice recognition character information, text information, and correspondence information to the structuring processing unit 18.
[0060]
The structured processing unit 18 associates each scene (scene) into which the text information is divided and each time code based on the time code and the correspondence information, and generates structured information. As shown in FIG. 2, the structured information is information indicating a one-to-one relationship between each scene of the associated text information and each time code.
[0061]
The structuring processing unit 18 calculates each time code that is a start time and an end time of each segment of the text information from each time code added to the divided image recognition character information and the divided speech recognition character information.
[0062]
Specifically, the structuring processing unit 18 corresponds to each delimiter of the text information from all or part of the time codes (1) to (10) added to the divided image recognition character information. The time code to be calculated is calculated. The structuring processing unit 18 associates the text information with the text or dialogue from the start time of the first word and the end time of the last word of each segment.
[0063]
The structuring processing unit 18 uses any of the time codes in association with the overwritten portion of the text information instructing the situation explanation or the photographing condition. Regarding the correspondence with speech for each speaker, the structuring processing unit 18 uses not only the person time code indicating the speaker but also other time codes by using the situation explanation and proper nouns in the content of the speech. Is associated.
[0064]
Further, the structuring processing unit 18 calculates a time code corresponding to each delimiter of the text information from all or part of the time codes (11) to (16) added to the divided speech recognition character information. To do.
[0065]
That is, the structuring processing unit 18 calculates a time code that is the appearance start time and end time of the speaker in the divided speech recognition text information from the speaker time code added to the divided speech recognition character information. Is associated. Further, the structuring processing unit 18 associates the start time and end time of music or sound effect in the writing with the music time code or sound effect time code. Further, the structuring processing unit 18 associates the switching of the speaker and the switching of the scene in the text information from the silent section time code.
[0066]
Further, the structuring processing unit 18 generates a text media file and a time code file in association with the writing or dialogue for each speaker based on the structured information. The text media file is a file that stores text information, and the time code file is a file that stores time code.
[0067]
The structuring processing unit 18 stores the time code corresponding to each segment in the time code file in the order of appearance in the text information. The structuring processing unit 18 can also generate a plurality of text media files and time code files each corresponding to each delimiter.
[0068]
The data storage unit 2 includes a video media storage unit 21, a time code storage unit 22, and a text media storage unit 23.
[0069]
The video media storage unit 21, the time code storage unit 22, and the text media storage unit 23 store video media files, time code files, and text media files, respectively. The video media storage unit 21 and the text media storage unit 23 correspond to a video information storage unit and a text file storage unit, respectively.
[0070]
The structured processing unit 18 stores the video media file, the generated time code file, and the text media file in the video media storage unit 21, the time code storage unit 22, and the text media storage unit 23, respectively. Further, the structured processing unit 18 stores the structured information shown in FIG. 2 in the text media storage unit 23.
[0071]
The synchronous data output unit 3 searches for data to be searched from the data stored in the data storage unit 2, and outputs the corresponding data. The input / output unit 31, the search control unit 32, And a processing unit 33.
[0072]
The input / output unit 31 receives an input of a search target requested by the user, supplies the received input of the search target to the search control unit 32 as search information, and outputs search result information obtained as a result of the search. Is. Search information includes, for example, keywords such as date and time, program title, speaker, and actor name.
[0073]
The search control unit 32 searches the data stored in the text media storage unit 23 according to the search information supplied from the input / output unit 31, and the scene portion of the text information corresponding to the search information is input / output unit 31. Output to.
[0074]
Specifically, the search control unit 32 retrieves text information including the phrase selected by the user from the text media storage unit 23 for each scene based on the search information. In addition, the search control unit 32 takes out the time code for delimiting the text information selected by the user from the time code storage unit 22 based on the structured information stored in the text media storage unit 23, and extracts the extracted time code. The code is output to the synchronization processing unit 33.
[0075]
The synchronization processing unit 33 sends the start time and end time indicated by the time code output from the search control unit 32 to the input / output unit 31 as the start time and end time of video information output, respectively, from the start time to the end time. Output video information. The synchronization processing unit 33 processes the text information based on the time code, and outputs the processed text information to the input / output unit 31. As a processing method at this time, for example, there is a method of scrolling text information.
[0076]
Next, a hardware configuration for realizing such a content processing apparatus will be described.
The content processing apparatus shown in FIG. 1 is realized by a computer system as shown in FIG.
[0077]
That is, the computer system includes terminals 41 and 43 and a storage device 42. The terminals 41 and 43 and the storage device 42 are connected via a communication line 44.
[0078]
The content processing apparatus may be constructed on a computer system as shown in FIG. 3 or may be constructed on the same terminal. In the case of a computer system as shown in FIG. 3, a LAN (Local Area Network), the Internet, or the like is used for the communication line 44, and the terminals 41 and 43 and the storage device 42 are connected by a network.
[0079]
Terminals 41 and 43 are computers, each having functions of a synchronous data generation unit 1 and a synchronous data output unit 3. The storage device 42 corresponds to the data storage unit 2 and is configured by a magnetic disk device or the like.
[0080]
As shown in FIG. 4, the terminals 41 and 43 include a CPU 51, a ROM 52, a RAM 53, a display device 54, an input device 55, an HDD 56, and a drive device 57.
[0081]
A ROM (Read Only Memory) 52 is a memory for storing a program (data) for causing the CPU 51 to function as the synchronization data generation unit 1 and the synchronization data output unit 3.
A CPU (Central Processing Unit) 51 executes a program stored in the ROM 52.
[0082]
A RAM (Random Access Memory) 53 is a memory for storing data necessary for the CPU 51 to execute a program. Note that the RAM 53 of the terminal 41 is also used to store text information output by the scene prediction unit 12.
[0083]
The display device 54 includes a liquid crystal display that displays data. The input device 55 is for inputting data, and includes a keyboard, a mouse, a microphone, an image scanner, a camera, a video capture interface, and the like. The display device 54 and the input device 55 of the terminal 43 function as the input / output unit 31 of the synchronous data output unit 3.
[0084]
An HDD (Hard Disk Drive) 56 is a storage device for storing data.
The drive device 57 is for loading external recording media such as video media 5 and text media 6 and reading recorded data from the external recording media. The drive device 57 of the terminal 41 functions as the text input unit 11 and the video / audio input unit 13 of the synchronous data generation unit 1.
[0085]
Next, the operation of the content processing apparatus according to the present embodiment will be described based on the flowchart shown in FIG.
The video / audio input unit 13 and the text input unit 11 of the synchronous data generation unit 1 input the video / audio information and text information from the video media 5 and the text media 6, respectively (step S101).
The text input unit 11 determines whether or not the input text information is digital data (step S102).
[0086]
If it is determined that the text information is composed of digital data (Yes in step S102), the text information is output to the scene prediction unit 12.
[0087]
On the other hand, when it is determined that the text information does not consist of digital data, ie, analog data (No in step S102), the text input unit 11 uses the OCR or the like to store the text information recorded on the text media 6 Is digitized (step S103). Then, the text input unit 11 outputs the digitized text information to the scene prediction unit 12.
[0088]
The scene prediction unit 12 predicts the screen configuration and audio configuration of each scene based on the text information output from the text input unit 11, and adds the predicted screen configuration and audio configuration of each scene to the text information for mapping. Part 1 7 (Step S104).
[0089]
The video / audio input unit 13 inputs video information and audio information from the video medium 5, and determines whether or not the input video information and audio information are digital data (step S105).
[0090]
When it is determined that the video information and the audio information are composed of digital data (Yes in step S105), the video / audio input unit 13 outputs the video information and the audio information to the image recognition unit 14 and the audio recognition unit 15, respectively.
[0091]
On the other hand, when it is determined that the video information and audio information are not composed of digital data, that is, are composed of analog data (No in step S105), the video and audio input unit 13 converts the analog video information and analog audio information into Using video capture or the like, it is converted into information consisting of digital data in AVI format or MPEG format (step S106). Then, the video / audio input unit 13 outputs the digitized video information and audio information to the image recognition unit 14 and the audio recognition unit 15, respectively.
[0092]
The image recognition unit 14 performs image recognition processing on the video information supplied from the video / audio input unit 13 to generate image recognition character information (step S107). The image recognition unit 14 outputs information indicating the division position of the video information together with the video information to the time code generation unit 16.
[0093]
The voice recognition unit 15 performs voice recognition processing on the voice information output from the video / audio input unit 13, and generates voice recognition character information as a result of the voice recognition processing (step S108). The voice recognition unit 15 outputs information indicating the division position of the voice information together with the voice information to the time code generation unit 16.
[0094]
The time code generation unit 16 generates all or part of the time codes (1) to (16) related to video and audio based on the information output from the image recognition unit 14 and the voice recognition unit 15 (steps). S109). The time code generation unit 16 outputs the generated video and audio time codes to the image recognition unit 14 and the voice recognition unit 15, respectively.
[0095]
The image recognition unit 14 adds a time code related to the video output from the time code generation unit 16 to the image recognition character information (step S110). The voice recognition unit 15 adds the time code related to the voice output from the time code generation unit 16 to the voice recognition character information (step S110). The image recognition unit 14 and the voice recognition unit 15 output the image recognition character information and the voice recognition character information divided by adding the time code to the mapping unit 17 together with the video information and the voice information, respectively.
[0096]
The mapping unit 17 temporarily stores the text information output from the scene prediction unit 12 and divides the text information according to the text or the dialogue for each speaker.
The mapping unit 17 compares the text information with the divided image recognition character information and voice recognition character information, and based on the paragraph break position of the text information, the image recognition character information and voice recognition character divided with the text information. Correlate with information.
[0097]
Further, the mapping unit 17 generates each paragraph of the text information and the correspondence information shown in FIG. 2 (step S111), and processes the correspondence information together with the text information, the divided image recognition character information and voice recognition character information. To the unit 18.
[0098]
The structured processing unit 18 generates structured information shown in FIG. 2 based on the time code added to the divided image recognition character information and speech recognition character information (step S112). Based on the structured information, the structured processing unit 18 generates a text media file and a time code file by associating the text media file and the dialogue for each speaker.
[0099]
The structured processing unit 18 stores the generated text media file in the text media storage unit 23 of the data storage unit 2 and stores the time code file in the time code storage unit 22. The structured processing unit 18 The media file is stored in the video media storage unit 21 (step S113).
And the synchronous data production | generation part 1 complete | finishes this process.
[0100]
Next, when the user inputs search conditions, the synchronous data output unit 3 searches for data and outputs the searched data. The synchronous data output process of the synchronous data output unit 3 will be described based on the flowchart shown in FIG.
[0101]
When the user inputs search information, the input / output unit 31 outputs the search information to the search control unit 32 in response to the input operation (step S201).
[0102]
The search control unit 32 searches the data stored in the data storage unit 2 based on the search condition output from the input / output unit 31 (step S202).
[0103]
The search control unit 32 determines whether there is corresponding data that matches the search condition (step S203).
If it is determined that there is no corresponding data (No in step S203), the search control unit 32 displays that there is no corresponding data and outputs a sound to that effect (step S206).
[0104]
On the other hand, when it is determined that the corresponding data is present (Yes in step S203), the search control unit 32 specifies the paragraph of the text information and extracts all the corresponding data (step S204).
[0105]
Based on the structured information stored in the text media storage unit 23, the search control unit 32 takes out the time code of the text information selected by the user from the time code storage unit 22, and extracts the extracted time code. The data is output to the synchronization processing unit 33 (step S204).
[0106]
The synchronization processing unit 33 sets the start time and end time of the time code output by the search control unit 32 as video information corresponding to the paragraph of the text information as the start time and the final time of reproduction, respectively. 31 and supplies the text information, video information, and audio information of the selected paragraph to the input / output unit 31 (step S205).
[0107]
The input / output unit 31 displays a script and video based on the supplied information, and outputs audio (step S206).
In this way, the synchronous data output unit 3 provides data to the user.
[0108]
Next, specific operations will be described in more detail.
For example, the video media 5 and the text media 6 are stored in the content holder 4 of the production company in charge of the production of the drama program, and are taken out from the content holder 4.
[0109]
The text input unit 11 of the synchronous data generation unit 1 takes out text information from the text media 6 in which a content scenario is expressed by character data. The text input unit 11 outputs text information related to the dialogue to the scene prediction unit 12.
[0110]
As shown in FIG. 7, the scene prediction unit 12, in order from the scene 1 to the scene 7, in order from the scene 1 to the scene 7, the face of the criminal A, the face of the criminal A, the faces of all the criminal sections, the face of the criminal B, A screen configuration such as the faces of criminal A and criminal B, no face, and the face of criminal A (the face of section C will be added later) is predicted (processing in step S104 in FIG. 5). The scene prediction unit 12 adds this prediction content to the text information and outputs it to the mapping unit 17.
[0111]
The image recognition unit 14 recognizes the image, and the voice recognition unit 15 recognizes the voice and generates an image recognition result and a voice recognition result as shown in FIG. 7 (processing in steps S107 and S108), respectively.
[0112]
For example, the time code generation unit 16 generates “00: 00: 02: 13,00: 00: 02: 26” as the time code of the image recognition result “the face of criminal A” generated by the image recognition unit 14. (Processing of step S109). “00: 00: 02: 13” and “00: 00: 02: 26” indicate the start time and end time of the image recognition result “face of criminal A”, respectively.
[0113]
In addition, the time code generation unit 16 sets “00: 00: 02: 15,00: 00” as the time code as the time code as the voice recognition result generated by the voice recognition unit 15 to the voice “robbery” and “gray” of the criminal A. : 02: 26 "is generated (processing in step S109). “00: 00: 02: 15” and “00: 00: 02: 26” indicate the voice recognition result, the start time and the end time of criminal A's voice “robbery” and “gray”, respectively.
[0114]
The mapping unit 17 sequentially compares the text information, the image recognition character information, and the voice recognition character information using a DP matching method. And the mapping part 17 performs matching as shown in FIG. 7 (process of step S111).
[0115]
For example, as shown in FIG. 7, in the script, “Receiving a criminal A call” is described in the scene 1, and the time code “00: 00: 02: 13 to 00:00:02” of the image recognition result is written. : 26 ”includes“ the face of criminal A ”. The voice recognition character information of the time code “00: 00: 02: 15 to 00: 00: 02: 26” of the voice recognition result includes “voice of criminal A”. In the script scene 1, the image recognition result, and the speech recognition result, “criminal A” matches. Therefore, the mapping unit 17 performs DP matching to determine that the similarity distance between the script scene 1 and the image recognition result and the speech recognition result is short, and the script scene 1 and the image recognition result and the speech recognition result. Is associated.
[0116]
Similarly, the mapping unit 17 sequentially associates the script scenes 2 to 7 with the image recognition result and the voice recognition result.
[0117]
In this case, if the appearance time of the criminal A's face differs from the appearance time of the criminal A's voice as the image recognition result and the voice recognition result, the mapping unit 17 may increase the scene time even though the scenes are the same. To.
[0118]
For example, when the appearance start times are different as in the case of the scene 1, the mapping unit 17 adopts the time “00: 00: 02: 13” where the appearance is earlier. Further, when the appearance end times are different as in the case of the scene 5, the mapping unit 17 adopts the time “00: 00: 02: 38” at which the exit or disappearance is later. Alternatively, as shown in FIG. 7, the mapping unit 17 can take a method in which the image recognition result is regarded as the time code of the writing portion and the speech recognition result is regarded as the time code of the dialogue portion. The mapping unit 17 can also consider that a time code exists for each scene element such as a face, an object, a voice, or a sound in the scene even in the same scene.
[0119]
For the characters, the mapping unit 17 can also identify the speaker based on the speech recognition result obtained by the speech recognition process. The mapping unit 17 realizes more robust person identification by using the image and the sound at the same time.
[0120]
For example, a person can be identified by image recognition even when the character is silent or when voice recognition is difficult due to music or ambient noise. On the other hand, a person can be identified by voice recognition even when lighting conditions such as backlighting are poor, or even when a character is facing backward or downward and the face cannot be seen.
[0121]
If the image information, the sound information, and the text information are inconsistent with each other, the mapping unit 17 determines that each recognition result is correct and performs the association. For example, in a scene, if only people A and B are detected in image recognition, and people A and C are detected in voice recognition, the mapping unit 17 responds as if all A, B, and C names exist. To do.
[0122]
For example, when the face of the detective A cannot be detected as in the scene 5, the mapping unit 17 can make the correspondence based on the order of the appearance order of the faces, and can detect the detective B by voice recognition at almost the same time. When the words “injury” and “unreasonable” are detected in the voice, the association is performed using these words. In this way, the association is performed with higher reliability.
[0123]
On the other hand, even if voice recognition cannot be performed in this scene, if the faces of detective A and detective B can be detected, the mapping unit 17 can also associate the script with the image recognition result and the voice recognition result. . Further, even when image recognition and voice recognition cannot be performed in this scene, the mapping unit 17 can perform association even in a scene that could not be recognized as long as it can be associated from the appearance order of faces and voices in the preceding and following scenes.
[0124]
However, the mapping unit 17 can also associate all combinations. For example, in this case, the mapping unit 17 associates with the assumption that A, B, and C3 names exist, associates with the assumption that only two names A and B exist, and only two names A and C exist. Then, the assumed association and the association assumed that only A exists can be performed.
[0125]
In addition, the mapping unit 17 similarly takes correspondences with respect to cuts, camera work, backgrounds, human motions, objects in frames, motions of words, words in speech, music, and sound effects, and detects a break position. be able to. If there is a contradiction with each other, the recognition results may be associated with each other, or all combinations may be associated with each other.
[0126]
When the mapping is performed, the mapping unit 17 estimates the length of the scene not only based on the order but also from the length of the script and the writing, etc., and the appearance time of the detected face is within a predetermined range (for example, estimation) It is also possible to determine whether or not it is supported by whether it is within the range of 0.5 to 1.5 times the length of the scene.
[0127]
The structured processing unit 18 associates the text information and the time code as follows, and generates a text media file and a time code file (processing in step S112).
[0128]
That is, for example, the structuring processing unit 18 sets the file name of the first line text media file to “line 1.txt” and the file name “time 1.txt” of the first line time code file.
The structuring processing unit 18 associates the first dialogue file with the file name excluding the extension “1” at the end.
[0129]
Similarly, the structuring processing unit 18 associates the files in the second paragraph by setting the second lines to “line 2.txt” and “time 2.txt”, respectively. Similarly, the structuring processing unit 18 performs association in the third and fourth paragraphs. If there is a writing, the structuring processing unit 18 associates the writing as well.
[0130]
Then, the structuring processing unit 18 stores the text media file, the time code file, and the video media file in the data storage unit 2 (Step S113).
[0131]
Next, when the user operates the terminal 43 to search the stored data, the terminal 43 displays a search screen as shown in FIG. 8 on the display device 54 in response to this operation.
[0132]
The display device 54 of the terminal 43 displays on this search screen input fields for date and time, program title, speaker, and keyword as a search condition input screen of the video search system. Further, the display device 54 of the terminal 43 displays a search execution button for designating search execution on this search screen.
[0133]
When the user inputs “actor D” as a keyword, for example, according to the displayed search condition input screen, the display device 54 of the terminal 43 displays “actor D” in the keyword input field in response to this operation. To do.
[0134]
When searching, general search processing such as full-text matching search and AND search based on each search condition can be used.
[0135]
When the user clicks the search execution button, the input / output unit 31 of the synchronous data output unit 3 supplies the search information to the search control unit 32 in response to this operation (processing in step S201), and the search control unit 32 starts the search (processing of step S202).
[0136]
The search control unit 32 specifies a paragraph of text information having data matching the search condition from the data stored in the data storage unit 2 based on the input search condition.
[0137]
The search control unit 32 converts “actor D” input as a keyword phrase into “criminal B” from the cast table in the script. Then, the search control unit 32 searches the data stored in the data storage unit 2 based on “criminal B”.
[0138]
If there are 12 data corresponding to “criminal B”, the search control unit 32 retrieves the 12 corresponding data and supplies the text information scene to the input / output unit 31 as a search result (steps S203 and S204). Processing).
[0139]
Further, the synchronization processing unit 33 notifies the input / output unit 31 of the start time and the final time for reproducing the video information, and synchronizes the video information and audio information with the text information of the selected paragraph. Supply (processing of step S205).
[0140]
The display device 54 of the terminal 43 displays the script and video as shown in FIG. 9 supplied to the input / output unit 31 and outputs audio (processing in step S206).
[0141]
When extracting video information, the display device 54 of the terminal 43 can also display a representative image as a thumbnail image together with text information, as shown in FIG. 8, for example. Further, the display device 54 can also display a moving image from the start time to the end time of a paragraph, regardless of the still image.
[0142]
For example, as illustrated in FIG. 9, the synchronization processing unit 33 operates according to the operation information when each switch unit such as rewind, playback, stop, pause, and fast-forward displayed on the display device 54 is operated. .
[0143]
For example, when the user operates the input device 55 of the terminal 43 so as to reproduce the video media 5 from the position 2 minutes 31 seconds from the beginning, the synchronization processing unit 33 causes the text information to be synchronized with the reproduction timing of the video information. A process of scrolling information is executed.
[0144]
The display device 54 of the terminal 43 performs scroll display of the text information portion including, for example, “Healed by injury” as the text information of the selected paragraph in accordance with the reproduction timing of the video information.
[0145]
Further, the display device 54 of the terminal 43 displays an italic character with an underline, as shown in FIG. 9, in order to indicate that the text information is reproduced.
[0146]
As described above, according to the present embodiment, since the text information is associated with the divided image recognition character information and voice recognition character information, the text information and the video information through the time code, Audio information can be easily structured.
[0147]
In addition, the division part in the text information that matches the search condition input by the user is specified, the time code corresponding to the specified division part is extracted, and the video information corresponding to the extracted time code is specified and specified. Provided video information to users. Therefore, desired video information can be provided to the user.
[0148]
Furthermore, by using image information and audio information together, it is possible to associate text information and video information by associating with audio information or associating with video information even if it is not associated only with video information and audio information. Structuring can be performed more accurately.
[0149]
In carrying out the present invention, various forms are conceivable and the present invention is not limited to the above embodiment.
For example, in the above embodiment, the video information is supplied from the video / audio input unit 13 to the data storage unit 2 via the image recognition unit 14, the voice recognition unit 15, the mapping unit 17, and the structured processing unit 18. Configured. However, the video information may be directly supplied from the video / audio input unit 13 to the data storage unit 2.
[0150]
Further, the structuring processing unit 18 generates, as management information, the start address and end address of each segment of the text information in the text media file, and the start address and end address of each segment of the time code in the time code file. May be configured.
[0151]
To do so, the content processing apparatus includes a management file storage unit 24 for storing management information in the data storage unit 2 as shown in FIG. The management file storage unit 24 corresponds to a management information storage unit. The structured processing unit 18 stores the management information in the management file storage unit 24.
[0152]
Further, as shown in FIG. 11, the content processing apparatus can be configured to combine text information and a time code to generate a structured result. In this case, the data storage unit 2 includes a video media storage unit 21 and a text media storage unit 23 as shown in FIG. The structured processing unit 18 combines the text information and the time code to generate a structured text media file including the time code, and stores the generated text media file in the text media storage unit 23. The data storage unit 2 need not include the time code storage unit 22.
[0153]
Further, the content processing apparatus may be configured to generate and store an XML (Extensible Markup Language) file as shown in FIG. The XML file is an XML file having a structural description in the MPEG7 (moving picture expert group 7) format in the XML (Extensible Markup Language) language which is an extended markup language.
[0154]
As shown in FIG. 14, the data storage unit 2 includes an XML file storage unit 25. In this case, the data storage unit 2 does not need to include the time code storage unit 22 and the text media storage unit 23. The structuring processing unit 18 generates an XML file.
[0155]
The structuring processing unit 18 generates an XML file according to the flowchart shown in FIG.
[0156]
That is, the structuring processing unit 18 uses the tag to insert the storage position (folder position) information of the video information in the computer system into the XML template (step S301).
[0157]
The structured processing unit 18 uses the tag to insert program title information described in the text media 6 into the XML template (step S302).
[0158]
The structuring processing unit 18 inserts the start time and end time information of each paragraph into the XML template using the tag (step S303).
[0159]
The structuring processing unit 18 inserts the character information into the XML template using the tag (step S304).
[0160]
The structured processing unit 18 inserts text information into the XML template using the tag (step S305).
[0161]
The structuring processing unit 18 determines whether or not the end time of the inserted dialogue has reached the final time of the time code (step S306).
When it is determined that the final time has not been reached (No in step S306), the structuring processing unit 18 inserts again the start time, end time information, character information, and text information of each paragraph in the XML template ( Steps S303 to S305).
[0162]
If it is determined that the final time has been reached (Yes in step S306), the structuring processing unit 18 stores the XML file in the XML file storage unit 25 of the data storage unit 2 (step S307).
[0163]
In the XML file shown in FIG. xml> and <Mpeg7> are XML templates defined in advance as the MPEG7 standard.
[0164]
The <Media Locator> tag and the <MediaUri> tag are tags indicating the storage location (folder location) of the video information. “C: ¥ meta information ¥ video data ¥ drama video 020913.mpg” indicates a storage position in the video media storage unit 21 in which the video information inserted by the structured processing unit 18 is to be stored (processing result of step S301). .
[0165]
The <Title> tag in the <CreationInformation> tag is a tag for inserting program title information. “Drama program 020913” indicates program title information inserted by the structuring processing unit 18 (processing result of step S302).
[0166]
A <MediaTime> tag, a <MediaRelTimePoint> tag, and a <MediaDuration> tag are tags for inserting information about the start time and end time of each paragraph. The structuring processing unit 18 uses the tag to insert information about the start time and end time of each paragraph (processing result in step S303).
[0167]
The <Name> tag, <GivenName> tag, and <FamilyName> tag are characters information insertion tags. “Criminal A” is the character information inserted by the structuring unit 18 (processing result of step S304).
[0168]
The <TextAnnotation> tag and the <FreeTextAnnotation> tag are text information insertion tags. The dialogue including “XXX 1 robbery case at 1 chome” is text information inserted by the structuring processing unit 18 using these tags (processing result of step S305).
[0169]
If the end time of the inserted dialogue has not reached the final time of the time code, the structuring processing unit 18 inserts each piece of information of the next dialogue (step S306). When the end time of the dialogue reaches the final time of the time code, the structuring processing unit 18 stores the XML file in the XML file storage unit 25 of the data storage unit 2 (processing in step S307).
[0170]
After the structured processing unit 18 stores such an XML file in the XML file storage unit 25, the user inputs desired words and phrases to the synchronous data output unit 3 to request desired video information and text information. Shall.
[0171]
The input / output unit 31 outputs the word / phrase input by the user to the search control unit 32. The search control unit 32 searches the data stored in the XML file storage unit 25 for the paragraph of the text information including the word, and outputs the paragraph part to the input / output unit 31 if there is corresponding text information. To do.
[0172]
If the user selects a specific range of certain text information, the input / output unit 31 requests the search control unit 32 to output video information synchronized with the text information selected by the user.
[0173]
The above is an application example of a content processing apparatus configured to generate and store an XML (Extensible Markup Language) file as shown in FIG.
[0174]
The input / output unit 31 of the synchronous data output unit 3 can also be configured to input voice spoken by the user. In this case, the input device 55 of the terminal 43 is provided with a voice recognition unit that converts the user's utterance content into text data. The input / output unit 31 corresponding to the voice recognition unit outputs text information converted from the user's utterance content to the search control unit 32.
[0175]
The synchronization data output unit 3 can also be configured to identify a speaker by inputting the voice of a person to be searched and output the person name to the search control unit 32.
[0176]
Furthermore, the synchronous data output unit 3 is a cut image similar to a desired scene, a similar camerawork moving image, a background image, a person image, a person operation moving image, or an object image. Any one of the moving image of the motion of the object, music, sound effect, and sound of the same silent section arrangement is input, each is recognized, and all or part of the recognition result is output to the search control unit 32. It can also be configured.
[0177]
In the present embodiment, the time code generation unit 16 outputs the generated time code to the image recognition unit 14 and the voice recognition unit 15, respectively, and the image recognition unit 14 and the voice recognition unit 15 respectively generate the generated image recognition character information. The time code is added to the voice recognition character information. However, the present invention is not limited to this, and the time code generator 16 may supply the generated time code to the structured processor 18 as in the content processing apparatus shown in FIG.
[0178]
In such a configuration, the image recognition unit 14 and the voice recognition unit 15 do not add a time code to the generated image recognition character information and voice recognition character information, respectively. Based on the correspondence between the character information and the voice recognition character information and each time code, the correspondence between each time code and the text information is determined.
[0179]
In addition, a program for operating the computer as all or part of the playback apparatus or executing the above-described processing is stored in a computer-readable recording medium such as a flexible disk, MD, CD-ROM, or DVD. May be distributed and installed in a computer and operated as the above-mentioned means, or the above-described steps may be executed.
[0180]
Furthermore, the program may be stored in a disk device or the like included in a server device on the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.
[0181]
【The invention's effect】
As described above, according to the present invention, content can be easily used.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a content processing apparatus according to a first embodiment of the present invention.
FIG. 2 is an explanatory diagram showing a relationship between pieces of information processed by the content processing apparatus of FIG. 1;
FIG. 3 is a block diagram showing a hardware configuration of the content processing apparatus shown in FIG. 1;
4 is a block diagram showing a configuration of a terminal shown in FIG. 3. FIG.
FIG. 5 is a flowchart showing an operation of the synchronous data generation unit of FIG. 1;
6 is a flowchart showing the operation of the synchronous data output unit of FIG.
7 is an explanatory diagram illustrating processing contents of a mapping unit in FIG. 1; FIG.
FIG. 8 is an explanatory diagram showing a search screen displayed on the display device shown in FIG. 4;
FIG. 9 is an explanatory diagram showing a search result screen displayed on the display device shown in FIG. 4;
10 is a block diagram illustrating a configuration of a data storage unit including a management file storage unit as an application example (1) of the data storage unit illustrated in FIG. 1;
11 is an explanatory diagram showing an application example of data structured by the structured processing unit of FIG. 1; FIG.
12 is a block diagram showing a configuration including only a video media storage unit and a text media storage unit as an application example (2) of the data storage unit shown in FIG. 1; FIG.
13 is an explanatory diagram showing a description example of an XML file processed by a synchronous data generation unit as an application example of the content processing apparatus shown in FIG. 1;
14 is a block diagram showing a configuration including an XML file storage unit as an application example (3) of the data storage unit shown in FIG. 1; FIG.
FIG. 15 is a flowchart showing an operation of generating an XML file by the synchronous data generation unit shown in FIG. 1;
16 is a block diagram showing a configuration to which the content processing apparatus shown in FIG. 1 is applied.
[Explanation of symbols]
1 Synchronization data generator
2 Data storage
3 Synchronous data output section
16 Time code generator
17 Mapping section
18 Structured processing unit
31 I / O section
32 Search control unit
33 Synchronization processor

Claims (10)

In the content processing apparatus for correspondence between the script information representing the plot of the content and the content including the video information and audio information in the character data,
A recognition unit that performs recognition processing of the content, and generates image recognition character information and voice recognition character information that represent characterizing data of each scene included in the content;
A time code generator for generating a time code indicating the said characteristic portion of the separator to extract each recognition unit generated image recognition character information and the voice recognition character information, separated by a start and end times of each part,
Acquires the characteristic portion of the script information, delimiting the script information for each scene based on the characteristic part acquired, the image recognition character information which the recognizer is generated and with said voice recognition character information, the time Dividing at the position indicated by the time code generated by the code generator, and even if the divided image recognition character information and the voice recognition character information do not match, each is determined to be correct, and each scene of the script information Associating the divided image recognition character information with the voice recognition character information, and generating correspondence information indicating a correspondence relationship between each scene of the script information and the divided image recognition character information and the voice recognition character information A mapping unit to
Based on the correspondence information generated by the mapping unit and the time code generated by the time code generation unit, structured information indicating the relationship between each scene of the script information and the time code of each scene is obtained as necessary content. A structured processing unit for generating information for searching,
A content processing apparatus. A screen predicting unit configured to predict the screen configuration of the script information, and adding the predicted screen configuration to the script information to output to the mapping unit;
The content processing apparatus according to claim 1. A data storage unit for storing the script information, the content, the time code generated by the time code generation unit, and the structured information generated by the structured processing unit;
The content processing apparatus according to claim 1, wherein the content processing apparatus is a content processing apparatus. The data storage unit
A content storage unit for storing the content;
A text file storage unit for storing the script information and the structured information;
A time code storage unit for storing the time code,
The content processing apparatus according to claim 3 . The structuring processing unit generates a script information file storing the script information and a time code file storing the time code in the text file storage unit, and a start address and an end address of each segment of the script information file And generating management information indicating the start address and end address of each time code in the time code file,
The data storage unit includes a management information storage unit that stores the management information.
The content processing apparatus according to claim 4 . The data storage unit
A markup language file storage unit for storing the script information as a markup language file;
The content processing apparatus according to claim 5. A synchronization data output unit that outputs the script information and the content stored in the data storage unit in synchronization with each other based on an input search condition;
Content processing apparatus according to any one of claims 3 to 6, characterized in that. The synchronous data output unit
An input / output unit that inputs search conditions for extracting necessary scenes from the script information and content, and outputs the script information and content of scenes corresponding to the search conditions;
A search control unit that identifies a scene in the script information corresponding to the search condition input to the input / output unit, and extracts a time code corresponding to the identified scene;
A synchronization processing unit that identifies the scene of the content corresponding to the extracted time code, and synchronizes the content of the identified scene and the script information corresponding to the search condition;
A synchronization processing unit that outputs content and script information corresponding to the scene synchronized by the synchronization processing unit to the input / output unit;
The content processing apparatus according to claim 7 . A content processing method for associating content including video information and audio information with script information in which a scenario of the content is expressed in character data,
Performing recognition processing of the content, and generating image recognition character information and voice recognition character information expressing the characterizing portion of each scene included in the content with character data;
Extracting and separating the characteristic portions of the generated image recognition character information and speech recognition character information, and generating a time code indicating a start time and an end time of each divided portion;
The feature information of the script information is acquired, the script information is divided for each scene based on the acquired feature portion, the generated image recognition character information and voice recognition character information are generated, and the generated time code is When the divided image recognition character information and the voice recognition character information do not match, it is determined that each is correct, and each scene of the script information and the image recognition character information divided Performing correspondence with the voice recognition character information and generating correspondence information indicating a correspondence relationship between the image recognition character information and the voice recognition character information divided from each scene of the script information;
Based on the generated correspondence information and the generated time code, structured information indicating the relationship between each scene of the script information and the time code of each scene is generated as information for searching for necessary content. Steps,
Storing the script information, content, time code, and structured information,
A content processing method characterized by the above. On the computer,
A procedure for performing recognition processing of video information and audio information included in the content, and generating image recognition character information and voice recognition character information in which the characterizing portion of each scene included in the content is represented by character data,
A procedure for extracting and separating the feature parts of the generated image recognition character information and voice recognition character information, and generating a time code indicating a start time and an end time of each divided part,
Acquiring a feature portion of the script information in which the scenario of the content is expressed by character data, dividing the script information into each scene based on the acquired feature portion, and generating the generated image recognition character information and the voice recognition character Information is divided at the position indicated by the generated time code, and even if the divided image recognition character information and the voice recognition character information do not match, each of the scenes of the script information is determined to be correct. Correspondence information indicating the correspondence between each scene of the script information and the divided image recognition character information and the voice recognition character information. Steps to generate,
Based on the generated correspondence information and the generated time code, structured information indicating the relationship between each scene of the script information and the time code of each scene is generated as information for searching for necessary content. procedure,
A procedure for storing the script information, content, time code, structured information;
A program for running

Images