JP2003244539A - Sequential automatic caption production processing system - Google Patents

Abstract

(57) [Summary] [Problem] To complete automatic caption production processing and preview / correction processing within a time close to the real time of a program. SOLUTION: In a designated section of input speech, the start and end timings of announce speech in sentence units are detected, and the detected timing is used as the start of a display unit subtitle sentence.
An automatic subtitle production unit 111 that adds subtitle timing information to be applied as at least a part of the end timing and generates subtitle data for each predetermined processing unit; and monitors at least video and audio of a television program and its subtitle data. Preview of subtitle data displayed on the device 155 and displayed on the monitor 155 when at least key operation of the preliminary preview is input from the key input device 159 and information on the operation timing and key type is recorded in the storage device 153. A correction support unit 151;

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sequential automatic subtitle production processing system capable of reducing production time.

[Summary of the Invention] The present invention relates to a sequential automatic subtitle production processing system for the purpose of reducing production time. The automatic subtitle production processing system sequentially detects, for example, an appropriate segment of program sound and sets it as a subtitle production processing unit, and executes subtitle production for each processing unit.
Speed up so that all automatic subtitle production can be completed within the time of this processing unit, and immediately proceed to the subtitle production processing of the next processing unit. After all, although the subtitle program material continuously played back from the VTR is delayed by the processing unit time, chasing automatic subtitle production becomes possible, and the subtitle production time can be greatly shortened. .

[0003]

2. Description of the Related Art With the remarkable progress of computerization in society, hearing impaired persons are in many situations restricted in obtaining information due to their functional impairment. As an effective means for hearing-impaired people to use and enjoy broadcasting as well as healthy people, captioned broadcasting is currently being implemented for some programs, but in light of the needs of hearing-impaired people, The implementation status is extremely insufficient.

However, since most of the production process of subtitle broadcasting depends on manual work at present, it takes a lot of labor, cost and time to produce a program, which is a factor that hinders the spread of subtitle broadcasting. It is one. In order to further spread subtitle broadcasting in the future, it is essential to streamline and streamline the program production process by developing subtitle program production technology that efficiently creates subtitle data.

Explaining the processing procedure in the conventional manual caption production system, first, as a caption creating material, a program tape in which a time code is superposed on a video, a program tape in which the time code is recorded in an audio channel, a program script, etc. are used.

[0006] This is requested to a person with specialized knowledge such as OB who has experience in broadcasting, and a summary transcription of the program announcement and an image of caption display (refer to a separately prepared caption manuscript, etc.) and its start Create a subtitle manuscript by entering the end time code.

An operator creates an electronic subtitle based on this subtitle original.

The digitized subtitles are previewed / corrected in the presence of the person in charge of subtitle production, the manuscript creator, the digitized operator, and the like to obtain the completed subtitles.

[0009]

However, in the conventional manual subtitle production system, in the production of subtitle originals, a program tape or a script in which a time code is multiplexed and recorded is used, and a line as a subtitle display unit is used. In addition to transcribing and subtitle display images, the time code on the screen is read and the start and end time codes are entered, but it depends on human intelligence and ability. You need ten times as much time.

[0010] Also, previewing and modifying are works that must be done manually, and are the final check as a program, so their importance lies in the advanced abilities of experienced experts. , Requires a lot of manpower and several times as long as the program time.

Normally, the preview / correction work involves restraining a plurality of valuable human resources in a highly tense state for a long time (for example, the number of people is 3, and the working time is 2.7 times the program time). In addition, in a typical example of preview / correction work, first, the subtitle program video / audio / timecode is continuously played, and the display unit subtitles corresponding to the timecode are sequentially displayed. Make a note of the subtitle numbers and possible outlines of the subtitles that are considered to be defective (tentatively called preliminary screening). Next, the defect status is examined in detail for each subtitle of the memo created in the preliminary preview, and the problem is corrected immediately on the spot, or a specific correction item of the defect is noted for separate batch correction.

[0012] In this preliminary preview, the work of taking note of the subtitle number of the subtitle that seems to be defective and the outline of the defect is 7-10 items related to the defect within the display time (3-6 seconds) of the subtitle of one page. I need a check and a summary note,
This is a fairly demanding task that must be repeated for the number of subtitle pages of a program (for example, 450 pages for a 45-minute program).

The present invention has been made in view of the above circumstances. For a program in which a subtitle text is present in advance, automatic subtitle production is performed, especially when the timing is added at high speed, and a preview / correction The objective is to provide a sequential automatic caption production system that can significantly reduce the time required for caption production by effectively supporting correction work.

[0014]

To achieve the above object, according to the present invention, in claim 1, at least a start of an announcement voice in sentence units is started in a designated section delimited by at least a pause of input voice. A detection unit for detecting the end timing at high speed, and a unit for adding information to the subtitle for applying the detected timing as at least a part of the page break and start / end timing of the display unit subtitle sentence, and processing of the designated section The feature is that caption data is automatically produced sequentially for each unit.

According to a second aspect of the present invention, at least a detection unit that detects at high speed the start and end timings of the announcement voice in sentence units in the designated section delimited by at least the pause of the input voice is displayed, and the detected timing is displayed. An automatic caption production unit that includes means for adding information to captions to be applied as at least part of page breaks and start / end timings of unit caption sentences, and an automatic caption production unit that creates caption data for each predetermined processing unit, and at least a video of a TV program For a monitor device that displays audio and subtitle data thereof and outputs program audio, and for subtitle data displayed on the monitor device, when there is a key input for preliminary preview from the key input device, at least the operation timing and A preview consisting of a caption correction information collection device that records information about key types in a storage device. A primary support unit is provided, and a preliminary preview is performed by the preview / correction support unit in parallel with the progress of the automatic subtitle production process executed by the automatic subtitle production unit, and the subtitle program production is sequentially processed. There is.

According to a third aspect of the present invention, in the sequential automatic caption production processing system according to the first or second aspect, the detecting means detects a pause section of the voice by a block cepstrum flux method to start and end the voice. It is characterized by detecting the timing.

[0017]

DESCRIPTION OF THE PREFERRED EMBODIMENTS <Description of Principle and Background of the Present Invention> Prior to the description of the embodiments, the principle of the present invention will be described.

Among the subtitle programs currently being broadcast, there are some programs in which an announcement manuscript is created in advance, and it is presumed that the manuscript becomes the actual broadcast subtitles with almost no modification. In these programs, the announcement sound and the subtitle content are almost the same, and it can be inferred that the almost common manuscript is used for both the announcement and the subtitle.

Therefore, the present inventors use a manuscript common to both the announcement and the subtitle, which is very similar in content of the announcement voice and the subtitle, and the manuscript is digitized. When a program is assumed, high-precision timing information corresponding to the division is automatically added to each of the display unit subtitles after the subtitle text is automatically divided at appropriate places according to a predetermined display format. The subtitle data is automatically created sequentially for each predetermined processing unit, and if necessary, the subtitle data for each processing unit that is created is also previewed and modified sequentially, greatly reducing the time to create subtitle programs. We have come up with the idea of a sequential automatic subtitle production processing system.

FIG. 2 is an explanatory view showing the principle of the present invention. As will be described later, an example of the sequential automatic subtitle production processing system according to the present invention includes an automatic subtitle production unit and a preview / correction support unit. For example, an appropriate division of program audio is used as a subtitle production processing unit. While performing automatic caption production for each processing unit, preview and correction processing of caption data is performed for each processing unit if necessary.

FIG. 2A shows a VTR reproduction output, and FIG.
Shows the processing in the automatic caption production unit, and (C) shows the processing in the preview / correction support unit. As shown in (B), the automatic caption production unit first executes the creation processing 1 divided into predetermined processing units. When the creation process 1 is executed,
As shown in (C), the preview / correction support unit sequentially executes preview / correction processing 1. Immediately after the automatic caption creation processing is completed, the preview / correction processing of the created caption data is executed in a pipeline manner. In this way, if the automatic subtitle production process can be completed within the processing unit time, it is possible to immediately proceed to the next processing unit subtitle production process, and the preview / correction process can also be executed within the processing unit time. Although, after all, it means that it is possible to perform chasing automatic subtitle production and preview / correction processing for the subtitle program material continuously played back from the VTR by the processing unit time, but the production time from the start of the VTR playback The work up to here can be completed in + processing unit time x 2.

The elements of the sequential subtitle production processing are (1) high-speed processing capable of completing all automatic subtitle production during the processing unit time, (2) setting of an appropriate subtitle processing unit, and (3) within the set time. It is an appropriate preview / correction process that can be completed.

(1) Regarding high-speed processing, the current automatic caption production system requires a processing time of three times or more, so that it is impossible as it is. The synchronization system takes most of the time in the processing system of the automatic caption production system, and it is necessary to speed up this part or apply another high speed method.

As the method, high-speed processing is enabled by performing timing detection using features such as voice level represented by the block cepstrum flux method.
In the case of a program for which the timing assignment is completed only by this method, it is possible to perform high-speed processing within a fraction of the program audio time.

(2) Regarding the setting of an appropriate subtitle production processing unit, basically, it is set so as not to affect the generation of the display unit subtitle. Specifically, it is appropriate to divide the program sound by a non-speech portion of the program sound for a certain time or longer. However, this method alone has the drawback that the breaks sometimes become significantly longer. As a simple measure in this case, if a break is set to a time corresponding to 10 pages of display unit subtitles that is empirically standard, there is almost no effect.

(3) As a result of executing an appropriate preliminary preview in the preview / correction support unit in parallel with the progress of the automatic subtitle production processing executed in the automatic subtitle production unit, the production of these subtitle programs is performed. By sequentially processing the whole, the subtitle program production has been speeded up.

<Description of Embodiments> FIG. 1 is a block diagram showing a configuration example of a sequential automatic subtitle production processing system according to the present invention.

In the following description, the entire set of subtitle sentences to be displayed is referred to as "subtitle sentence text", and a sub-set of a set of subtitle sentences that is appropriately separated as subtitles from the subtitle sentence text is referred to as "subtitle sentence text". "Subunit text" is the subtitle that is the display unit on the display screen of the display is "display unit subtitle". When expressing individual subtitles of each line included in the display unit subtitle, this is referred to as "display unit subtitle line". When expressing an arbitrary character in the display unit subtitle line, this will be referred to as a “subtitle character”.

As shown in the figure, the sequential automatic subtitle program production system 101 includes an automatic subtitle production section 111 and a preview / correction support section 151. Automatic subtitle production department 1
Reference numeral 11 denotes an electronic document recording medium 113 and the synchronization detection device 1.
15, an integration device 117, a morphological analysis unit 119,
The division rule storage unit 121 and the program material VTR, for example, a digital video tape recorder (hereinafter referred to as "D-VT").
(Referred to as “R”) 123. The preview / correction support unit 151 also includes a storage device 153, a monitor device 155, a delay device 157, and a keyboard 1.
And 59.

<< Structure and Operation of Automatic Subtitle Production Unit 111 >> The electronic document recording medium 113 is composed of, for example, a hard disk storage device, a floppy (registered trademark) disk device, or the like, and represents a whole set of subtitles to be displayed. I remember the sentence text. Note that, in the present embodiment, it is assumed that almost common digitized manuscripts are used for both announcements and subtitles. Therefore, the content of the subtitle sentence text stored in the digitized manuscript recording medium 113 is displayed. Not only does it match the subtitles you want, but it also matches the announcement sound recorded on the material VTR.

The synchronization detection device 115 has a function of detecting the time synchronization between the synchronization detection pointed subtitle sentence and the announcement voice that has read the sentence. In this case, the synchronization detection points usually specify the beginning and end of each sentence of subtitle sentence text. More specifically, the synchronization detection device 115
The subtitle sentence with synchronization detection point added by the integration device 117, the announcement voice corresponding to the subtitle sentence captured from the program material VTR, and the time code thereof are input to the pause point included in the announcement voice. There is a function to detect and verify the accuracy, and a function to detect the timing information of the unsynchronized specified sync detection point, that is, the time code by the voice recognition processing method. The time code and pause section data detected by these functions are integrated into the integrated device. It has a function of sending to 117.

Note that the synchronization detection between the announcement voice and the subtitle sentence text including the voice recognition processing for the announcement voice can be realized with high accuracy at a low speed by applying the technique already researched and developed by the present inventors. Is.

The pause time detecting function in the embodiment does not perform the voice recognition processing as described above, but the material V
The block cepstrum flux method, which will be described later, is used to detect, for example, a start and end time code in which the level is equal to or less than a designated level and continues for a predetermined time, from the level and duration of the sound supplied from TR and the time code. It is executed by a method such as.

The uniting device 117 extracts unit caption sentences which sequentially extract unit caption sentences based on the punctuation starting from the beginning of the sentence and the required number of characters range from the caption sentence text read from the computerized manuscript recording medium 113. Function and unit subtitle sentence extraction function to convert the unit subtitle sentence extracted to display unit subtitle according to the desired display format and display unit subtitle conversion function For the display unit subtitles, a timing information adding function is provided that uses the time code and the pause point sent from the synchronization detecting device 115 and adds the timing information by an appropriate interpolation process.

The morpheme analysis unit 119 expresses, for each morpheme that is divided by demonstrating the division function, by dividing the morpheme into unit subtitle sentences written in kanji and kana mixed sentences. It has an additional information addition function that adds additional information such as shape, part of speech, reading, standard expression, etc., and an information element sequence acquisition function that groups each morpheme into phrases or clause units and obtains several information element sequences. ing. As a result, the unit subtitle sentence is expressed as a surface element sequence, a symbol element sequence (part of speech sequence), a standard element sequence, and an information element sequence.

The division rule storage unit 121 has a function of storing a division rule that is referred to when optimizing a line feed / page break portion for display unit subtitle conversion for a unit subtitle sentence.

The D-VTR 123 has a function of reproducing and outputting video, audio, and their time codes from the program material VTR tape in which program materials are recorded.

Next, the internal structure of the integration device 117 that plays a major role in the automatic caption production unit 111 will be described.

The integrating device 117 includes the unit caption sentence extraction unit 1
33, a display unit captioning unit 135, and a timing information adding unit 137.

The unit subtitle sentence extraction unit 133 uses, for example, a punctuation mark or about 70 to 90 subtitle characters as a guideline among the subtitle sentence texts in which the unit subtitle sentences are arranged in the display time order, which is read from the electronic document recording medium 113. , And has a function of sequentially extracting text sentences as processing units by utilizing the added delimitable part information and the like. Note that the morpheme analysis data with phrase data obtained by the morpheme analysis unit 119 and the division rule (line feed / page break data) stored in the division rule storage unit 121 can be used as the delimitable part information. . Here, the above-mentioned division rule (line feed / page break data) is described. The recommended line feed / page feed point defined by the division rule (line feed / page break data) is first after the punctuation and secondly at the reading point. After, the third is between bunsetsu, the fourth is between morphological parts of speech,
When applying the division rule (line feed / page break data), it is preferable to apply the division rule preferentially from the beginning of the description order.

The display unit subtitle conversion unit 135, based on the unit subtitle sentence extracted by the unit subtitle sentence extraction unit 133, the delimitable part information added to the unit subtitle sentence, the information from the synchronization detection device 115, etc. The timing information addition unit 137 has a display unit subtitle conversion function of converting the unit subtitle sentence extracted by the sentence extraction unit 133 into at least one display unit subtitle according to a desired display format, and the timing information addition unit 137 includes a display unit subtitle conversion unit 135.
With respect to the display unit subtitles converted by
Using the time code as pause information and synchronization detection point information for each sentence of the subtitle sentence text sent from
Further, it has a timing information adding function for adding timing information using an appropriate timing interpolation method.

Next, an example of a method for adding timing information to subtitles, which is executed by the sequential automatic subtitle production processing system according to the present invention, will be described with reference to FIGS. 3 to 7.

As described above, the synchronization detection of the timing information regarding the subtitle corresponding to the announcement voice is performed between the announcement voice and the subtitle sentence text including the voice recognition processing for the announcement voice which has been already researched and developed by the present inventors. Although it can be realized with high accuracy by applying the synchronization detection technology of No. 1, this synchronization detection processing is considerably complicated as described above and requires a lot of processing time. For this reason, if the synchronization detection technology is applied to all the start points / end points of each display unit subtitle, it takes a very long time to create a subtitle program, including an excessive number of synchronization detection points. Processing is impossible.

Further, when the subtitle text is divided into display unit subtitle sentences of which the number of lines and the number of characters are suitable for subtitle display, it is not preferable to divide the subtitle over a long announcement pause (narration gap). However, in the case of extremely short poses, it is preferable to treat them as continuous sentences, so
It is necessary to consider the pause time of the announcement for subtitle division. Regarding this point, for example, by applying the block cepstrum flux method and the like, by skillfully processing features such as the voice level and its duration, for example, the start of an announcement voice in sentence units to subtitle sentence text is started. It is possible to detect the end timing and the inter-sentence pause time associated therewith as pause data. Moreover, these processes can be performed at high speed in a time period which is a fraction of the program audio time or less.

However, this method has a factor that hinders the correct timing detection, such as when a voice other than the announce voice is mixed in the voice, so the detection result must be verified and only the highly accurate timing should be used. I won't. However, if you apply an appropriate method, a considerable part of the start and end timings in each subtitle text will be
It is possible to apply the timing detected using the voice level or the like. And it applies to the start and end timings of display unit subtitle sentences,
It is given by an appropriate timing interpolation method described later. For the part where the required result cannot be obtained by the timing detection using the voice level or the like, the conventional matching method utilizing the voice processing technology of the subtitle sentence text and the announcement voice is applied.

As an example of the interpolation method, as shown in the flowchart of FIG. 3, when the subtitle sentence text is used as it is as a display unit subtitle sentence without being summarized, first, the pause start timing and the duration are obtained from the audio data. The suitability is verified and selected (step ST1).

Next, the subtitle text is divided into blocks with a relatively long pause (for example, 2 seconds or more), and the start and end timings thereof are given as block text sentences (steps ST2, 3).

Next, the duration of the block text sentence is divided by the total number of readings (calculation estimation) to obtain the average reading speed in the range (step ST4).

Next, each block text sentence is divided into display unit subtitle sentences in which the pause portion of the middle length serves as a line feed point. In this case, the division rule is applied (step ST
5).

Then, the timings corresponding to the beginning and end of each divided subtitle sentence are calculated and added based on the start and end timings of the block text sentence, the pause timing, and the average reading speed. In this case, the number of characters, the character type method, or the pronunciation method is applied (step ST6), the display time of each display unit subtitle is checked, and the end timing is corrected if necessary (step ST7).

Next, FIG. 4 shows a processing procedure for adding timing information when the above-mentioned summary processing shown in FIG. 3 is not performed.
Through the specific example shown in FIGS.

FIG. 4 is a subtitle original text in an example of timing addition to display unit subtitles utilizing the pause information by voice pause detection, and FIG. 5 is a kana number of each subtitle original text shown in FIG. The number of kanji, reading, time, pause, start time of text, stop time of text, start time of next text sentence, and reading speed are shown. Further, FIG. 6 shows display unit subtitle sentences (the left side of / is the first line and the right side of / is the second line) created from the subtitle original text shown in FIG. 4 and its timing information.

As can be understood from FIG. 5, in the subtitle original text "No. 1 in FIG. 4"Today's stage is the Great Plains of Kenya in East Africa ", the number of kana is" 12 "and the number of kanji is" 8 ". , The reading number (yomi) is “25.02”, the reading time is “3850 mS”, and the pause time is “101”.
0 ms ”, start time is 49150 ms (49.1
50S), stop time is "53000mS (= 5
3.000S), next start time is "54010mS
(54.010S) ", the speech speed is 15.39 mS (= 3
85 / 25.02). Here, "yomi" is
Since the kanji part (and the number part) can be represented in about 1.86 times the reading time of kana, 12 + 7 × 1.
It is calculated as 86 = 25.02.

The subtitle text is divided into blocks with a relatively long pose (for example, 2 seconds or more) in the pose information thus obtained, and the start and end timings thereof are added as block text sentences to create ( The display unit caption text created in the processing of steps ST1 to ST5 of FIG. 3 is shown in FIG. In FIG. 6, the number surrounded by a thick solid line is the time obtained by calculation, and it is shown that the designation as a synchronization detection point that may require a long processing time can be significantly reduced. Further, the delimiter shown as “block” at the right end has a pause of about 2 seconds or more between the blocks, and the subtitle processing unit is set on the basis of this. In other words, line breaks, page breaks, and timing processing for subtitles are breaks that can be completed within each processing unit, and timing addition processing and the like are completed within the time of the block (according to the speedup of the present invention). If it becomes possible, the automatic subtitle data creation will be terminated after the block time has elapsed, and this part will be ready for preliminary preview, and the successive subtitle production / testing method will be applicable. The time required for this method is almost the same as the program time, and when previewing after creating all the subtitle data, it is twice or more the time, so there is a great effect that the time can be greatly shortened.

<< Detection of Voice Pause Section by Block Cepstrum Flux Method >> In the present invention, as an example of a method of detecting a voice pause section, for example, a block cepstrum flux method using voice level information is used. Was performed using. Block cepstrum
The flux method is a method of more stably detecting the switching point of the acoustic data content by comparing a plurality of LPC cepstrum vectors in the acoustic data with each other from the reference frame.

FIG. 7 shows the result of analysis of the sound of an actual TV program (hand dolphin) using the block cepstrum flux method. Although there are actually left and right channel sounds, FIG. 7 shows only the right channel results. Further, in FIG. 7, the bar graph shows the actually investigated voice (speech) section. Set the analysis waveform to an appropriate level (eg, 0.0 in FIG. 7).
When sliced at 55) and comparing the above range as a speech (speech) section, it can be seen that at a predetermined duration or longer, it is in good agreement with the speech (speech) section shown by the bar graph. On the other hand, the part indicated by "↑" indicates the pause section, which also agrees considerably.

<< Improvement of Pose Detection Method and Verification Method of Detected Pose >> As an improvement method of the pose detection method and a verification method of the detected pose, the following method is applied.

For example, as a method of pause detection that makes use of the main features of the announcement voice with respect to other background sounds, first, a band-limited voice is formed from the input voice, and then the voice level of the designated section. Is standardized (normalized by high-level voice in the specified section). Next, the power value of the voice is obtained and its integration processing is performed (the window function is associated with band limitation).

As a method of verifying the pose, first,
Set the threshold of voice power (ex. 1/4, 1/9, 1/16 with respect to the highest level). Then, the pose at each of the set thresholds is obtained (P1, P2, P3 in order). Next, each pose accuracy is calculated. Next, the pause is checked by the duration (valid for a certain time or more), and the timing correlation with the punctuation marks and the reading marks of the subtitle text is checked. In this check, if it is within a certain time, it is treated as valid.

In this way, more accurate timing can be given by improving the method of detecting the pose and verifying the detected pose for the detected pose. There is also an improved method as a pause detection method, for example, by creating approximate speech data and using it to easily grasp the speech segment, the start and end timing of speech can be grasped. Then, the pose is detected. Next, this pose detection will be described with reference to FIGS.

FIG. 8 is an example in which a voice data waveform 51 is displayed as the speech approximate data.

The horizontal axis is a timeline showing the passage of time of the program, and when the sound is reproduced, the cursor is displayed at a position corresponding to this elapsed time and moves with the passage of time. Therefore, the reproduced voice and the voice waveform at each position of the cursor can be associated with each other.

The speech timing can be understood to some extent from the sound waveform data depending on the background sound in the sound being sufficiently small or the experience of the waveform, but in the normal program sound, there are various background sounds and their levels are also different. Because of the variety, it is generally difficult to accurately grasp the start and end timings of speech from this voice waveform data.

Here, by using the speech approximation data in which the speech component is emphasized, it is possible to increase the accuracy of grasping the taming.

FIG. 9 shows an example in which speech approximation data obtained by specially processing voice data is used. In FIG. 9, the waveform 6
1 is a cflx analysis value of the voice (by the block cepstrum flux method), a waveform 62 is a specific frequency range (for example, 4 to 7 Hz) component extraction value of the voice power value, and a waveform 63 is a slice of the waveform 62 at an appropriate level. This is binarized data.

In the waveform 63, the high level range represents a speech section and the low level range represents a non-speech (pause) section. In this example, almost coincides with the actually measured timing, but the waveform 62 is more accurate. high. Therefore,
From the waveform 63, it is possible to grasp the start and end timings of the speech in the voice with some accuracy.

As described above, by using the speech approximation data obtained by specially processing the voice data as a guideline for designating the speech section, it is possible to use it as a pose with higher accuracy.

<< Structure and Operation of Preview / Correction Support Unit 151 >>
As shown in FIG. 1, the preview / correction support unit 151 includes a storage device 153, a monitor device 155, and a delay device 157.
And a keyboard 159.

The storage device 153 has an automatic caption production section 111.
In addition to storing the subtitle data created in step 1, the correction data found on the monitor device 155 is stored.

The monitor device 155 is the automatic subtitle production unit 1.
11 and at least the subtitle data and the delay device 1
Upon receiving the video and audio output from 57, the video and subtitles are displayed on the monitor screen and the audio is output. That is, in the monitor device 155, the delay device 1 corresponding to the processing unit time is used.
The subtitles that have been automatically produced are displayed together with the video / audio via 57, and in parallel with the successive subtitle production, a preliminary preview within a range that can be performed in real time can be performed on the produced subtitles. Further, the monitor device 155 is not a simple monitor, but an input from the keyboard 159, a storage device 1
It has the input / output of 53 and a simple signal processing function.

The delay device 157 uses the program material VTR123.
The video, audio, and time code, if necessary, can be delayed by at least the time of the subtitle production processing unit, and the delay time can be variably operated.
The output of the delay device 157 is given to the monitor device 155.

Here, as the "processing unit", for example,
It is possible to set a subtitle production processing unit time as a delimiter of subtitle processing in consideration of the existence period of a relatively long non-speech section (for example, 3 seconds or more). As a result, the program material VTR1
In response to the sound continuously reproduced from No. 23, caption data is sequentially produced for each processing unit time. The caption data generated by the automatic caption production unit 111 is given to the monitor device 155.

In order to support the work at the time of this preliminary preview and enable checking with as much content as possible in real time, the monitor device 155 uses the keyboard 159 to input the subtitle text displayed on the monitor screen. When there is, a process of recording the correction data including the type of the key operated for caption correction and the caption page number or time code at that time in the storage device 153 is performed.

The monitor device 155 is the storage device 1.
The contents recorded in 53 are totaled for each type of key or each timing information, and the totalized result is stored in the storage device 153 so that it can be displayed as a list on the monitor screen. Then, upon receiving a command from the keyboard 159 during the subtitle correction work, a process of reading the list data from the storage device 153 and displaying it on the monitor screen is executed.

The keyboard 159 is provided with various keys to which various functions necessary for preview / correction processing are assigned, as shown in FIG. 10, in order to input various kinds of correction data.

<Operation of Preview / Correction Unit 151> Next, referring to FIG.
A subtitle correction information collection function and the like for supporting the preview / correction will be described with reference to FIGS. Note that FIG. 10 is a diagram showing a specific example of key function addition. FIG. 11 shows an example of the structure of a key operation recording file. 12 is the same as FIG.
It is a figure which shows the specific example of the correction press key total information 161 shown in FIG. FIG. 12 shows the modification press key detailed information 1 shown in FIG.
It is a figure which shows the specific example of 63. FIG. 13 is a diagram showing an example of contents of recording files displayed in a list at the time of correction work.

The recorded contents will be described. As shown in FIG. 11, the content of the correction press key is the correction press key total information 16
1 and the correction press key detailed information 163 are recorded separately.

In FIG. 10, "↑ key" is a key for changing the subtitle position upward. "↓ key" is a key for changing the subtitle position downward. "Page Down
The "key" is a key indicating that the display timing is shifted forward. “Page Up key” is a key indicating that the display timing is shifted backward. "End key" is a key indicating that the display timing is normal. "F1
"key" is a key indicating that the content of the subtitle text is abnormal.

Therefore, the person in charge of preliminary preview refers to the video / audio of the program and operates the keyboard as follows while watching the automatically produced subtitles displayed on the monitor screen to specify the defective subtitles. It is possible to point out and record the rough contents of the defect (or the contents of the correction).

Specifically, (1) When it is desired to correct the subtitle position upward, the "↑" key is pressed. For example, to avoid interfering with open subtitles. (2) If the "↑" key is pressed too much, press the "↓" key. (3) Press the "Page Down" key when the display timing is shifted to the front. (4) Similarly, "PageUp" at the back
Press the key. (5) Press the "End" key when OK. (6) Press the "F1" key when the content of the subtitles is not good. (7) In addition, a memo using the word processing function can be attached.

Therefore, the modification press key detailed information 163.
The file of "correction key pressing time code" shown in Fig. 13 is displayed for each pressing of the correction key, which is the indication of the problem.
"Corrected key name""Subtitlesentence""Start time code"
The "end time code" is recorded.

The "correction key press time code" is
An hour, minute, second frame from the beginning is indicated by 2 bytes each. The "correction press key name" shows the key name of the item in FIG. The "subtitle sentence" shows the subtitle sentence displayed on the monitor when the correction key is pressed. The “start time code” and “end time code” are added when the caption data is created, and the hour, minute, and second frames are indicated by 2 bytes each from the beginning.

When the person in charge of preliminary preview performs the above operation to the end of the program, the storage device 153 stores the correction press key detailed information 163 in FIG. The information shown in is automatically recorded.

When the preliminary preview is completed, the list data displayed as a list when the caption is corrected is accumulated in the recording file of the storage device 153. As shown in FIG. 14, the list data includes “correction key total information” and “correction key detailed information”.

As shown in FIG. 12, the file of the correction press key total information 161 is composed of columns of "item" and "contents description and total value". In the "Item" column, "↑ ke
y total ”,“ ↓ key total ”,“ PageDownkey total ”,“ Pag
"eUpkey total", "Endkey total", "F1 key total", "Other key total" are recorded respectively. In the "contents description and total value" field, the content description and the total value for the corresponding key name in the item field are recorded.

The "correction key total information" is, for example, "F1key:
Subtitle sentence content error: 3 "" pageDownkey: Deviation before display timing: 1 "" ↑ key: Change subtitle position up: 0 "
"↓ key: Change subtitle position downward: 0""pageUpkey: Deviation before display timing: 0""Endkry: Display timing normal: 0""Others:0""Correction total coefficient: 4".

The "correction key detailed information" is "0000000".
5, pageDown, 00000008, F1, in Hida region of Gifu prefecture, 0000000
1,00000009, Awa Tunnel directly connected is now 00000020,0000
0029 ”and the like. 00000005 and 00000008 are two digits each and represent hour, minute, second, and frame.
1,00000009 and 00000020,00000029 are time codes.

Then, the contents of the recording file as shown in FIG. 14 are displayed on the monitor screen in response to a command from the keyboard 21 when the caption is corrected. The caption correction operator can display the “correction key total information” in the list display on the monitor screen.
You can grasp the whole state of the correction contents from,
From "y detailed information", it is possible to accurately grasp the part to be corrected for each subtitle sentence. Therefore, it is possible to effectively support the subtitle correction work in the preview / correction.

[0089]

As described above, according to the present invention, the speedup and improvement are performed so that each subtitle creation process is sufficiently completed within the program time, and the subtitle data is sequentially generated for each set subtitle production processing time. I made it possible to produce. As a result, it is possible to create subtitles one after another, and the automatic subtitle production is completed in real time from the start of the automatic production work, and after the program real time + maximum processing unit time, automatic subtitle production and acquisition of correction support data by preliminary preview Can be completed, the burden on the operator is small, and the working time can be significantly reduced.

Further, not only by the voice recognition, but mainly by using the features such as the voice level, for example, by detecting the start and end timings of the announcement voice in sentence units,
By detecting the pause section where the announcer is not speaking and applying the timing as at least a part of the start and end timing of the display unit subtitle sentence, the dependency on the voice recognition method is reduced and the timing information is automatically added. Can be speeded up.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of a sequential automatic caption production processing system according to the present invention.

FIG. 2 is an explanatory diagram showing the principle of a sequential automatic caption production processing system according to the present invention.

FIG. 3 is a flowchart showing a processing procedure for adding timing to a displayed caption text when the summary processing is not performed.

FIG. 4 is an explanatory diagram showing an original text for a subtitle in a process of timing addition in a case where a summary process is not performed.

5 is an explanatory diagram showing a processing result for the subtitle original text shown in FIG. 4. FIG.

FIG. 6 is an explanatory diagram showing a display unit subtitle sentence created from the subtitle original text shown in FIG. 4;

FIG. 7: Speech (speech) section using the block cepstrum flux method (pause section if the viewpoint is changed)
It is explanatory drawing which shows the result of having detected.

FIG. 8 is an explanatory diagram showing a voice data waveform as speech approximate data.

FIG. 9 is an explanatory diagram showing speech approximation data obtained by specially processing voice data.

FIG. 10 is a diagram showing a specific example of key function addition.

FIG. 11 is a configuration example of a key operation recording file.

FIG. 12 is a diagram showing a specific example of total information of modified pressed keys shown in FIG. 11.

FIG. 13 is a diagram showing a specific example of correction press key detailed information shown in FIG. 11.

FIG. 14 is a diagram showing the contents of recording files displayed as a list at the time of preview / correction.

[Explanation of symbols]

101 Automatic Subtitle Production Processing System 111 Automatic Subtitle Production Unit 113 Digitized Manuscript Recording Medium 115 Sync Detection Device 117 Integration Device 119 Morphological Analysis Unit 121 Division Rule Storage Unit 123 Digital Video Tape Recorder (D-
VTR) 133 Unit subtitle sentence extraction unit 135 Display unit subtitle conversion unit 137 Timing information addition unit 151 Preview / correction support unit 153 Storage device 155 Monitor device 157 Delay device 159 Keyboard

   ─────────────────────────────────────────────────── ─── Continued front page    (71) Applicant 000004352             Japan Broadcasting Corporation             2-2-1 Jinnan, Shibuya-ku, Tokyo (72) Inventor Eiji Sawamura             2-31-19 Shiba, Minato-ku, Tokyo Communications and Broadcasting Organization             Within (72) Inventor Takao Kadoma             2-31-19 Shiba, Minato-ku, Tokyo Communications and Broadcasting Organization             Within (72) Inventor Noriyoshi Uraya             2-31-19 Shiba, Minato-ku, Tokyo Communications and Broadcasting Organization             Within (72) Inventor Katsuhiko Shirai             2-31-19 Shiba, Minato-ku, Tokyo Communications and Broadcasting Organization             Within F-term (reference) 5C023 AA18 BA16 CA01 CA05                 5D015 CC11 CC14 DD03 FF01 FF04

Claims (3)

[Claims] 1. A detection unit that detects at high speed the start and end timings of at least an announcement voice in sentence units in at least a designated section of the input voice with a pause as a delimiter, and the detected timing is a display unit subtitle sentence. Sequential automatic subtitle production processing characterized by including means for adding information to subtitles applied as at least part of page breaks, start and end timings, and automatically producing subtitle data for each processing unit in a specified section. system. 2. A detection unit that detects at high speed the start and end timings of the announcement voice in sentence units at least in a designated section where at least a pause of the input voice is used as a delimiter, and the detected timing is a display unit subtitle sentence. An automatic caption production unit that includes means for adding information to captions applied as at least part of the page breaks and start / end timings, and creates caption data for each predetermined processing unit, and at least video, audio, and Regarding the monitor device that displays the caption data and outputs the program sound, and the caption data displayed on the monitor device, at least when the key input device performs a preliminary preview key input, at least the operation timing and the key type A preview / correction support unit including a caption correction information collection device for recording information in a storage device And a subtitle program production is sequentially processed by executing a preliminary preview by the preview / correction support unit in parallel with the progress of the automatic subtitle production process executed by the automatic subtitle production unit. Production processing system. 3. The sequential automatic caption production processing system according to claim 1, wherein the detection unit detects a pause period of voice by a block cepstrum flux method to detect timing of start and end of voice. Sequential automatic subtitle production processing system characterized by: