KR20200124456A - a broadcast closed caption generating system - Google Patents

Abstract

Disclosed is a real-time system for producing broadcast subtitle for a hearing-impaired person. According to one embodiment of the present invention, the system for producing broadcast subtitle comprises: a speech recognition unit receiving a speech signal from content, recognizing the speech signal, and covering the speech signal into a character; a shorthand input unit acquiring subtitle shorthand input through a shorthand input device; and an integrated subtitle processing unit producing the final subtitle by integrating speech-to-text conversion data acquired from the speech recognition unit and shorthand input data acquired from the shorthand input unit.

Description

Broadcast closed caption generating system {a broadcast closed caption generating system}

The present invention relates to a broadcast caption production system. Specifically, the present invention relates to a broadcast caption production system using both automatic voice recognition technology and shorthand input.

The dictionary meaning of shorthand refers to any activity in which intangible voice language, video or audio are recorded quickly and accurately and converted into text. Shorthand is a kind of handwriting in terms of characterizing the spoken language, so it is essentially different from the recording for preserving the spoken language.

Such shorthand has evolved from handwritten essay shorthand to typewriter shorthand, and computer-based shorthand for inputting content on a computer keyboard, and became possible with the development of software and hardware even for parts that were difficult for humans to write.

In real-time broadcasting for the hearing impaired, in the past, subtitles for broadcasting contents were generated by relying on shorthand, but in recent years, with the development of machine learning, voice-to-text conversion tools have become popular, and captions can be generated in real time without relying on professional stenographers. Became.

However, in the case of using a voice-to-text conversion tool, it is possible to generate subtitles for broadcasting more easily than conventional methods that rely on the input of a stenographer, but sometimes the subtitles are not generated when voice recognition is not accurate or voice recognition is not performed. There may be a problem. Hereinafter, a caption generation system capable of complementing each of the problems by using both voice recognition and shorthand input will be described.

An object of the present invention is to propose a system for generating real-time captions for the hearing impaired by solving shorthand input for a problem that may occur when captions are generated only by voice recognition.

A broadcast caption production system according to an embodiment of the present invention includes a voice recognition unit for receiving a voice signal from content and converting the voice signal to a text, a shorthand input unit for obtaining a shorthand input through a shorthand input device, and the voice recognition. And an integrated caption processing unit for generating a final caption by integrating the voice-to-text conversion data obtained from the unit and the shorthand input data obtained from the shorthand input unit.

The broadcast caption production system according to an embodiment of the present invention can accurately compensate for inaccurate captions that may occur in generating captions through voice recognition through shorthand input.

In addition, since the broadcast caption production system according to an embodiment of the present invention generates captions based on voice recognition, it is possible to produce more broadcast captions for the hearing impaired while using a minimum of shorthand manpower.

1 is an overall configuration diagram of a broadcast caption production system according to an embodiment of the present invention.
2 shows the configuration of a voice recognition unit according to an embodiment of the present invention.
3 is a block diagram showing the configuration of a shorthand input unit according to an embodiment of the present invention.
4 is a flowchart illustrating an operation of a caption generation system according to an embodiment of the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the following embodiments, and those skilled in the art who understand the spirit of the present invention can easily add, change, delete, and add components to other embodiments included within the scope of the same idea. It may be suggested, but it will be said that this is also included within the scope of the inventive concept.

In the accompanying drawings, in explaining the overall structure in order to easily understand the spirit of the invention, minute parts may not be specifically expressed, and when describing the minute parts, the overall structure may not be specifically reflected. In addition, even if specific parts such as the installation location are different, if the action is the same, the same name is given, so that the convenience of understanding can be improved. In addition, when there are a plurality of identical configurations, only one configuration will be described, and the same description will be applied to other configurations, and the description will be omitted.

1 is an overall configuration diagram of a broadcast caption production system according to an embodiment of the present invention.

The broadcast caption production system disclosed in the present invention is not a general broadcast caption, but a real-time caption generation system for the hearing impaired.

As shown in Fig. 1, the content provider 20 delivers broadcast content to the broadcast caption production system 10, and the broadcast caption production system 10 generates broadcast captions based on the broadcast content and generates a broadcast caption based on the broadcast content, and the set-top box ( 30). The set-top box 30 encodes the received broadcast caption and outputs it together with the broadcast content.

The content provider 20 may be a broadcasting station as a representative example, and may include a business operator or an intermediate distributor that produces and provides content other than the broadcasting station. The content provider 20 delivers content including audio and video to the broadcast caption production system 10.

The broadcast caption production system 10 generates broadcast captions based on the content received from the content provider 20. Specifically, the broadcast caption production system 10 may include a voice recognition unit 100, a shorthand input unit 200, and an integrated subtitle processing unit 300.

When the content delivered from the content provider 20 is played, the voice recognition unit 100 automatically recognizes the voice and converts it into text. The voice recognition unit 100 may be a commonly used voice-to-text conversion tool, and a specific example may be Google Cloud Speech API or Amazon Transcirbe. A detailed configuration of the voice recognition unit 100 will be described below.

2 shows the configuration of the voice recognition unit 100 according to an embodiment of the present invention.

As shown in FIG. 2, the voice recognition unit 100 according to an embodiment of the present invention may include a voice receiver 110, a voice-to-text conversion unit 120, and an accuracy calculation unit 130.

The voice receiver 110 obtains a voice signal from the content. For example, the voice receiver 110 may be a microphone. The voice receiver 110 may collect all voice signals transmitted from the content, convert the collected voice signals into digital signals, and transmit them to the voice-text converter 120.

In another embodiment, the voice receiver 110 may receive the voice of the shorthand. When it is difficult for the stenographer to input shorthand through the shorthand keyboard in a specific situation, the voice receiver 110 may receive the voice of the stenographer and convert it into text. However, data obtained by converting the voice of the short story into text may be processed differently from the conversion of the voice signal of the content into text and transmitted to the integrated caption processing unit 300. The integrated caption processing unit 300 may treat the text data converted from the voice of the shorthand in the same way as the supplementary shorthand input and use it for final caption generation.

The voice-to-text conversion unit 120 converts the voice signal received from the voice receiving unit 110 into text. Specifically, the speech-to-text conversion unit 120 may be a machine learning application for automatic speech recognition through deep learning. The voice-to-text conversion unit 120 may transcribe an audio file stored in a general format such as WAV and MP3 and add a time stamp for each word.

The accuracy calculator 130 may calculate the accuracy of speech recognition between automatic speech recognition. Specifically, the accuracy calculation unit 130 may distinguish a human voice (raising) and noise from the voice signal, and calculate the accuracy based on the size of the human voice, the ratio between the human voice and noise, or the voice recognition result. I can.

In an embodiment, the accuracy calculator 130 may consider that the accuracy is low when the human voice is small among the voice signals, and the accuracy is high when the human voice is large. In other words, the accuracy calculation unit 130 may calculate the accuracy in proportion to the loudness of a person's voice. For example, there may be a case where a speaker in the content speaks away from the microphone or the speech sound is relatively low. The criterion for determining whether the human voice is large or small may be the size of the human voice in general content, and a specific value may be obtained through machine learning.

In another embodiment, the accuracy calculation unit 130 may be considered to have a lower accuracy as the noise ratio increases in the ratio between human voice and noise among voice signals. In other words, the accuracy calculator 130 may calculate the accuracy in inverse proportion to the noise ratio. For example, in the content, there may be a case where the interior of the hall is noisy or non-voiced sound is the center.

In another embodiment, the accuracy calculator 130 may calculate the accuracy based on the recognition result. The accuracy calculation unit 130 may calculate the accuracy by recognizing the voice and determining whether a result of converting it into a character conforms to the standard language notation. For example, there may be a case in which the conversion result of the accuracy calculation unit 130 does not fit the spelling, and there may be a case where the speaker in the content speaks a dialect.

The accuracy calculator 130 may separately record a timestamp of a corresponding word or section when the speech recognition result is less than or equal to a specific value. The time stamp recorded here may be transmitted to the integrated caption processing unit 300 or may be transmitted to the shorthand input unit 200.

It comes back to FIG. 1 again.

The shorthand input unit 200 acquires a shorthand input. The shorthand input unit 200 may obtain a shorthand input from the shorthand. A specific configuration of the shorthand input unit 200 will be separately described below.

3 is a block diagram showing the configuration of a shorthand input unit 200 according to an embodiment of the present invention.

As shown in FIG. 3, the shorthand input unit 200 according to an embodiment of the present invention may include a notification display unit 210 and a shorthand input device 220.

The notification display unit 210 displays a notification to the stenographer. Here, the notification displayed to the stenographer may indicate that the recognition accuracy is less than a certain value by the voice recognition unit 100. When the speech recognition accuracy is less than a certain value, there is a high probability that the result of the speech-to-text conversion is not accurate. In this case, the stenographer may directly input the subtitle to correct the result of the automatic speech recognition. The notification display unit 210 may be a display device or an audio device, and the notification display unit 210 may provide a visual or audible notification to the stenographer.

The shorthand input device 220 obtains a shorthand input from the shorthand. The shorthand input device 220 may obtain a shorthand input from a shorthand keyboard and convert it into text data. The shorthand input device 220 may be a commonly used shorthand keyboard. In addition, the shorthand keyboard may be a shorthand keyboard for both English and Korean. The shorthand input device 220 may further include a display device. The display device may display a caption input through a shorthand keyboard.

It comes back to FIG. 1 again.

The integrated caption processing unit 300 generates a final caption by integrating the text received from the voice recognition unit 100 and the shorthand input unit 200. Specifically, the integrated caption processing unit 300 generates a final caption by combining the voice-to-text conversion data received from the speech recognition unit 100 and the shorthand input data transmitted from the shorthand input unit 200.

In one embodiment, the integrated caption processing unit 300 is based on the voice-to-text conversion data received from the voice recognition unit 100, and supplements some of the voice-to-text conversion data with shorthand input data received from the shorthand input unit 200 Thus, the final subtitle can be generated. As described above, the text conversion result may be inaccurate due to the low recognition accuracy of the speech recognition unit 100 in a specific situation. In this case, the final subtitle may be generated by supplementing the incorrect text conversion result with the direct input of the stenographer. I can.

The integrated caption processing unit 300 may obtain a word to be supplemented with an accuracy of less than a certain value or timestamp information of a section to be supplemented from the speech recognition unit 100. In addition, the integrated caption processing unit 300 may generate a final caption by synchronizing the shorthand input data and the voice text conversion data by comparing the timestamp of the complementary word or section with the shorthand input start time.

In a specific embodiment, the integrated caption processing unit 300 compares the timestamp of the complementary word or section with the start time of the supplementary shorthand input, and matches the complementary shorthand input with the smallest difference between the complementary shorthand input and the final subtitle. Create

In another embodiment, the integrated caption processing unit 300 compares and matches only the temporal order of one or more supplementary target words or sections with the temporal order of one or more supplementary shorthand inputs to generate a final caption. Since the number of supplementary words or sections and the number of supplementary shorthand inputs will be the same, a final subtitle may be generated by comparing only each order and replacing the supplementary words or sections in order with supplementary shorthand input.

4 is a flowchart illustrating an operation of a caption generation system according to an embodiment of the present invention.

The caption generation system converts the voice included in the first section into text through an automatic voice recognition tool (S10). Here, the automatic speech recognition tool may be an automatic speech recognition tool currently used as described above. The first section means a partial section of the entire timeline of the content subject to speech recognition.

The caption generation system acquires the accuracy of automatic speech recognition, and determines whether the accuracy is greater than or equal to a specific value (S20). The accuracy of automatic speech recognition may be determined based on at least one of a size of a speech signal, a ratio between a human voice and noise, or a result of automatic speech recognition. In addition, the specific reference value used as the threshold value here may be a randomly input value or a value obtained through machine learning.

When the accuracy of the voice-to-text conversion in the first section is greater than or equal to a specific value, the subtitle generation system automatically recognizes the voice included in the other section and converts it into a letter (S30).

Meanwhile, when the accuracy of the voice-to-text conversion of the first section is less than a specific value, the caption generation system records the start time of the first section (S40). In general, an automatic speech recognition tool records a timestamp obtained by acquiring a voice converted into a text, and the caption generation system may separately record and manage a timestamp for a word or section whose accuracy is less than a specific value.

When the conversion accuracy of the first section is less than or equal to a specific value, the caption generation system outputs a notification to the shorthand (S50). The caption generation system may output a notification to the stenographer in a visual or audible manner.

The caption generation system acquires a shorthand input for the first voice (S60). The caption generation system may obtain a shorthand input for the first voice through a shorthand keyboard. In addition, the caption generating system may obtain a shorthand input for the first voice through voice recognition. Here, the object of speech recognition may be the voice of a shorthand article.

The caption generation system generates a final caption by integrating the voice recognition result and the shorthand input result based on the recorded start time of the first section and the shorthand input start time (S70).

In an embodiment, the caption generation system compares the start time of a section (hereinafter referred to as supplementary section) whose conversion accuracy is less than a certain value and a time when shorthand input for the supplementary section is started, and compares the supplementary section with the smallest difference and the supplementary shorthand input. Matching to generate final subtitles. Supplementary shorthand input refers to shorthand input data input by a shorthand by a shorthand notice.

In another embodiment, the caption generation system generates a final caption by comparing and matching only the start times of one or more supplementary sections and a time sequence of one or more supplementary shorthand inputs. Since the number of supplementary words or sections and the number of supplementary shorthand inputs will be the same, a final subtitle may be generated by comparing only each order and replacing the supplementary words or sections in order with supplementary shorthand input.

The subtitle generation system delivers the final subtitles generated to the set-top box. The set-top box can output closed caption broadcasting for the hearing impaired by displaying the transmitted caption together with the image.

The present invention described above can be implemented as a computer-readable code in a medium on which a program is recorded. The computer-readable medium includes all types of recording devices storing data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is also a carrier wave (eg, transmission over the Internet).

The above detailed description should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (9)

A voice recognition unit for receiving a voice signal from content, recognizing the voice signal, and converting it into text;
A shorthand input unit for obtaining a caption shorthand input through the shorthand input device; And
An integrated caption processing unit for generating a final caption by integrating the speech-to-text conversion data obtained from the speech recognition unit and the shorthand input data obtained from the shorthand input unit,
The integrated caption processing unit
When the accuracy of the voice-to-text conversion of the first section among the voice signals from the voice recognition unit is less than a specific value, the shorthand input for the first section is obtained from the shorthand input device, and the time when the first section is received Matching the shorthand input time and generating the final subtitle
Broadcast subtitle production system. The method of claim 1,
The voice recognition unit
A voice receiver for receiving a voice signal from content
A voice-to-text conversion unit for converting a voice signal into text,
Comprising an accuracy calculation unit for calculating the accuracy of the speech recognition
Broadcast subtitle production system. The method of claim 2,
The accuracy calculation unit calculates the accuracy based on the size of the voice included in the voice signal.
Broadcast subtitle production system. The method of claim 2,
The accuracy calculation unit calculates the accuracy based on the ratio between the sound and noise excluding the sound included in the voice signal.
Broadcast subtitle production system. The method of claim 2,
The accuracy calculation unit calculates the accuracy based on the character conversion result of the speech signal.
Broadcast subtitle production system. The method of claim 3,
The shorthand input unit
A notification display unit for displaying a notification to a stenographer when the accuracy of the speech-to-text conversion is less than a specific value, and
Including a shorthand input device including a shorthand keyboard
Broadcast subtitle production system. Converting a voice included in the first section of the voice signal of the content into text through voice recognition;
Determining whether an accuracy of speech-to-text conversion for the first section is greater than or equal to a specific value;
Outputting a notification for obtaining a shorthand input for the first section when the accuracy of the speech-to-text conversion for the first section is less than a specific value;
Obtaining a shorthand input for a first section corresponding to the notification; And
And generating a final caption by integrating the speech recognition result and the shorthand input result based on the start time of the first section and the shorthand input start time.
How to create broadcast subtitles. The method of claim 7,
Generating the final subtitles,
Comprising the step of comparing the start time of the first section with a time when the shorthand input for the first section is started, and matching the supplemental section with the smallest difference with the supplementary shorthand input to generate a final caption
How to create broadcast subtitles. The method of claim 7,
Generating the final subtitles,
And generating a final caption by matching at least one complementary target section including the first section and at least one supplementary shorthand input in chronological order.
How to create broadcast subtitles.

Images