CN118522302A - Voice recognition voice and word synchronization method and device - Google Patents

Abstract

The present invention relates to a method for speech recognition and audio-character synchronization, comprising the following steps: obtaining target speech information; performing speech recognition on the target speech information to obtain speech recognition content, wherein the speech recognition content includes speech content and text content; determining the start time and end time corresponding to each character in the text content in the speech content, and taking the start time of the first character as the initial time when the speech content is displayed, based on which the relative time positions of all characters in the speech content when the speech content is displayed are determined; taking the start time of the first character in the text content as the initial time when the text content is displayed, based on which the relative time positions of all characters in the text content when the text content is displayed are determined; performing speech recognition and audio-character synchronization display at the relative time positions of all characters in the speech content and the text content determined in the above manner when displayed. The present invention can intelligently ensure speech recognition and audio-character synchronization.

Description

A method and device for speech recognition sound-word synchronization

Technical Field

The present invention relates to the technical field related to speech recognition, and in particular to a method and device for speech recognition sound-word synchronization.

Background Art

Speech recognition, also known as Automatic Speech Recognition (ASR), is a technology that converts spoken language into text through computer programs. It is widely used in many intelligent scenarios.

In some intelligent scenarios, it is necessary to display the text content of voice recognition, and during the display process, it is necessary to ensure that the display frequency of the text content is synchronized with the played voice content as much as possible to bring users a more suitable sensory experience. Currently, most of the methods on the market are manually arranged, and a small number of them are roughly arranged through intelligent networks and finally realized through manual fine review and adjustment. These methods undoubtedly bring huge workload to the staff, and due to the differences in human experience levels, the final results may be uneven.

Summary of the invention

The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a method and device for speech recognition pronunciation and character synchronization.

In order to achieve the above object, the present invention adopts the following technical solutions:

Specifically, a method for speech recognition and pronunciation-word synchronization is proposed, comprising the following steps:

Obtain target voice information;

Performing speech recognition on the target speech information to obtain speech recognition content, wherein the speech recognition content includes speech content and text content;

Determine the start time and end time corresponding to each character in the text content in the voice content, and use the start time of the first character as the initial time when the voice content is displayed, based on which determine the relative time position of all characters in the voice content when the voice content is displayed;

The starting time of the first word in the text content is used as the initial time when the text content is displayed, and based on this, the relative time positions of all words in the text content when the text content is displayed are determined;

The speech recognition audio and text are displayed synchronously based on the relative time positions of the speech content and all the characters in the text content determined in the above manner when they are displayed.

Further, specifically, the target voice information is subjected to voice recognition through an HMM voice recognition model to obtain text content, and the voice content refers to the voice information obtained after the original target voice information is subjected to a preprocessing operation.

Further, specifically, the speech information obtained after the original target speech information is preprocessed includes:

The original target voice information is pre-emphasized by the following formula to obtain the first voice information:

_Bn ＝ _An -a* _An -1,

Wherein _Bn represents the first speech information, _An represents the original target speech information, a is a preset constant, and the value range of a is [0.95, 1.00];

The first voice information is subjected to windowing processing by the following formula to obtain the second voice information:

_Cn ＝ _Bn *W(n),

Where _Cn represents the second speech information, W(n) is the window function, and W(n) is specifically:

The second speech information is subjected to endpoint detection based on a short-time energy determination method to distinguish unvoiced segments and voiced segments of the second speech to obtain a plurality of speech segments.

Further, specifically, determining the start time and end time corresponding to each character in the text content in the voice content includes:

For each speech segment,

The feature vector of speech segment is obtained by feature extraction algorithm;

Based on the feature vector of the speech segment, the speech segment is recognized by the speech recognition model to obtain the text corresponding to the speech segment;

The starting time and initial time of each speech segment are used as the starting time and ending time of its corresponding text, so as to determine the starting time and ending time corresponding to each text in the text content.

Further, specifically, the feature vector of the speech segment is obtained by the feature extraction algorithm, including:

Process each speech segment by Fourier transform to obtain a time domain signal c(n) of each speech segment;

The time domain signal c(n) is padded with 0 at the end to obtain a sequence x(n) with a sequence number of N. The sequence is processed by discrete Fourier transform to obtain a linear spectrum X(k), which is calculated as follows:

The linear spectrum X(k) is processed by a MEL bandpass filter to obtain a MEL spectrum, and the MEL spectrum is converted to obtain a logarithmic spectrum Q(m).

The transfer function of each bandpass filter in the MEL bandpass filter is:

Where 0≤m≤M, M is the number of bandpass filters,

Where _fh and _fl represent the maximum frequency and minimum frequency of the bandpass filter respectively, _Fs is the sampling frequency, Y is the window width of the window function for discrete Fourier transform, and x represents the independent variable.

The MFCC coefficient d(n) is obtained by discrete cosine transform of Q(m):

Furthermore, the method also includes, when performing speech recognition and audio-character synchronous display,

The voice content and the corresponding text content are buffered in advance, that is, the voice content and the corresponding text content paragraphs pre-annotated by the staff are obtained, and when the current paragraph is displayed, the relevant content of the next paragraph is preloaded.

Furthermore, the method also includes, when performing speech recognition and audio-character synchronous display,

After the current paragraph is displayed, the real-time voice information of the current paragraph is obtained synchronously;

Performing voice recognition on the real-time voice information to obtain real-time voice recognition content;

Determining the start time and the end time of all characters in the real-time voice recognition content when presenting the voice content;

The start time and the end time of each character when the voice content is displayed are averaged and recorded as the voice timestamp of the character;

Get the start time and end time of each character when displaying the text content;

The start time and the end time of each character when displaying the text content are calculated and recorded as the text timestamp of the character;

The difference between the speech timestamp and the text timestamp of each character is recorded as the synchronization tolerance parameter of the character;

Traverse the synchronization tolerance parameters of all characters, find the characters whose absolute values of the synchronization tolerance parameters are greater than the first threshold, and count their number to obtain the number P;

If the number P is less than the second threshold, the text content display speed of the next paragraph will not be adjusted. If the number P is neither less than the second threshold nor greater than the third threshold, the average value of all synchronization tolerance parameters will be calculated. If the average value is a positive number, the text content display speed of the next paragraph will be increased by a preset proportion. If the average value is a negative number, the text content display speed of the next paragraph will be reduced by a preset proportion. If the average value is exactly 0, the text content display speed of the next paragraph will not be adjusted. If the number P is greater than the third threshold, an alarm will be issued to the relevant staff.

The present invention also provides a device for speech recognition and pronunciation-word synchronization, comprising the following:

A data acquisition module, used to acquire target voice information;

A speech recognition module, used for performing speech recognition on the target speech information to obtain speech recognition content, wherein the speech recognition content includes speech content and text content;

A first time determination module is used to determine the start time and end time corresponding to each character in the text content in the voice content, and use the start time of the first character as the initial time when the voice content is displayed, based on which the relative time positions of all characters in the voice content when the voice content is displayed are determined;

A second time determination module is used to use the starting time of the first word in the text content as the initial time when the text content is displayed, and based on this, determine the relative time position of all words in the text content when the text content is displayed;

The synchronous display module is used to perform speech recognition audio and character synchronous display based on the relative time positions of all characters in the speech content and text content determined in the above manner when displaying.

The beneficial effects of the present invention are:

The present invention provides a method for speech recognition and audio-text synchronization, which obtains speech content and text content by performing speech recognition on target speech information, and then synchronously annotates the speech content and text content according to the relative time position relationship of the text of the speech content when it is displayed, thereby ensuring that the speech recognition content achieves audio-text synchronization, and in addition, when the speech recognition and audio-text are synchronously displayed, the speech recognition and audio-text synchronization is ensured as much as possible by means of preloading and feedback mechanisms.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features of the present disclosure will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present disclosure represent the same or similar output voltages. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:

FIG. 1 is a flow chart of a method for speech recognition and pronunciation-word synchronization according to the present invention.

DETAILED DESCRIPTION

The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

Referring to FIG. 1 , Embodiment 1, the present invention proposes a method for speech recognition and pronunciation-word synchronization, comprising the following steps:

Step 110: Obtain target voice information;

Step 120: Perform voice recognition on the target voice information to obtain voice recognition content, where the voice recognition content includes voice content and text content;

Step 130: determining the start time and end time corresponding to each character in the text content in the voice content, and taking the start time of the first character as the initial time when the voice content is displayed, based on which the relative time positions of all characters in the voice content when the voice content is displayed are determined;

Step 140: Taking the starting time of the first character in the text content as the initial time when the text content is displayed, based on which the relative time positions of all characters in the text content when the text content is displayed are determined;

Step 150: Perform speech recognition audio-character synchronization display based on the relative time positions of the speech content determined in the above manner and all the characters in the text content when being displayed.

In this embodiment 1, voice content and text content are obtained by performing voice recognition on the target voice information, and then the voice content and text content are synchronously annotated based on the relative time position relationship of the text of the voice content when it is displayed, based on this, it can be ensured that the voice recognition content can achieve audio-text synchronization.

As a preferred embodiment of the present invention, specifically, the target voice information is subjected to voice recognition through an HMM voice recognition model to obtain text content, and the voice content refers to the voice information obtained after the original target voice information is subjected to a preprocessing operation.

In this preferred embodiment, the text content is obtained by performing speech recognition using a mature HMM speech recognition model, which can ensure that the recognized text content is accurate.

As a preferred embodiment of the present invention, specifically, the original target voice information is obtained after the preprocessing operation, including:

The original target voice information is pre-emphasized by the following formula to obtain the first voice information:

_Bn ＝ _An -a* _An -1,

Wherein _Bn represents the first speech information, _An represents the original target speech information, a is a preset constant, and the value range of a is 0.95, 1.00];

The first voice information is subjected to windowing processing by the following formula to obtain the second voice information:

_Cn ＝ _Bn *W(n),

Where _Cn represents the second speech information, W(n) is the window function, and W(n) is specifically:

The second speech information is subjected to endpoint detection based on a short-time energy determination method to distinguish unvoiced segments and voiced segments of the second speech to obtain a plurality of speech segments.

In this preferred embodiment, by finding the speech segments in the above manner, omissions can be avoided when finding each word in the text content.

As a preferred embodiment of the present invention, specifically, determining the start time and the end time corresponding to each character in the text content in the voice content includes:

For each speech segment,

The feature vector of speech segment is obtained by feature extraction algorithm;

Based on the feature vector of the speech segment, the speech segment is recognized by the speech recognition model to obtain the text corresponding to the speech segment;

The starting time and initial time of each speech segment are used as the starting time and ending time of its corresponding text, so as to determine the starting time and ending time corresponding to each text in the text content.

In this preferred embodiment, the process of determining the start time and the end time corresponding to each word in the text content in the voice content includes the following:

1. Speech segmentation: The speech recognition system first segments the audio into smaller speech segments. These segments are usually determined based on features such as changes in speech characteristics, silence detection, or the energy of the speech signal.

2. Feature extraction: For each speech segment, the feature extraction algorithm converts it into feature vectors, which usually include MFCC (Mel frequency cepstral coefficients), power spectrum density, etc. These feature vectors can help identify the acoustic characteristics of each word.

3. Speech recognition model: Next, a speech recognition model (such as a deep learning model) is used to recognize each segment and convert it into text. This step usually produces a probability distribution for each word, indicating the probability of the word appearing in the segment.

4. Timestamp generation: Based on the duration of the speech segment and the output of the speech recognition model, the time when each word appears in the segment can be estimated. Usually, this is calculated based on the start and end times of the speech segment and the relative position of each word.

5. Post-processing: After getting the timestamp for each word, some post-processing steps may be required to improve the accuracy. This may involve some techniques such as time alignment, smoothing, etc. to ensure that the timestamp for each word is accurate.

As a preferred embodiment of the present invention, specifically, the feature vector of the speech segment is obtained by a feature extraction algorithm, including:

Process each speech segment by Fourier transform to obtain a time domain signal c(n) of each speech segment;

The time domain signal c(n) is padded with 0 at the end to obtain a sequence x(n) with a sequence number of N. The sequence is processed by discrete Fourier transform to obtain a linear spectrum X(k), which is calculated as follows:

The linear spectrum X(k) is processed by a MEL bandpass filter to obtain a MEL spectrum, and the MEL spectrum is converted to obtain a logarithmic spectrum Q(m).

The transfer function of each bandpass filter in the MEL bandpass filter is:

Where 0≤m≤M, M is the number of bandpass filters,

Where _fh and _fl represent the maximum frequency and minimum frequency of the bandpass filter respectively, _Fs is the sampling frequency, Y is the window width of the window function for discrete Fourier transform, and x represents the independent variable.

The MFCC coefficient d(n) is obtained by discrete cosine transform of Q(m):

In this preferred embodiment, by calculating the MFCC coefficients as the feature vectors of the speech segment in the above manner, the MFCC coefficients can be accurately obtained to facilitate the subsequent related recognition calculation process.

As a preferred embodiment of the present invention, the method further comprises, when performing speech recognition, sound and word synchronous display,

The voice content and the corresponding text content are buffered in advance, that is, the voice content and the corresponding text content paragraphs pre-annotated by the staff are obtained, and when the current paragraph is displayed, the relevant content of the next paragraph is preloaded.

In this preferred embodiment, a certain amount of voice content and corresponding text content is buffered in advance to ensure that the synchronization of voice and text can be maintained even in the case of network delay or processing delay. This can be achieved by preloading the next segment of voice and text content to ensure that the next segment of content is ready before the current segment of content is played.

As a preferred embodiment of the present invention, the method further comprises, when performing speech recognition, sound and word synchronous display,

After the current paragraph is displayed, the real-time voice information of the current paragraph is obtained synchronously;

Performing voice recognition on the real-time voice information to obtain real-time voice recognition content;

Determining the start time and the end time of all characters in the real-time voice recognition content when presenting the voice content;

The start time and the end time of each character when the voice content is displayed are averaged and recorded as the voice timestamp of the character;

Get the start time and end time of each character when displaying the text content;

The start time and the end time of each character when displaying the text content are calculated and recorded as the text timestamp of the character;

The difference between the speech timestamp and the text timestamp of each character is recorded as the synchronization tolerance parameter of the character;

Traverse the synchronization tolerance parameters of all characters, find the characters whose absolute values of the synchronization tolerance parameters are greater than the first threshold, and count their number to obtain the number P;

If the number P is less than the second threshold, the text content display speed of the next paragraph will not be adjusted. If the number P is neither less than the second threshold nor greater than the third threshold, the average value of all synchronization tolerance parameters will be calculated. If the average value is a positive number, the text content display speed of the next paragraph will be increased by a preset proportion. If the average value is a negative number, the text content display speed of the next paragraph will be reduced by a preset proportion. If the average value is exactly 0, the text content display speed of the next paragraph will not be adjusted. If the number P is greater than the third threshold, an alarm will be issued to the relevant staff.

In this preferred embodiment, the display speed and time of the text content are dynamically adjusted according to the actual playback situation to ensure synchronization with the voice content. If it is found that the text content display speed is too fast or too slow, it can be corrected by real-time adjustment through the above feedback mechanism, wherein the first threshold, the second threshold, and the third threshold are all artificially set by pre-experimentation.

The present invention also provides a device for speech recognition and pronunciation-word synchronization, comprising the following:

A data acquisition module, used to acquire target voice information;

A speech recognition module, used for performing speech recognition on the target speech information to obtain speech recognition content, wherein the speech recognition content includes speech content and text content;

A first time determination module is used to determine the start time and end time corresponding to each character in the text content in the voice content, and use the start time of the first character as the initial time when the voice content is displayed, based on which the relative time positions of all characters in the voice content when the voice content is displayed are determined;

A second time determination module is used to use the starting time of the first word in the text content as the initial time when the text content is displayed, and based on this, determine the relative time position of all words in the text content when the text content is displayed;

The synchronous display module is used to perform speech recognition audio and character synchronous display based on the relative time positions of all characters in the speech content and text content determined in the above manner when displaying.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

Although the description of the present invention has been quite detailed and specifically described with respect to several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventors, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and/or implementation method can have various modifications and changes.

Claims (8)

1. A method of voice recognition word synchronization, comprising the steps of:

acquiring target voice information;

performing voice recognition on the target voice information to obtain voice recognition content, wherein the voice recognition content comprises voice content and text content;

determining the starting time and the ending time corresponding to each text in the text content in the voice content, taking the starting time of the first text as the initial time when the voice content is displayed, and determining the relative time positions of all the text in the voice content when the voice content is displayed based on the starting time and the ending time;

taking the initial time of the first text in the text content as the initial time of the text content display, and determining the relative time positions of all the text in the text content when the text content is displayed based on the initial time;

And performing voice recognition voice and word synchronous display according to the relative time positions of the voice content and all the words in the text content, which are determined in the mode, when the voice content and the words are displayed.

2. The method for synchronizing voice and word in accordance with claim 1, wherein specifically, text content is obtained by performing voice recognition on the target voice information through an HMM voice recognition model, and the voice content refers to voice information obtained by preprocessing the original target voice information.

3. The method of claim 2, wherein the voice information obtained by preprocessing the original target voice information comprises,

Pre-emphasis processing is carried out on the original target voice information to obtain first voice information,

B_n＝A_n-a*A_n-1，

Wherein B _n represents first voice information, A _n represents original target voice information, a is a preset constant, and the value range of a is [0.95,1.00];

The first voice information is windowed to obtain second voice information,

C_n＝B_n*W(n)，

Wherein C _n represents the second speech information, W (n) is a window function, W (n) is specifically

And carrying out end point detection on the second voice information based on the judging mode of the short-time energy so as to distinguish an unvoiced sound section and a voiced sound section of the second voice to obtain a plurality of voice sections.

4. A method for synchronizing speech recognition and voice characters according to claim 3, wherein determining the start time and the end time of each text in the text content comprises,

For each segment of speech,

Obtaining a feature vector of the voice segment through a feature extraction algorithm;

Based on the feature vector of the voice segment, the voice segment is identified through a voice identification model to obtain the text corresponding to the voice segment;

The starting time and the initial time of each voice segment are used as the starting time and the ending time of the corresponding text, so that the starting time and the ending time corresponding to each text in the text content are determined.

5. The method of claim 4, wherein the feature extraction algorithm is used to obtain feature vectors of the speech segments,

Processing each voice segment through Fourier transformation to obtain a time domain signal c (n) of each voice segment;

The method comprises the steps of carrying out last-bit compensation on a time domain signal c (N) to obtain a sequence X (N) with a sequence number of N, and processing the sequence by discrete Fourier transformation to obtain a linear frequency spectrum X (k), wherein the calculation formula is as follows:

Processing the linear frequency spectrum X (k) by a MEL band-pass filter to obtain a MEL frequency spectrum, converting the MEL frequency spectrum to obtain a logarithmic frequency spectrum Q (m),

Wherein the transfer function of each of the MEL bandpass filters is:

Wherein M is more than or equal to 0 and less than or equal to M, M is the number of band-pass filters,

Wherein F _h and F _l respectively represent the maximum frequency and the minimum frequency of the band-pass filter, F _s is the sampling frequency, Y is the window width of the window function for performing discrete Fourier transform, andX represents an independent variable;

the MFCC coefficient d (n) obtained by discrete cosine transform on Q (m) is:

6. the method of claim 1, further comprising, when performing the voice recognition word synchronization presentation,

And (3) buffering the voice content and the corresponding text content in advance, namely acquiring the voice content and the corresponding text content paragraph marked in advance by the staff, and preloading the related content of the next paragraph when the current paragraph is displayed.

7. The method of claim 6, further comprising, when performing the voice recognition word synchronization presentation,

After the current paragraph is displayed, synchronously acquiring real-time voice information of the current paragraph;

performing voice recognition on the real-time voice information to obtain real-time voice recognition content;

Determining the starting time and the ending time of all characters in the real-time voice recognition content when the voice content is displayed;

Averaging the starting time and the ending time of each character when the voice content is displayed, and marking the starting time and the ending time as voice time stamps of the characters;

acquiring the starting time and the ending time of each character when the character content is displayed;

averaging the starting time and the ending time of each character when the character content is displayed, and marking the average value as a character time stamp of the character;

Marking the voice time stamp of each character and the character time stamp as the synchronous tolerance parameter of the character;

traversing synchronous tolerance parameters of all characters, finding out characters with absolute values larger than a first threshold value of the synchronous tolerance parameters, and counting the number of the characters to obtain a number P;

If the number P is smaller than the second threshold value, the display speed of the text content of the next paragraph is not adjusted, if the number P is not smaller than the second threshold value and is not larger than the third threshold value, the average value of all synchronous tolerance parameters is calculated, if the average value is positive, the display speed of the text content of the next paragraph is increased by a preset proportion, if the average value is negative, the display speed of the text content of the next paragraph is reduced by the preset proportion, if the average value is just 0, the display speed of the text content of the next paragraph is not adjusted, and if the number P is larger than the third threshold value, the alarm reminding is carried out on related staff.

8. An apparatus for voice recognition word synchronization, comprising:

The data acquisition module is used for acquiring target voice information;

the voice recognition module is used for carrying out voice recognition on the target voice information to obtain voice recognition content, wherein the voice recognition content comprises voice content and text content;

The first time determining module is used for determining the starting time and the ending time corresponding to each word in the text content in the voice content, taking the starting time of the first word as the initial time when the voice content is displayed, and determining the relative time position of all words in the voice content when the voice content is displayed based on the starting time;

The second time determining module is used for determining the relative time positions of all the characters in the text content when the text content is displayed by taking the initial time of the first character in the text content as the initial time when the text content is displayed;

And the synchronous display module is used for synchronously displaying the voice recognition voice and the word according to the relative time positions of the voice content and all the words in the text content, which are determined in the mode.