CN108510981B - Method and system for acquiring voice data - Google Patents

Abstract

The present invention provides a method and system for acquiring voice data, including: when a user makes a voice call, saving the voice data stream transmitted in real time in the intelligent terminal system, saving the input voice data stream of the microphone as the first voice data, and storing the earpiece The output voice data stream is saved as the second voice data; detect whether the first voice data and the second voice data meet the training requirements of the voice recognition model, if so, continue to judge whether the first voice data comes from the application object of the voice recognition model, if so, set the The first voice data is marked as application target voice data, and the second voice data is marked as non-application target voice data; if not, the first voice data and the second voice data are marked as non-application target voice data. Based on the method of the present invention, by improving the voice acquisition method, the user's burden of training the voice recognition model is reduced, and the user experience is improved.

Description

Method and system for acquiring voice data

technical field

The invention relates to the field of artificial intelligence, in particular to a method and system for acquiring voice data.

Background technique

Mobile terminal speech recognition is divided into two categories: semantic recognition and speaker recognition.

Speaker recognition is often referred to as voiceprint recognition. Generally, it is divided into two categories: Text-dependent and Text-independent.

Text-related speech recognition usually requires the user to repeat the fixed words and sentences 2-3 times. Record related feature information as Enroll. When using, the user is also required to read the same fixed words and phrases for speech discrimination (Predict).

Non-text-related speech recognition does not require users to follow fixed sentences. The user inputs a large amount of speech data as the machine learning training (Train), and the user's characteristic information is highly purified under the training of a large amount of data. The voice data used for training needs to include the user's voice data (the application object of the voice recognition model) and other people's voice data. There is also no need to read fixed words and phrases during speech discrimination. Normal speech data can be used for speech discrimination.

In the prior art, the mobile intelligent terminal cannot distinguish the user identity for voice recognition, and does not distinguish the voice feature values of different users, so that the same mobile intelligent terminal can serve the voice commands of different users at the same time, confidentiality and exclusivity. poor.

Taking the voice assistant as an example, existing mobile smart terminals need to have a fixed wake-up process when the voice assistant service is enabled. This is a defect of text-related speech recognition, which cannot escape the limitation of fixed text, and cannot respond quickly to any voice command of the user (application object). All voice commands need to be activated after the voice assistant is awakened. Any user can wake up the voice assistant through fixed words and issue voice commands. The voice assistant cannot perform voice recognition on the user's identity, and all voice commands will be executed.

Non-text-related speech recognition uses machine learning technology to obtain highly purified user feature information and model parameters by establishing a complete learning model and inputting a large amount of speech data for training. Based on the trained model, users can achieve highly accurate speaker speech recognition through arbitrary speech input, without being restricted by fixed text.

However, to realize non-text-related speech recognition on a mobile intelligent terminal, it is necessary to acquire a large amount of voice data of registrants and non-registrants. The training process is long and boring. The user experience is a big challenge. Users do not want to spend time and effort entering voice data. In addition, acquiring speech data of an application object of a non-speech recognition model is also an embarrassing problem for end users. High recognition accuracy cannot be achieved without sufficient training data. Therefore, there is no non-text-related speech recognition system in the existing mobile smart terminals.

In view of the above problems, especially the acquisition method of speech data in the non-text speech recognition model of terminal application, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

The present invention provides a voice data acquisition method and system, which reduces the burden on users by improving the voice data acquisition process.

The present invention provides a method for acquiring voice data. The voice data is used for training a voice recognition model, and the method includes the following steps:

Step A-1: When the user makes a voice call, save the voice data stream transmitted in real time in the intelligent terminal system, save the input voice data stream of the microphone as the first voice data, and save the output voice data stream of the earpiece as the second voice data;

Step A-2: Detect whether the first voice data and the second voice data meet the training requirements of the voice recognition model, and if so, perform Step A-3;

Step A-3: judge whether the first speech data is from the application object of the speech recognition model, if so, execute step A-4, if not, execute step A-5;

Step A-4: mark the first speech data as application object speech data, mark the second speech data as non-application object speech data, and use the application object speech data for the speech feature learning of the application object in the speech recognition model; Speech data is used for speech feature learning of non-application objects in speech recognition model;

Step A-5: Mark the first voice data and the second voice data as non-application target voice data.

The present invention also provides a system for acquiring voice data, the voice data is used for training a voice recognition model, and the system includes:

Save module: when the user makes a voice call, save the voice data stream transmitted in real time in the intelligent terminal system, save the input voice data stream of the microphone as the first voice data, and save the output voice data stream of the earpiece as the second voice data;

Detection module: detects whether the first voice data and the second voice data meet the training requirements of the voice recognition model, and if so, executes the user judgment module;

User judgment module: judge whether the first voice data comes from the application object of the voice recognition model, if so, execute the voice object marking module 1, if otherwise, execute the voice object marking 2;

Voice object marking 1: mark the first voice data as application object voice data, mark the second voice data as non-application object voice data, and use the application object voice data for the voice feature learning of the application object in the speech recognition model; Speech data is used for speech feature learning of non-application objects in speech recognition model;

Voice object marking 2: marking the first voice data and the second voice data as non-application object voice data.

The present invention saves the voice data during the user's voice call, uses the input voice data (first voice data) of the microphone for the voice feature learning of the application object in the voice recognition model, and uses the output voice data (second voice data) of the earpiece as For the speech feature learning of non-application objects in the speech recognition model, the training speech data is transmitted to the speech recognition model in a "silent" way in the background of the mobile intelligent terminal, and the user does not need to do boring and complicated input work, which reduces the user's training burden. Improved user experience. At the same time, the method and system of the present application can be applied to any speech recognition model based on neural network, and has a wide range of applications. Based on the voice data acquisition method and acquisition system of the present application, non-text-related voice recognition can be implemented on mobile intelligent terminals, breaking through the limitations of existing text-related voice recognition, allowing terminals to more intelligently understand the characteristics of each user, Use habits to enhance exclusivity and security.

Description of drawings

Fig. 1 is the flow chart of the acquisition method of speech data of the present invention;

Fig. 2 is an embodiment of Fig. 1;

Fig. 3 is the structure diagram of the acquisition system of speech data of the present invention;

FIG. 4 is an embodiment of FIG. 3 .

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

Fig. 1 is the flow chart of the acquisition method of speech data of the present invention, comprises the following steps:

Step A-1 (S101): when the user makes a voice call, save the voice data stream transmitted in real time in the intelligent terminal system, save the input voice data stream of the microphone as the first voice data, and save the output voice data stream of the earpiece as second voice data;

Step A-2 (S102): Detect whether the first voice data and the second voice data meet the training requirements of the voice recognition model, if so, execute Step A-3;

Step A-3 (S103): judge whether the first speech data comes from the application object of the speech recognition model, if so, execute step A-4, if not, execute step A-5;

Step A-4 (S104): the first voice data is marked as application object voice data, the second voice data is marked as non-application object voice data, and the application object voice data is used for the voice feature learning of the application object in the voice recognition model; The non-application object speech data is used for the speech feature learning of non-application objects in the speech recognition model;

Step A-5 (S105): Mark the first voice data and the second voice data as non-application target voice data.

In step A-1, the voice call includes not only audio calls, but also video calls such as VoIP and VoLTE; it also includes real-time audio and video calls of other instant messaging apps, such as WeChat's "video chat" or "voice chat".

When the user initiates a voice call, the method of FIG. 1 is triggered to be executed. When applied to WeChat and QQ, when a corresponding action is detected, such as pressing or taking effect of a "video chat" or "voice chat" button, the method in FIG. 1 is triggered to be executed.

Step A-1 saves the work of the voice data stream, which can be set at the hardware device operation layer in the operating system of the mobile intelligent terminal. When the user starts a voice call, the hardware operation layer of the system backs up and saves the input voice data and data of the microphone in real time. The output voice data of the receiver, wherein the input voice data of the microphone represents the voice data of the terminal user, and the output voice data of the receiver represents the voice data transmitted by the opposite end to the terminal user in real time.

Taking the Android system as an example, the hardware device operation layer is Android HAL, and the call status can be judged by referring to the related attributes of call_connected in tiny_audio_device in AudioHAL. When adev->call_connected is true, it means that it is used to open and in a voice call.

In AudioHAL, when audio_hw_device is AUDIO_DEVICE_IN_BUILTIN_MIC, it means that the current microphone device is in working state. When audio_hw_device is AUDIO_DEVICE_OUT_EARPIECE, it means that the current earpiece device is in working state.

Further, in the out_write() function of AudioHAL, backup and save the voice data before the earpiece output, and write corresponds to the playback process. Similarly, the voice data input by the microphone is backed up and saved in the in_read() function, and read corresponds to the recording process.

In addition, in step A-1, the first voice data and the second voice data may be stored in the ROM or RAM of the mobile smart terminal.

The voice feature value extraction technology of the neural network speech recognition model needs to input a large number of personal voices in advance for training to obtain the speaker's voiceprint features. The existing method uses a designated program to allow the user to input speech data sentence by sentence for training, which requires the user to spend extra time to train personal voice characteristics, and the process is complicated and tedious.

The training method shown in FIG. 1 of the present application collects instant messaging voice data generated by users in daily work and life through intelligent terminals, and uses the collected voice data to train a voice recognition model, so as to continuously improve the recognition accuracy of the voice recognition model. . Compared with the prior art, the user does not need to do boring and complicated input work, which reduces the training burden of the user and improves the user experience.

At the same time, using the training method shown in FIG. 1 of the present application, over time, non-text-related speech recognition can be implemented on mobile intelligent terminals, breaking through the limitations of existing text-related speech recognition, enabling the terminal to more intelligently understand the characteristics of each user , usage habits, enhance exclusivity and security.

Taking the voice assistant as an example, after adding the non-text voice recognition function, it can realize the identification of the user's identity, and then only process the voice commands of the application object of the voice recognition model, which enhances security and exclusivity.

In step A-2 of FIG. 1 of the present application, it is detected whether the first voice data and the second voice data meet the model training requirements. For example, it can be detected whether the first voice data and the second voice data contain non-silent features. If No, it means that there is no human voice recorded in the first voice data and the second voice data, it does not meet the model training requirements, if yes, continue to detect whether the voices in the first voice data and the second voice data are clear, if not clear, It is useless for model training and does not meet the model training requirements.

Optionally, in step A-2 of FIG. 1 , before step A-3 is performed, voice cleaning may be further performed on the first voice data and the second voice data, and step A-3 is performed after voice cleaning. The voice cleaning includes denoising, noise reduction and other processing, which can make the first voice data and the second voice data have better quality, thereby improving the model training effect.

In step A-3 of FIG. 1 of the present application, to determine whether the voice of the first voice data comes from the application object of the voice recognition model, face recognition or fingerprint determination or output dialog box may be used to allow the user to confirm the identity. Taking face recognition as an example, the camera actively collects the user's face information, and determines whether it is the application object of the speech recognition model through comparison. If the collection fails, the user is prompted for input. Fingerprint determination, generally need to prompt the user to input the fingerprint of the designated finger, and determine whether it is the application object of the speech recognition model through comparison.

In FIG. 1 of the present application, in order to save the training time of the speech recognition model, the general feature module in the speech recognition model, especially the non-application object speech feature module, may be pre-trained.

On the other hand, in order to avoid occupying terminal system resources and leaking user privacy, immediately after step A-4 or step A-5, the first voice data and the second voice data are used to train the speech recognition model. If the first voice data and the second voice data are not updated, the first voice data and the second voice data are cleared, and the process of FIG. 1 is exited. In other words, after the training is over, clear the relevant voice data and exit the process of Figure 1.

Fig. 2 expands the method of Fig. 1, and provides an embodiment of a specific application, including the following steps:

Step A-11 (S201): when the user makes a voice call, the input voice data stream of the microphone is saved as the third voice data, and the output voice data stream of the earpiece is saved as the fourth voice data;

Step A-12 (S202): when the third voice data reaches the preset duration, make the first voice data equal to the third voice data, and simultaneously make the third voice data empty, execute step A-2, and return to step A- 11.

Step A-13 (S203): when the voice of the fourth voice data reaches the preset duration, make the second voice data equal to the fourth voice data, and simultaneously make the fourth voice data empty, execute step A-2, and return to step A-11.

Step A-2 (S204): Detect whether the first voice data and the second voice data meet the training requirements of the voice recognition model, if so, execute Step A-3;

Step A-31 (S205): utilize the speech recognition model to judge whether the first speech data comes from the application object of the speech recognition model, and output (judgment) the confidence of the result; if the confidence is less than the threshold, then execute step A-32; if If the judgment result is the application object of the speech recognition model and the confidence is greater than or equal to the threshold, then step A-4 is performed; if the judgment result is not the application object of the speech recognition model and the confidence is greater than or equal to the threshold, then step A-5 is performed;

Step A-32 (S206): in this voice call, whether the user has confirmed the identity, if not, let the user confirm whether it is the application object of the speech recognition model, and record the user confirmation result, if it is the application object of the speech recognition model , go to step A-4, if it is not the application object of the speech recognition model, go to step A-5.

Step A-4 (S207): the first voice data is marked as application object voice data, the second voice data is marked as non-application object voice data, and the application object voice data is used for the voice feature learning of the application object in the voice recognition model; The non-application object speech data is used for the speech feature learning of non-application objects in the speech recognition model;

Step A-5 (S208): Mark the first voice data and the second voice data as non-application target voice data.

The method of FIG. 1 of the present application can save all the data of a voice call as the first voice data and the second voice data, and then train the voice recognition model based on the data, or it can be set to save while training as shown in the method of FIG. 2 , Meanwhile, steps A-11 to A-13 in FIG. 2 can also be replaced with A-1 in FIG. 1 , which can be selected according to actual needs.

In steps A-12 and A-13 of FIG. 2 , the preset time is greater than 10 seconds, or greater than the time taken to perform steps A-2 to A-5 in FIG. 2 .

In step A-31 of FIG. 2 of this application, face recognition or fingerprint recognition is not used for user identity authentication, but the voice recognition model itself is used for user authentication. Considering the initial training of the voice recognition model, the judgment error is large, and the user is assisted in manual confirmation. Identity, when the recognition accuracy of the model is getting higher and higher, there is no need for manual participation. The method in Figure 2 of this application can be implemented in a "silent" way, running in the background, and continuously training the speech recognition model without the user's perception. .

The present invention also includes a system for acquiring voice data, as shown in Figure 3, including:

Save module: when the user makes a voice call, save the voice data stream transmitted in real time in the intelligent terminal system, save the input voice data stream of the microphone as the first voice data, and save the output voice data stream of the earpiece as the second voice data;

Detection module: detects whether the first voice data and the second voice data meet the training requirements of the voice recognition model, and if so, executes the user judgment module;

User judgment module: judge whether the first voice data comes from the application object of the voice recognition model, if so, execute the voice object marking module 1, if otherwise, execute the voice object marking 2;

Voice object marking 1: mark the first voice data as application object voice data, mark the second voice data as non-application object voice data, and use the application object voice data for the voice feature learning of the application object in the speech recognition model; Speech data is used for speech feature learning of non-application objects in speech recognition model;

Voice object marking 2: marking the first voice data and the second voice data as non-application object voice data.

Optionally, as shown in FIG. 4 , the recording module may include: a cyclic recording module, a transmission module 1 and a transmission module 2 .

Loop recording module: save the input voice data stream of the microphone as the third voice data, and save the output voice data stream of the earpiece as the fourth voice data;

Transfer module 1: when the third voice data reaches a preset duration, set the first voice data to be equal to the third voice data, and at the same time set the third voice data to be empty, execute the detection module, and return to the loop recording module at the same time.

Transmission module 2: when the voice of the fourth voice data reaches a preset duration, set the second voice data to be equal to the fourth voice data, and at the same time set the fourth voice data to be empty, execute the detection module, and return to the loop recording module at the same time.

Optionally, as shown in Figure 4, the user judgment module may include: a voice recognition model user judgment module and a user confirmation module,

Voice recognition model user judgment module: use the voice recognition model to judge whether the first voice data comes from the application object of the voice recognition model, and output the confidence of the result; if the confidence is less than the threshold, execute the user confirmation module; if the judgment result is voice recognition If the application object of the model and the confidence level is greater than or equal to the threshold, then perform voice object marking 1; if the judgment result is not the application object of the speech recognition model and the confidence level is greater than or equal to the threshold, then perform voice object marking 2;

User confirmation module: In this voice call, whether the user has confirmed the identity, if not, let the user confirm whether it is the application object of the speech recognition model, and record the user confirmation result, if it is the application object of the speech recognition model, execute the voice object Mark 1, if it is not the application object of the speech recognition model, execute the speech object mark 2.

Optionally, in the detection module, detecting whether the first voice data and the second voice data meet the training requirements of the voice recognition model includes:

Detect whether the first voice data and the second voice data contain non-silent features, if not, it does not meet the model training requirements, if so, continue to detect whether the voices in the first voice data and the second voice data are clear, if not. It is clear and does not meet the model training requirements.

Optionally, in the detection module, if yes, executing the user judgment module includes:

If so, the user judgment module is executed after voice cleaning is performed on the first voice data and the second voice data.

It should be noted that, the embodiment of the voice data acquisition system of the present invention has the same principle as the embodiment of the voice data acquisition method, and relevant parts can be referred to each other.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the technical solution of the present invention shall be should be included within the protection scope of the present invention.

Claims (10)

1. A method for obtaining speech data, wherein the speech data is used for training a speech recognition model, the method comprising the steps of:

step A-1: when a user carries out voice call, a voice data stream transmitted in real time in the intelligent terminal system is stored, an input voice data stream of a microphone is stored as first voice data, and an output voice data stream of a receiver is stored as second voice data;

step A-2: detecting whether the first voice data and the second voice data meet the training requirements of a voice recognition model, if so, executing the step A-3;

step A-3: judging whether the first voice data come from an application object of the voice recognition model, if so, executing the step A-4, and if not, executing the step A-5;

step A-4: marking the first voice data as application object voice data, and marking the second voice data as non-application object voice data, wherein the application object voice data is used for voice feature learning of an application object in the voice recognition model; the non-application object voice data is used for voice feature learning of a non-application object in the voice recognition model;

step A-5: and marking the first voice data and the second voice data as the non-application object voice data.

2. The method of claim 1, wherein in the step a-2, if so, performing the step a-3 comprises:

and if so, performing voice cleaning on the voice of the first voice data and the voice of the second voice data, and then executing the step A-3.

3. The method of claim 1, wherein saving an input voice data stream of a microphone as first voice data and an output voice data stream of an earpiece as second voice data comprises:

step A-11: saving the input voice data stream of the microphone as third voice data, saving the output voice data stream of the receiver as fourth voice data, and executing the step A-12 and the step A-13;

step A-12: when the third voice data reaches the preset time length, enabling the first voice data to be equal to the third voice data, enabling the third voice data to be empty, executing the step A-2, and returning to the step A-11;

step A-13: and when the voice of the fourth voice data reaches the preset duration, enabling the second voice data to be equal to the fourth voice data, enabling the fourth voice data to be empty, executing the step A-2, and returning to the step A-11.

4. The method of claim 1, wherein the step a-3 further comprises:

step A-31: judging whether the first voice data comes from an application object of the voice recognition model or not by using the voice recognition model, and outputting the confidence coefficient of a judgment result; if the confidence is less than the threshold, performing step A-32; if the judgment result is the application object of the voice recognition model and the confidence coefficient is larger than or equal to the threshold value, executing the step A-4; if the judgment result is not the application object of the speech recognition model and the confidence coefficient is larger than or equal to the threshold value, executing the step A-5;

step A-32: in the voice call, whether the user confirms the identity or not is judged, if not, the user is judged whether the user is the application object of the voice recognition model or not and the confirmation result of the user is recorded, if the user is the application object of the voice recognition model, the step A-4 is executed, and if the user is not the application object of the voice recognition model, the step A-5 is executed.

5. The method of claim 1, wherein detecting whether the first speech data and the second speech data comply with speech recognition model training requirements comprises:

and detecting whether the first voice data and the second voice data contain non-mute characteristics or not, if not, not meeting the requirement of model training, if so, continuously detecting whether the voices in the first voice data and the second voice data are clear or not, and if not, not meeting the requirement of model training.

6. A system for obtaining speech data for use in training a speech recognition model, the system comprising:

a storage module: when a user carries out voice call, a voice data stream transmitted in real time in the intelligent terminal system is stored, an input voice data stream of a microphone is stored as first voice data, and an output voice data stream of a receiver is stored as second voice data;

a detection module: detecting whether the first voice data and the second voice data meet the training requirements of a voice recognition model, if so, executing a user judgment module;

a user judgment module: judging whether the first voice data come from an application object of the voice recognition model, if so, executing a voice object marking module 1, and otherwise, executing a voice object marking module 2;

voice object flag 1: marking the first voice data as application object voice data, and marking the second voice data as non-application object voice data, wherein the application object voice data is used for voice feature learning of an application object in the voice recognition model; the non-application object voice data is used for voice feature learning of a non-application object in the voice recognition model;

voice object marker 2: and marking the first voice data and the second voice data as the non-application object voice data.

7. The system of claim 6, wherein in the detecting module, if yes, the executing the user determination module further comprises:

and if so, executing a user judgment module after carrying out voice cleaning on the first voice data and the second voice data.

8. The system of claim 6, wherein the save module further comprises:

a cycle recording module: saving the input voice data stream of the microphone as third voice data, saving the output voice data stream of the receiver as fourth voice data, and executing a transmission module 1 and a transmission module 2;

the transmission module 1: when the third voice data reaches a preset time length, enabling the first voice data to be equal to the third voice data, enabling the third voice data to be empty, executing a detection module, and returning to a circulating recording module;

the transmission module 2: and when the voice of the fourth voice data reaches the preset duration, making the second voice data equal to the fourth voice data, and making the fourth voice data empty, executing the detection module, and returning to the circulating recording module.

9. The system of claim 6, wherein the user determination module further comprises:

the voice recognition model user judgment module: judging whether the first voice data comes from an application object of the voice recognition model or not by using the voice recognition model, and outputting the confidence coefficient of a judgment result; if the confidence is less than the threshold, executing a user confirmation module; if the judgment result is the application object of the voice recognition model and the confidence coefficient is larger than or equal to a threshold value, executing a voice object mark 1; if the judgment result is not the application object of the voice recognition model and the confidence coefficient is larger than or equal to a threshold value, executing a voice object mark 2;

a user confirmation module: in the voice call, whether the user confirms the identity or not is judged, if not, the user is judged whether the user is the application object of the voice recognition model or not, the confirmation result of the user is recorded, if the user is the application object of the voice recognition model, the voice object mark 1 is executed, and if the user is not the application object of the voice recognition model, the voice object mark 2 is executed.

10. The system of claim 6, wherein the detecting module detects whether the first speech data and the second speech data conform to speech recognition model training requirements comprises:

and detecting whether the first voice data and the second voice data contain non-mute characteristics or not, if not, not meeting the requirement of model training, if so, continuously detecting whether the voices in the first voice data and the second voice data are clear or not, and if not, not meeting the requirement of model training.

Images