CN115240702B - Voice separation method based on voiceprint characteristics - Google Patents

Abstract

The present invention proposes a method for speech separation based on voiceprint features, and the implementation steps are: obtaining a training sample set and a test sample set; constructing a speech separation model based on voiceprint information; iteratively training the speech separation model; and obtaining a speech separation result. The speech separation model constructed by the present invention includes a Conv‑TasNet model. In the process of training the speech separation model and obtaining the speech separation result, the FiLM‑DCTasNet fusion algorithm is used to perform mask calculation on each pair of mixed audio coding and voiceprint feature vectors, so that the separation network can integrate the voiceprint features of the target speaker multiple times. At the same time, the separation network in the Conv‑TasNet model is used to perform mask calculation, which effectively improves the signal distortion ratio and separation efficiency of the separated audio signal.

Description

Speech separation method based on voiceprint features

Technical Field

The present invention belongs to the technical field of speech recognition and relates to a speech separation method, and in particular to a speech separation method based on voiceprint features.

Background Art

Speech separation technology is to use hardware or software methods to separate the target speaker's audio from background interference. Speech separation methods include traditional methods and deep learning methods. Compared with traditional speech separation methods, deep learning methods have the characteristics of high precision, high accuracy, and end-to-end training and testing, and are currently the mainstream speech separation methods. Deep learning methods mainly include methods based on permutation invariant training, methods based on deep attractor networks, and methods based on deep clustering. These methods usually train the separation model based on a fixed number of speakers, and can only separate the audio of all speakers in multi-person audio at the same time, but cannot separate the audio of a target speaker alone, and cannot separate high-quality multiple audios for multiple speakers with similar sound characteristics.

With the continuous improvement of voiceprint feature extraction models, the fusion of voiceprint features in speech separation models has been applied in the target speaker's speech separation method. For example, the patent application with application publication number CN 113990344A, entitled "A method, device and medium for multi-person speech separation based on voiceprint features", discloses a multi-person speech separation method based on voiceprint features, which first extracts voiceprint features from the target speaker's audio, performs short-time Fourier transform on the mixed audio to obtain the spectral features of the mixed audio, concatenates the extracted voiceprint features and spectral features, and uses the deep clustering model DPCL to generate a mask, multiplies the generated mask with the spectral features of the mixed audio to obtain the spectrum of the target speaker's pure audio, and finally obtains the target speaker's pure audio through inverse short-time Fourier transform. This method improves the accuracy of speech separation by splicing the spectral features of the mixed audio and the voiceprint features of the target speaker at the input of the speech separation network. However, it converts the mixed audio from the time domain signal to the amplitude spectrum using short-time Fourier transform, which results in the use of a noisy phase spectrum in the process of restoring the separated amplitude spectrum to the time domain signal, thereby reducing the signal distortion ratio of the separated audio. In addition, the separation model DPCL network used in this method has a complex structure and takes a long time to train.

Summary of the invention

The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a speech separation method based on voiceprint features, aiming to improve the signal distortion ratio and efficiency of separating the target speaker's audio from multi-person audio.

To achieve the above object, the technical solution adopted by the present invention includes the following steps:

(1) Obtain training sample set and test sample set:

(1a) Obtain clean audio data F = {f ₁ ,f ₂ ,…,f _n ,...,f _N } of N non-target speakers and M clean audio data K = {k ₁ ,k ₂ ,...,k _m ,...,k _M } of the target speaker, and mix each clean audio data _km of the target speaker with the clean audio data f _p , f _q of any two speakers in F and natural noise to obtain a mixed audio dataset G = {g ₁ ,g ₂ ,...,g _m ,...,g _M }, where N ≥ 200, f _n represents a clean audio data of the nth speaker, M ≥ 8000, _km represents the mth clean audio data of the target speaker, p ∈ [1, N], q ∈ [1, N], p ≠ q, and g _m represents the mixed audio data corresponding to _km ;

(1b) performing a short-time Fourier transform on each piece of pure audio data _km , and combining _km , _g and the auxiliary speech spectrum _h obtained by performing the short-time Fourier transform on _km to form sample data, thereby obtaining a set of M sample data, and then randomly selecting more than half of the sample data from the set to form a training sample set T, and the remaining sample data to form a test sample set R;

(2) Construct a speech separation model O based on voiceprint features:

Construct a speech separation model O including a voiceprint feature extraction module and a Conv-TasNet model; the Conv-TasNet model includes an audio encoding module, a speech separation network and an audio decoding module; the audio encoding module is connected to the audio decoding module, the audio encoding module and the voiceprint feature extraction module are connected to the speech separation network, and the speech separation network is connected to the audio decoding module; wherein the voiceprint feature extraction module includes a bidirectional LSTM layer, a fully connected layer and an average pooling layer connected in sequence; the audio encoding module includes a one-dimensional convolution layer; the speech separation network includes multiple normalization layers, multiple one-dimensional convolution layers, and multiple groups of time convolution modules TCN and PReLU layers cascaded in sequence; the audio decoding module includes a one-dimensional deconvolution layer;

(3) Iteratively train the speech separation model O:

(3a) Initialize the number of iterations j, the maximum number of iterations is J, J ≥ 100, the current speech separation model is O _j , and set j = 0;

(3b) Take the training sample set T as the input of the speech separation model O for forward propagation:

(3b1) the audio encoding module encodes the mixed audio data in each training sample and outputs the mixed audio code, while the voiceprint feature extraction module extracts the voiceprint features from the auxiliary speech spectrum in each training sample and outputs the voiceprint feature vector corresponding to the mixed audio code;

(3b2) The speech separation network uses the FiLM-DCTasNet fusion algorithm to perform mask calculation on each pair of mixed audio code and voiceprint feature vector to obtain the non-negative mask matrix corresponding to the target speaker audio in the mixed audio; the audio decoding module performs point multiplication of the mixed audio code and the non-negative mask matrix corresponding to the target speaker audio in the mixed audio, and then uses a one-dimensional deconvolution layer to decode, to obtain the target speaker audio data after separation of each training sample;

(3c) Calculate the scale-invariant signal-distortion ratio SI-SNR of the target speaker audio data separated from each training sample and its corresponding clean audio data, and use the Adam optimizer to update the weight of the speech separation model O by maximizing the SI-SNR value to obtain the voiceprint feature-based speech separation model O _j after the jth iteration;

(3d) Determine whether j≥J holds true. If so, obtain the trained speech separation model O' based on voiceprint features. Otherwise, set j=j+1 and execute step (3b);

(4) Obtain speech separation results:

The test sample set R is used as the input of the trained voiceprint feature-based speech separation model O' for forward propagation to obtain the separated target speaker audio data corresponding to the test sample set R.

Compared with the prior art, the present invention has the following advantages:

1. In the process of training the speech separation model and obtaining the speech separation results, the present invention adopts the FiLM-DCTasNet fusion algorithm to perform mask calculation on each pair of mixed audio codes and voiceprint feature vectors, so that the separation network can integrate the voiceprint features of the target speaker multiple times, which solves the problem in the prior art that the separated audio contains audio of other speakers with similar sound features to the target speaker caused by directly fusing voiceprint features by dot product, and effectively improves the signal-distortion ratio of the separated audio signal.

2. The present invention uses the separation network in the Conv-TasNet model to perform mask calculation in the speech separation model. The separation network is a deep convolutional neural network. Compared with the model constructed using a recurrent neural network in the prior art, it has a simpler model structure, faster training speed and better information abstraction ability, which shortens the training time of the speech separation model and increases the distortion ratio of the target speaker audio signal separated by the model.

3. The present invention adopts time domain coding in the selection of codec, that is, a one-dimensional convolutional layer is used to directly encode the audio time domain signal. Compared with time-frequency coding, this method has the characteristics of shorter frame length, trainable parameters, and no phase reconstruction. It avoids the signal loss caused by the conversion between time domain and time-frequency domain in the process of signal encoding in the prior art, and makes the separated audio signal have a higher distortion ratio.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is an implementation flow chart of the present invention;

FIG2 is a schematic diagram of the structure of the speech separation model based on voiceprint features constructed by the present invention.

DETAILED DESCRIPTION

The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Referring to Figure 1, the present invention comprises the following steps:

Step 1) Obtain training sample set and test sample set:

(1a) Obtain clean audio data F = {f ₁ ,f ₂ ,...,f _n ,...,f _N } of N non-target speakers and M clean audio data K = {k ₁ ,k ₂ ,...,k _m ,...,k _M } of the target speaker, and mix each clean audio data _km of the target speaker with the clean audio data f _p , f _q of any two speakers in F and natural noise to obtain a mixed audio dataset G = {g ₁ ,g ₂ ,...,g _m ,...,g _M }, where N ≥ 200, f _n represents a clean audio data of the nth speaker, M ≥ 8000, _km represents the mth clean audio data of the target speaker, p ∈ [1, N], q ∈ [1, N], p ≠ q, and g _m represents the mixed audio data corresponding to _km ;

The method of mixing the target speaker's pure audio with other speakers' audio and noise can make the pure audio the target audio separated by the speech separation model, so that the audio separated by the trained speech separation model is closer to the real audio. In this embodiment, N=300, M=10000;

(1b) performing a short-time Fourier transform on each piece of pure audio data _km , and combining _km , _gm, and the auxiliary speech feature _hm obtained by performing the short-time Fourier transform on _km to form sample data, thereby obtaining a set of M sample data, and then randomly selecting more than half of the sample data from the set to form a training sample set T, and the remaining sample data to form a test sample set R;

Different from the existing speech separation algorithm, in which the mixed audio is first subjected to short-time Fourier transform to extract the spectrum and then input into the speech separation model, the present invention directly inputs the time domain signal of the mixed audio into the speech separation model, which can avoid the addition of noise caused by the conversion of the time domain and the time-frequency domain, so that the audio obtained by the speech separation model has less noise. In this embodiment, the number of training samples selected is 8000 and the number of test samples is 2000;

Step 2) construct a speech separation model O based on voiceprint features, whose structure is shown in Figure 2;

Construct a speech separation model O including a voiceprint feature extraction module and a Conv-TasNet model; the Conv-TasNet model includes an audio encoding module, a speech separation network and an audio decoding module; the audio encoding module is connected to the audio decoding module, the audio encoding module and the voiceprint feature extraction module are connected to the speech separation network, and the speech separation network is connected to the audio decoding module; wherein the voiceprint feature extraction module includes a bidirectional LSTM layer, a fully connected layer and an average pooling layer connected in sequence; the audio encoding module includes a one-dimensional convolution layer; the speech separation network includes multiple normalization layers, multiple one-dimensional convolution layers, and multiple groups of time convolution modules TCN and PReLU layers cascaded in sequence; the audio decoding module includes a one-dimensional deconvolution layer;

The speech separation network includes a first normalization layer, a first one-dimensional convolution layer, a first TCN group, a second TCN group, a third TCN group, a PReLU layer, a second one-dimensional convolution layer and a second normalization layer, wherein each group of TCN includes eight TCNs cascaded in sequence, and the specific structure of each TCN is: a one-dimensional convolution layer, a ReLU layer, a normalization layer, a depth-separable convolution layer, a ReLU layer, a normalization layer and a one-dimensional convolution layer, and a residual connection is performed between the input and the output; each TCN has its own dilated convolution structure, wherein the dilation factor of the convolution process, i.e., the step size of the convolution kernel, starts from 1 and gradually increases with an exponential of 2, i.e., 1, 2, 4, ..., and the dilation factor of the first TCN in each group of TCN is reset to 1;

The TCN module used in the present invention is a new type of algorithm proposed by Shaojie Bai et al. in 2018 that can be used to solve time series prediction. Since the RNN network only reads and parses one word or character in the input text at a time, the deep neural network must wait until the previous word is processed before processing the next word. This means that the RNN cannot perform large-scale parallel processing like the convolutional neural network, which takes a long time. Therefore, compared with the RNN network, the TCN module performs better in the prediction task of using time series data. Using the Conv-TasNet model based on the TCN module in the separation model can improve the separation efficiency and the signal distortion ratio of the separated audio.

Step 3) Iteratively train the speech separation model O:

(3a) Initialize the number of iterations j, the maximum number of iterations is J, J ≥ 100, the current speech separation model is O _j , and set j = 0, in this embodiment, J = 100;

(3b) Take the training sample set T as the input of the speech separation model O for forward propagation:

(3b1) the audio encoding module encodes the mixed audio data in each training sample and outputs the mixed audio code, while the voiceprint feature extraction module extracts the voiceprint feature from the auxiliary speech feature in each training sample and outputs the voiceprint feature vector;

(3b2) The speech separation network performs mask calculation on the mixed audio code and voiceprint feature vector of each training sample to obtain a non-negative mask matrix corresponding to the target speaker audio in the mixed audio; the audio decoding module performs a dot multiplication of the mixed audio code and the non-negative mask matrix corresponding to the target speaker audio in the mixed audio, and then uses a one-dimensional deconvolution layer to decode, to obtain the target speaker audio data after separation of each training sample;

The implementation steps of the speech separation network for mask calculation of the mixed audio code and voiceprint feature vector of each training sample are as follows: the mixed audio code is sequentially input into the first normalization layer and the first one-dimensional convolution layer for forward propagation to obtain the intermediate representation _L0 of the separation network; the result of _L0 and the voiceprint feature vector after the affine transformation of the FiLM layer is input into the first TCN group for forward propagation to obtain the intermediate representation _L1 of the separation network; the result of _L1 and the voiceprint feature vector after the affine transformation of the FiLM layer is input into the second TCN group for forward propagation to obtain the intermediate representation _L2 of the separation network; the result of _L2 and the voiceprint feature vector after the affine transformation of the FiLM layer is input into the third TCN group for forward propagation to obtain the output of the last TCN module of the third TCN group; the output is summed with the outputs of all other TCNs and then input into the PReLU layer, the second one-dimensional convolution layer and the second normalization layer for forward propagation to obtain the non-negative mask matrix corresponding to the audio of the target speaker in the mixed audio; wherein _Li,c and the voiceprint feature vector after the affine transformation of the FiLM layer are calculated according to the following formula:

γ _i,c ＝f _c (e)

β _i,c = h _c (e)

FiLM(L _i,c |γ _i,c ,β _i,c )=γ _i,c L _i,c +β _i,c

Where e is the voiceprint feature vector extracted from the clean speech of the target speaker, f _c , h _c are the cth affine transformation functions of e, and Li _,c is the cth feature of the intermediate representation before the i-th TCN group fuses the voiceprint feature vector;

Compared with most speech separation algorithms that directly fuse voiceprint features by dot product, the present invention uses the FiLM-DCTasNet fusion algorithm to fuse voiceprint features in the separation network, and constructs a set of trainable parameters for each fusion process, which can better represent the voiceprint features of the target speaker in the non-negative mask matrix corresponding to the target speaker's audio, thereby improving the signal distortion ratio of the target speaker's audio obtained by the speech separation model;

(3c) Calculate the scale-invariant signal-distortion ratio (SI-SNR) of the target speaker audio data separated from each training sample and its corresponding clean audio data, and use the Adam optimizer to update the weight of the speech separation model O by maximizing the SI-SNR value to obtain the speech separation model O _j based on voiceprint features after the jth iteration;

The calculation method of SI-SNR is defined as follows:

Where S is the pure audio data, The separated target speaker audio data;

The performance evaluation index SI-SNR used in the present invention projects the generated vector to the vertical direction of the real vector. Compared with the signal distortion rate SNR used in the traditional method, it can more directly reflect the similarity between the separated target speaker audio data and the pure audio data, thereby obtaining a better speech separation model.

(3d) Determine whether j≥J holds true. If so, obtain the trained voiceprint feature-based speech separation model O'. Otherwise, set j=j+1 and execute step (3b)

Step 4) Get the speech separation result:

The test sample set R is used as the input of the trained voiceprint feature-based speech separation model O' for forward propagation to obtain the separated target speaker audio data corresponding to the test sample set R.

Claims (4)

1. A speech separation method based on voiceprint features, characterized in that it comprises the following steps: (1) Obtain training sample set and test sample set: (1a) Obtain clean audio data F = {f ₁ ,f ₂ ,...,f _n ,...,f _N } of N non-target speakers and M clean audio data K = {k ₁ ,k ₂ ,...,k _m ,...,k _M } of the target speaker, and mix each clean audio data _km of the target speaker with the clean audio data f _p , f _q of any two speakers in F and natural noise to obtain a mixed audio dataset G = {g ₁ ,g ₂ ,...,g _m ,...,g _M }, where N ≥ 200, f _n represents a clean audio data of the nth speaker, M ≥ 8000, _km represents the mth clean audio data of the target speaker, p ∈ [1, N], q ∈ [1, N], p ≠ q, and g _m represents the mixed audio data corresponding to _km ; (1b) performing a short-time Fourier transform on each piece of pure audio data _km , and combining _km , _g and the auxiliary speech spectrum _h obtained by performing the short-time Fourier transform on _km to form sample data, thereby obtaining a set of M sample data, and then randomly selecting more than half of the sample data from the set to form a training sample set T, and the remaining sample data to form a test sample set R; (2) Construct a speech separation model O based on voiceprint features: Construct a speech separation model O including a voiceprint feature extraction module and a Conv-TasNet model; the Conv-TasNet model includes an audio encoding module, a speech separation network and an audio decoding module; the audio encoding module is connected to the audio decoding module, the audio encoding module and the voiceprint feature extraction module are connected to the speech separation network, and the speech separation network is connected to the audio decoding module; wherein the voiceprint feature extraction module includes a bidirectional LSTM layer, a fully connected layer and an average pooling layer connected in sequence; the audio encoding module includes a one-dimensional convolutional layer; the speech separation network includes multiple normalization layers, multiple one-dimensional convolutional layers, and multiple groups of time convolution modules TCN and PReLU layers cascaded in sequence; the audio decoding module includes a one-dimensional deconvolutional layer; the speech separation network includes 2 normalization layers, 2 one-dimensional convolutional layers, and 3 TCN groups, and the specific structure of the speech separation network is: a first normalization layer, a first one-dimensional convolutional layer, a first TCN group, a second TCN group, a third TCN group, a PReLU layer, a second one-dimensional convolutional layer and a second normalization layer; (3) Iteratively train the speech separation model O: (3a) Initialize the number of iterations j, the maximum number of iterations is J, J ≥ 100, the current speech separation model is O _j , and set j = 0; (3b) Take the training sample set T as the input of the speech separation model O for forward propagation: (3b1) the audio encoding module encodes the mixed audio data in each training sample and outputs the mixed audio code, while the voiceprint feature extraction module extracts the voiceprint features from the auxiliary speech spectrum in each training sample and outputs the voiceprint feature vector corresponding to the mixed audio code; (3b2) The speech separation network uses the FiLM-DCTasNet fusion algorithm to perform mask calculation on each pair of mixed audio code and voiceprint feature vector to obtain the non-negative mask matrix corresponding to the target speaker audio in the mixed audio; the audio decoding module performs point multiplication of the mixed audio code and the non-negative mask matrix corresponding to the target speaker audio in the mixed audio, and then uses a one-dimensional deconvolution layer to decode, to obtain the target speaker audio data after separation of each training sample; The speech separation network uses the FiLM-DCTasNet fusion algorithm to perform mask calculation on each pair of mixed audio codes and voiceprint feature vectors. The implementation steps are as follows: the mixed audio code is sequentially input into the first normalization layer and the first one-dimensional convolution layer for forward propagation to obtain the intermediate representation _L0 of the separation network. The result of _L0 and the voiceprint feature vector after the affine transformation of the FiLM layer is input into the first TCN group for forward propagation to obtain the intermediate representation _L1 of the separation network. The result of _L1 and the voiceprint feature vector after the affine transformation of the FiLM layer is input into the second TCN group for forward propagation to obtain the intermediate representation _L2 of the separation network. The result of _L2 and the voiceprint feature vector after the affine transformation of the FiLM layer is input into the third TCN group for forward propagation to obtain the output of the last TCN module. The output is summed with the outputs of all other TCNs and then input into the PReLU layer, the second one-dimensional convolution layer, and the second normalization layer for forward propagation to obtain the non-negative mask matrix corresponding to the audio of the target speaker in the mixed audio; wherein _Li,c and the voiceprint feature vector after the affine transformation of the FiLM layer are calculated according to the following formula: γ _i,c ＝f _c (e) β _i,c = h _c (e) FiLM(L _i,c |γ _i,c ,β _i,c )=γ _i,c L _i,c +β _i,c Where e is the voiceprint feature vector extracted from the clean speech of the target speaker, f _c , h _c are the cth affine transformation functions of e, and Li _,c is the cth feature of the intermediate representation before the i-th TCN group fuses the voiceprint feature vector; (3c) Calculate the scale-invariant signal-distortion ratio SI-SNR of the target speaker audio data separated from each training sample and its corresponding clean audio data, and use the Adam optimizer to update the weight of the speech separation model O by maximizing the SI-SNR value to obtain the voiceprint feature-based speech separation model O _j after the jth iteration; (3d) Determine whether j≥J holds true. If so, obtain the trained speech separation model O' based on voiceprint features. Otherwise, set j=j+1 and execute step (3b); (4) Obtain speech separation results: The test sample set R is used as the input of the trained voiceprint feature-based speech separation model O' for forward propagation to obtain the separated target speaker audio data corresponding to the test sample set R. 2. According to the voiceprint feature-based speech separation method of claim 1, it is characterized in that the speech separation network described in step (2), wherein each group of TCN includes eight TCNs cascaded in sequence, and the specific structure of each TCN is: a one-dimensional convolution layer, a ReLU layer, a normalization layer, a depth-separable convolution layer, a ReLU layer, a normalization layer and a one-dimensional convolution layer, and a residual connection is performed between the input and the output. 3. According to the voiceprint feature-based speech separation method of claim 2, it is characterized in that in step (3), a time convolution module TCN is used to separate the target speaker's voice from the mixed audio, and each TCN has its own dilated convolution structure, wherein the dilation factor of the convolution process, i.e., the step size of the convolution kernel, starts from 1 and gradually increases with an exponential of 2, i.e., 1, 2, 4, ..., and the dilation factor of the first TCN in each group of TCNs is reset to 1. 4. The method for speech separation based on voiceprint features according to claim 1, characterized in that in step (3c), a scale-invariant signal-distortion ratio SI-SNR of each pair of separated target speaker audio data and pure audio data is calculated, and the calculation method of the SI-SNR is defined according to the following formula: Where S is the pure audio data, It is the separated target speaker audio data.