CN115691539A - Two-stage voice separation method and system based on visual guidance - Google Patents

Abstract

The invention provides a two-stage voice separation method and a two-stage voice separation system based on visual guidance, wherein in the first stage, the voices of speakers are obtained from the obtained mixed voice in a time domain; in the second stage, the independent voice features with the speaker information are extracted by means of the roughly separated voice in the first stage, and meanwhile, potential relevant features and complementary features between visual and audio modes are mined and multi-mode features are fused and separated, and finally pure target voice is obtained. The invention extracts the independent voice characteristics of the speaker by utilizing the first stage, avoids introducing pure reference voice, improves the voice separation performance and robustness through visual guidance, and solves the problem of label arrangement. The invention further improves the voice separation quality by dynamically adjusting the weight of the two-stage model, and the disclosed voice separation system is suitable for most application scenes.

Description

Two-stage speech separation method and system based on visual guidance

technical field

The invention belongs to the technical field of speech separation, and relates to a two-stage speech separation method and system based on visual guidance.

Background technique

The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

Speech separation refers to extracting one or more target speech signals from the mixed speech produced by multiple speakers. The speech separation problem comes from the "cocktail party effect", which describes a phenomenon where many different types of sound sources are present simultaneously in a noisy indoor environment, such as at a cocktail party. But even then, people can focus on one person's conversation while ignoring other conversations or noises in the background. This is a human ability to select hearing. We hope to have the ability to select and filter voices through machine learning. Speech separation has a wide range of applications and is a basic and important part of many downstream speech tasks. Only by separating high-quality pure speech can it be better applied in speech recognition [TORFIA, IRANMANESH S M, NASRABADI N, et al.3D Convolutional Neural Networks for CrossAudio-Visual Matching Recognition[J].IEEE Access,2017][T.AFOURAS J S C,A.SENIOR,O.VINYALS,AND A.ZISSERMAN.Deep Audio-Visual Speech Recognition[J].IEEE Transactions on Pattern Analysis&Machine Intelligence, 2018], human-computer interaction and other scenarios.

Traditional speech separation algorithms include Computational Auditory Scene Analysis (CASA, Computational Auditory Scene Analysis), non-negative matrix factorization, and hidden Markov models, etc., usually require certain assumptions and prior knowledge [JOSH H, MCDERMOTT.The cocktail party problem [J].Current Biology,2009], there is also the problem of poor effect when separating similar speaker voices [Rivet B,Wang W,Naqvi S M,etal.Audiovisual speech source separation:An overview of key methodologies[J] . IEEE Signal Processing Magazine, 2014, 31(3): 125-134.]. With the development of deep learning, the neural network has also achieved good results in the field of speech separation, and the neural network can learn the complex mapping relationship between the mixed speech signal and the target speech signal. The common audio-only method has the problem of label arrangement when training the separation network. Although the correct match can be screened out through permutation and combination—Permutation Invariant Training (PIT, Permutation Invariant Training), the calculation is more complicated; in addition , speech is greatly affected by the environment and noise, resulting in poor robustness of the separation system.

In real-world scenarios, people will assist their auditory perception by observing the speaker. When seeing the speaker's face or lips, hearing is easier. Secondly, in some fields of psychology, physiology, and psychiatry, there are also studies [Golumbic E Z, Cogan G B, Schroeder C E, et al.Visual input enhances selective speech envelope tracking in auditory cortex at a'cocktail party'[J].Journalof Neuroscience, 2013,33(4):1417-1426.] Prove that visual information is helpful for people to understand speech. Compared with speech, the speaker's visual information such as lip movement and facial appearance is more stable, and visual information has identity features, which can match the correct speaker label in the process of separating mixed speech. In the separation model proposed by Ephrat in 2017, only static images were introduced. Although the data dimension is reduced, the information on the visual timing is lost, and the separation performance is lost.

Some studies[AFOURAS T C J S, ZISSERMAN A.My lips are concealed Audio-visual speech enhancement through obstructions[J].Interspeech,2019,4295-9.][OCHIAI T,DELCROIX M,KINOSHITA K,et al.Multimodal SpeakerBeam:Single ChannelTarget Speech Extraction with Audio-Visual Speaker Clues[M].Interspeech2019.2019:2718-22.][R.Gu,S.-X.Zhang,Y.Xu,L.Chen,Y.Zou,and D.Yu , "Multi-modalmulti-channel target speech separation," IEEE Journal of Selected Topics in Signal Processing, 2020] proposed to use the speaker's additional reference speech extraction features to improve the separation effect. Wang et al. used the x-vector for speaker recognition in the separation model, and Luo et al. introduced the i-vector of the speaker's pure speech to assist separation. However, this method has two disadvantages. One is that the pure reference voice of the speaker needs to be input in advance to train the separation model; Speech can only be separated, so there are great limitations in real application scenarios.

Contents of the invention

In order to solve the above problems, the present invention proposes a two-stage speech separation method and system based on visual guidance. The present invention extracts the speaker's identity-distinguishing independent speech features through the first stage of speech separation to assist speech separation. The performance of speech separation can be improved through the feature extraction and fusion of visual and audio modal information to assist speech separation.

According to some embodiments, the present invention adopts the following technical solutions:

A two-stage speech separation method based on vision guidance, comprising the following steps:

In the first stage, the acquired mixed speech is separated in the time domain to obtain the roughly separated speaker speech;

In the second stage, with the help of the pure audio separation results of the first stage, independent speech features with speaker information are extracted, and then the potential correlation features and complementary features between the two modalities of vision and audio are mined, and the visual features and speech time-frequency The two modalities of domain features are fused and then separated, and the weights of the two stages are dynamically adjusted to finally obtain the pure target speech.

As an optional implementation, the first stage, the specific process includes:

Encoding the obtained mixed speech by using an encoder to extract the mixed speech features;

The mixed speech features are separated to obtain the mask of the target speech, the target speech features are determined, and the target speech features are decoded to obtain the coarsely separated target speech time domain signal.

As a further limitation, the specific process of separating the mixed speech features includes using the first separation network to process the mixed speech features. The first separation network is a temporal convolutional network structure, including a normalization layer, and multiple identical The stack module, where each stack module is composed of full convolution, expansion convolution and residual module, the output of the last stack module passes through the convolution layer and the PReLU activation layer to obtain the separated target mask.

As a further limitation, the target speech feature is calculated by multiplying the mask of the mixed speech and the target speech.

As an optional implementation, the second stage, the specific process includes:

Transform the mixed speech to obtain the complex spectrum map of the mixed speech, and obtain the complex spectral mask of the real pure speech according to it;

The time-domain signal of the target speech obtained in the first stage is converted to obtain the complex spectrogram of each speaker after separation; the complex spectrogram is extracted through the independent speech feature extraction network ResNet-18 to extract the independent speech features of each speaker.

Obtain and preprocess the visual information of the speaker synchronized with the mixed speech time, and extract static visual features and dynamic visual features from the preprocessed visual images, in which the static visual features contain distinguishing speaker identity information, It is similar to sound features such as timbre; dynamic visual features contain the content information of speech, and is similar to sound features such as phonemes. At the same time, combining these two visual features can obtain more speech while processing less dimensional information relevant features;

The mixed speech feature extraction network extracts speech features from the time-frequency domain information of the mixed speech, performs multi-modal feature fusion, separates the multi-modal features from the separation network 2, obtains the mask of the separated target speech, and uses the mask After multiplication with the complex spectrogram of the mixed speech, inverse transformation is performed, and the time-domain speech signal of the target speaker is obtained through joint training and dynamic optimization of weights separated by two stages.

As a further limitation, the specific process of transforming the mixed speech to obtain the complex spectrogram of the mixed speech includes performing short-time Fourier transform on the mixed speech signal, and then calculating the real part and the imaginary part respectively to obtain the complex spectrogram, the complex spectrogram Contains the amplitude and phase information of the speech.

As a further limitation, the time-frequency domain conversion is performed on the coarsely separated speech in the first stage to obtain the complex spectrogram; then the independent speech feature extraction network ResNet-18 is used to extract independent speech features from the complex spectrogram of each speaker; then the independent speech The features are time-dimension transformed to achieve dimensional consistency of both audio and video modal features.

As a further limitation, the specific process of obtaining and preprocessing the speaker's visual information synchronized with the time of the mixed voice includes reading the video file, intercepting the video with a set length to obtain a multi-frame image sequence, and randomly selecting a frame of facial image as the static visual information; then crop the image sequence of each frame, select the lip area with a set size to reduce the data dimension, and generate a file of the lip sequence as dynamic visual information.

As a further limitation, the specific process of extracting visual features includes normalization and data filling of lip images, and the preprocessed lip data sequentially passes through a dynamic visual feature extraction network, including a 3D convolutional layer, ShuffleNet v2, and The temporal convolutional network is used to extract time series features, which can more fit the content information of the speech, and finally obtain the lip features;

Standardize and size the face image, extract features containing speaker identity information through the static visual feature extraction network ResNet-18, and convert the facial features in the time dimension to ensure that the converted lip sequence features have the same time dimension .

As a further limitation, the specific process of multimodal feature fusion includes first mixing the sound complex spectrogram through the mixed voice feature extraction network to obtain the mixed voice features, and then cascading the speaker's visual features, independent voice features and mixed voice features. Speech features are spliced, and finally the fused multimodal features are obtained.

As a further limitation, the multimodal feature is separated by using a second separation network, and the second separation network is an upsampling network layer of U-Net.

As a further limitation, the weights of the loss function of the two-stage separation network are dynamically adjusted to maximize the use of the independent speech features of the first stage to assist the separation of the second stage.

A two-stage speech separation system based on vision guidance, including:

The first separation module is configured to, in the first stage, separate the acquired mixed speech in the time domain to obtain roughly separated speaker speech;

The second separation module is configured to, in the second stage, use the pure audio separation result of the first stage to extract sound features with speaker information, and simultaneously mine potential correlation features and complementary features between the two modalities of vision and audio, Carry out the fusion of visual features and speech time-frequency domain features before separation, and dynamically adjust the weights of the two stages to finally obtain the separated target speech;

The dynamic adjustment weight module dynamically adjusts its weight according to the performance of the separation model in the two stages, so as to maximize the use of the independent speech features extracted in the first stage to assist the second stage and achieve pure target speaker voice separation.

Compared with prior art, the beneficial effect of the present invention is:

(1) The two-stage speech separation scheme based on visual guidance proposed by the present invention can extract the speech features of a single speaker to assist speech separation in the case of only mixed speech, avoiding the introduction of additional pure reference speech.

(2) The present invention simultaneously utilizes dynamic visual features containing voice content information and static visual features containing identity information to mine the potential correlation and complementarity between the two modalities of vision and audio, and simultaneously solves the problem of pure audio speech separation. The label permutation problem avoids the computational complexity of the loss function, and at the same time improves the separation effect and the robustness of the separation system.

(3) The present invention proposes a method for dynamically adjusting loss function weights for two-stage speech separation. In the case of optimizing two training objectives at the same time, the independent speech features of the first stage of speech extraction are used to the greatest extent to assist the separation of the second stage, and finally pure speech with high performance indicators is obtained.

Description of drawings

The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, and the schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention.

FIG. 1 is a flowchart of a two-stage speech separation method based on visual guidance in the present invention.

FIG. 2 is a flowchart of a method for extracting visual features of the present invention.

FIG. 3 is a flowchart of a method for dynamically adjusting weight coefficients of a loss function according to the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the present invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It should be noted that the terminology used here is only for describing specific embodiments, and is not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and/or combinations thereof.

The present invention proposes a two-stage speech separation method and system based on visual guidance. The speaker's speech features are extracted through the first-stage rough separation speech to assist speech separation, and the feature extraction and fusion of multi-modal information is used to assist speech separation. separate.

The specific process includes:

1. The first stage of pure audio time domain separation

1. Get mixed voice. Randomly select and read the pure voice wav files of two speakers (take two-person mixed voice separation as an example), intercept a fixed length (take 2.55s as an example), and sample the read time-domain voice signal, the sampling rate It is 16kHz, and normalized, respectively recorded as x _A , x _B , adding the two pure voices to obtain the mixed voice x _mix = x _A + x _B .

2. The mixed voice x _mix passes through the encoder to extract the mixed voice features. The encoder uses a layer of 1D convolutional network.

3. The mixed speech feature passes through the separation network 1 to obtain the mask of the separated target speech. The separation network 1 uses a temporal convolutional network (TCN, Temporal Convolutional Network) structure. TCN includes a normalization layer (group normalization and one-dimensional convolution in sequence), n identical stack modules, where each stack module consists of full convolution, dilated convolution and residual modules. The output of the last stack module passes through the convolution layer and the PReLU activation layer to obtain the separated target mask.

4. Multiply the mixed speech x _mix by the mask of the target speech respectively to obtain the target speech features of the corresponding speaker.

5. Pass the above obtained target speech features through the decoder to obtain the time domain signal of the target speech, denoted as x' _A , x' _B . The decoder uses a layer of 1D transposed convolutional network.

2. The second stage of multi-modal time-frequency domain separation

6. Obtain the time-frequency domain information of the mixed speech—complex spectrogram (cIRM, complex Ideal Ratio Mask), denoted as S _mix . The complex spectrogram is an ideal ratio mask in the complex domain. It can represent the three-dimensional information of time, frequency and energy in a two-dimensional plane. The real and imaginary parts avoid the degradation of separation quality caused by the loss of phase information in the pure amplitude spectrum. First, the short-time Fourier transform (STFT, short-time Fourier transform) is performed on the mixed speech signal, and then the real part and the imaginary part are calculated respectively to obtain the complex spectrogram. Among them, the STFT transformation is shown in formula (1).

Where x(m) is the input speech signal and w(m) is the window function. X(n,ω) is a two-dimensional function of time n and frequency ω. In the present invention, the length window_size of the window function can be set to 400, the number of audio samples hop_size between adjacent STFT columns is 160, that is, the number of overlapping audio samples between adjacent windows is 160, and the windowed signal is filled with zeros The length of n_fft is 512, and the size of the final complex spectrogram is 2*F*T, that is, 2*257*256, where 2 represents the two dimensions of the real part and the imaginary part, and F and T represent the frequency and time dimensions, respectively.

7. The pure speech in the time domain is transformed through STFT (as shown in formula (1)) to obtain the corresponding complex spectrogram, and the complex spectral masks M _A and M _B of the real pure speech are obtained according to the complex spectrogram of the mixed speech.

Among them, Y _r and Y _i represent the real part and imaginary part of the mixed speech complex spectrogram, respectively, and S _r and S _i represent the real part and imaginary part of the pure speech complex spectrogram, respectively. According to the ideal complex mask and mixed speech, pure speech can be obtained, the formula is as follows:

S=M*Y (3)

Among them, * represents the multiplication of negative numbers.

8. Extract the independent speech features of the time domain signal output by the first stage. Firstly, the time-frequency domain conversion is performed on the separated speech x' _A , x' _B in the first stage, and the separated complex spectrograms S' _A , S' of each speaker are obtained through STFT transformation (as shown in formula (1) _B , and then obtain the speech features of each speaker through the ResNet-18 network, which are recorded as α _A and α _B , and the dimension is 128*1. The speech features extracted here come from the separated speech, so to a certain extent, it represents the identity characteristics of the speaker and can provide effective identity information for the second stage of separation.

9. Obtain and preprocess the speaker's visual information time-synchronized with the mixed speech. First read the video file and intercept the length of 2.55s. The video sampling rate is 75 frames per second, so a 64-frame image sequence can be obtained. Since the speaker's face does not change much except for the lip area within a range of speech, in order to reduce the complexity of data processing while retaining the identity of the speaker, a frame of facial image is randomly selected as the static vision information; then crop the 64-frame image sequence, select the lip area with a size of 88*88, and generate the h5 file of the lip sequence as dynamic visual information.

10. Extract static visual features containing identity information and dynamic visual features containing voice content information respectively. Firstly, the lip image is normalized and data filled to improve the accuracy and stability of the feature extraction model. The preprocessed lip data passes through a 3D convolutional layer, ShuffleNet v2 network, and then TCN to extract time series features. The final lip feature dimension is 512*1*64, which is recorded as f _{lip_A} and f _{lip_B} . The facial images are then normalized and resized to speed up the convergence of the feature extraction model. Since the facial image is a color image with 3 channels, the size of the preprocessed facial data is 3*224*224, and a feature with a dimension of 128*1 is extracted through a residual network ResNet-18. In order to fuse the lip features and facial features, the facial features need to be copied in the time dimension to ensure that the converted lip sequence features have the same time dimension, which is 128*1*64, recorded as f _{face_A} , f _{face_B} .

11. Acquire the speech features of the mixed speech and perform multi-modal feature fusion. Firstly, the mixed sound complex spectrogram S _mix is passed through the U-Net down-sampling network layer to obtain the mixed voice feature mix, with a dimension of 512*1*64. Then convert the speech features α _A , α _B extracted from the separated speech in the time dimension to keep the same dimension as the visual feature, and convert the speaker’s visual features f _{lip_A} , f _{lip_B} , f _{face_A} , f _{face_B} and Sound features α _A , α _B , and α _mix are spliced, and finally the fused multimodal feature dimension is 2048*1*64.

12. The above multi-modal features are passed through the separation network 2 to obtain the masks M” _A and M” _B of the separated target speech. Separation Network 2 consists of the upsampling network layers of U-Net.

13. Multiply the target masks M” _A , M” _B and the mixed speech complex spectrogram S _mix respectively to obtain the speaker’s complex spectrogram S” _A , S” _B after the second stage separation. Perform inverse short-time Fourier transform (iSTFT) on S” _A , S” _B to obtain the time-domain speech signal x” _A , x” _B of the target speaker. The calculation formula is as follows:

S” _A = M” _A *S _mix (4)

S” _B = M” _B *S _mix (5)

x” _A = iSTFT(S” _A ) (6)

x” _B = iSTFT(S” _B ) (7)

3. Dynamically adjust the weight of the two-stage loss function

In the present invention, the two-stage separation process and the simultaneous training of each network module achieve the optimization goal. The loss function that defines the overall network architecture is as follows:

loss＝λ ₁ loss ₁ +λ ₂ loss ₂ (8)

loss ₂ ＝||M _A -M' _A ||+||M _B -M' _B || (10)

x_noise定义为s’-x_target，其中s’表示分离的语音信号，s表示纯净语音信号。Among them, loss ₁ and loss ₂ are the training loss functions of the two stages respectively, and λ ₁ and λ ₂ are the training weights of the two loss functions respectively. For loss ₁ , x _target is defined as

x _noise is defined as s'-x _target , where s' represents the separated speech signal and s represents the pure speech signal.

In order to obtain effective speaker speech features from the separated speech in the first stage and further improve the separation effect in the second stage, the present invention proposes a method for dynamically adjusting the weight of the two-stage loss function. Since in the initial state of training, the quality of speech separated in the first stage is relatively poor, and the quality of the corresponding extracted speaker speech features is also relatively poor, so the weight of the loss function of the two stages is set to λ ₁ = λ ₂ at this time = 1, train and optimize two-stage separated networks simultaneously. As the quality of the separated speech provided by the first stage improves, the corresponding speech features become more discriminative, so the separation effect of the second stage can be significantly improved at this time. When the separation effect of the two stages reaches a threshold relationship, set the weights to λ ₁ =1 and λ ₂ =2 respectively, and focus on training the separation network in the second stage. The threshold relationship at this time can be judged by the separation loss of the two stages. Assuming that the loss of the single-stage pure audio speech separation is the negative value of the source distortion ratio (SDR, Source to Distortion Ratio), defined as e ₀ , the loss of the first stage of speech separation is the negative value of the SDR of the separated speech, defined as e ₁ , the loss of the speech separation in the second stage is the negative value of the SDR of the speech separated in the second stage, defined as e ₂ . When e ₁ -e ₀ <e ₁ -e ₂ , it indicates that the speech separated in the first stage improves the separation effect of the second stage, and the speech features at this time are effective for the second stage, so the weight distribution adjustment is performed. The definition of SDR is as follows:

Among them, s _target , e _interf , e _noise and e _artif represent target speaker's voice, interference generated by other speakers, noise interference and other interference generated by human processing, respectively.

Embodiment two

A two-stage speech separation system based on vision guidance, including:

The first separation module is configured to, in the first stage, separate the acquired mixed speech in the time domain to obtain independent speaker speech;

The second separation module is configured to, in the second stage, extract independent speech features with speaker information with the help of the audio-only separation results of the first stage, and simultaneously mine potential correlation features and complementary features between the two modalities of visual and audio , performing the fusion of visual features and voice time-frequency domain features and then separating them, and finally obtaining the separated target voice;

The dynamic adjustment weight module dynamically adjusts its weight according to the performance of the separation model in the two stages, so as to maximize the use of the independent speech features extracted in the first stage to assist the second stage and achieve pure target speaker voice separation.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Although the specific implementation of the present invention has been described above in conjunction with the accompanying drawings, it does not limit the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, those skilled in the art do not need to pay creative work Various modifications or variations that can be made are still within the protection scope of the present invention.

Claims (10)

1. A two-stage voice separation method based on visual guidance is characterized by comprising the following steps:

in the first stage, separating the obtained mixed voice on a time domain to obtain roughly separated speaker voice;

in the second stage, firstly, the independent sound characteristics with the speaker identity information are extracted from the roughly separated speech in the first stage, then the potential relevant characteristics and complementary characteristics between the visual mode and the audio mode are mined, secondly, the two modes of the visual characteristic and the speech time-frequency domain characteristic are fused and then separated, and finally the separated target speech is obtained after the weights of the two stages are dynamically adjusted.

2. The two-stage voice separation method based on visual guidance as claimed in claim 1, wherein the first stage comprises the following specific processes:

coding the obtained mixed voice by using a coder, and extracting the characteristics of the mixed voice;

and separating the mixed voice characteristics by using a first separation network to obtain a mask of the target voice, multiplying the mask and the mixed voice characteristics, and then decoding to obtain a time domain signal of the target voice.

3. The method as claimed in claim 2, wherein the step of separating the mixed speech features in the first stage comprises processing the mixed speech features by using a first separation network, wherein the first separation network is a time convolution network structure and comprises a normalization layer and a plurality of identical stack modules, each of which comprises a full convolution layer, a dilation convolution layer and a residual error module, and the output of the last stack module passes through the convolution layer and the PReLU activation layer to obtain a separated target mask;

or, the target voice feature is obtained by multiplying the mixed voice feature and the mask of the target voice, and the target voice feature is decoded to obtain the time domain signal of the target voice.

4. The two-stage voice separation method based on visual guidance as claimed in claim 1, wherein the second stage comprises the following specific processes:

transforming the mixed voice to obtain a complex spectrum image of the mixed voice, and acquiring a complex spectrum mask of the real pure voice according to the complex spectrum image;

converting the time domain signal of the target voice acquired in the first stage to obtain a separated complex spectrogram of each speaker, and extracting the independent voice feature of each speaker;

and acquiring visual information of the speaker in time synchronization with the mixed voice, preprocessing the visual information, and respectively extracting static visual features and dynamic visual features from the preprocessed visual image.

Extracting mixed voice features from the time-frequency domain information of the mixed voice complex spectrogram, performing multi-modal feature fusion, separating the multi-modal features to obtain a mask of the separated target voice, multiplying the mask and the complex spectrogram of the mixed voice, and performing inverse transformation to obtain a pure voice signal of the target speaker.

5. The method as claimed in claim 4, wherein the step of converting the time domain signal of the target speech obtained in the first stage comprises: firstly, roughly separating voice, and carrying out time-frequency domain conversion to obtain a complex spectrogram; then, using an independent voice feature extraction network ResNet-18 to extract independent voice features for the complex spectrogram of each speaker; the independent speech features are then time dimension transformed to achieve dimensional consistency of the audio and video modality features.

6. The two-stage voice separation method based on visual guidance as claimed in claim 4, wherein the specific process of obtaining and pre-processing the visual information of the speaker in time synchronization with the mixed voice comprises reading a video file, intercepting a video with a set length to obtain a multi-frame image sequence, and randomly selecting a frame of facial image as static visual information; and then cutting each frame of image sequence, selecting a lip region with a set size, and generating a file of the lip sequence as dynamic visual information.

7. The two-stage voice separation method based on visual guidance as claimed in claim 6, wherein the specific process of extracting visual features includes normalizing lip images and data filling, the preprocessed lip data sequentially passes through a 3D convolutional layer and a ShuffleNet v2 network, and then passes through a time convolutional network structure to extract time series features, and finally dynamic visual features are obtained, wherein the dynamic visual features include content information of voice;

standardizing and sizing the face image, and extracting static visual features through a residual error network ResNet-18, wherein the static visual features comprise identity information of speakers with distinctiveness; the static visual features are transformed to have the same time dimension as the dynamic visual sequence features.

8. The two-stage voice separation method based on visual guidance as claimed in claim 4, wherein the specific process of performing multi-modal feature fusion includes firstly obtaining mixed voice features by a mixed voice complex spectrogram through a U-Net down-sampling network layer, then performing cascade splicing on visual features, independent voice features and mixed voice features of a speaker, and finally obtaining fused multi-modal features;

or separating the multi-modal features by utilizing a second separation network, wherein the second separation network is an up-sampling network layer of the U-Net.

9. The method as claimed in claim 1, wherein the weight of the loss function for the two-stage speech separation is dynamically adjusted to maximally utilize the independent speech features of the first stage to assist the separation of the second stage.

10. A two-stage speech separation system based on visual guidance, comprising:

the first separation module is configured to separate the acquired mixed voice in a time domain to obtain independent speaker voice in a first stage;

the second separation module is configured to extract independent voice features with speaker information by means of the pure audio frequency separation result of the first stage at the second stage, simultaneously excavate potential relevant features and complementary features between the visual modality and the audio modality, perform fusion of the visual features and the voice time-frequency domain features, and then separate the two modalities to finally obtain separated target voice;

and the dynamic weight adjusting module dynamically adjusts the weight of the two stages according to the performance of the separation model of the two stages so as to utilize the independent voice characteristics extracted in the first stage to the maximum extent to assist the second stage and realize the voice separation of the pure target speaker.

Images