JP4565162B2 - Speech event separation method, speech event separation system, and speech event separation program - Google Patents

Description

  The present invention relates to an utterance event separation method, an utterance event separation system, and an utterance event separation program for separating utterance events when utterances such as simultaneous utterances and conflicts overlap during a conference.

  Conventionally, in meetings with customers in companies and meetings such as committees in public institutions, a method of manually creating conference minutes has been generally used to record the contents of meetings. Small meetings are often not worth the cost.

  Therefore, in recent years, methods for automatically creating conference minutes by speech recognition have been researched and developed, and commercialization has been made, but the recognition rate for natural utterances is only about 60%. It was necessary to manually correct the obtained results.

  In addition, video and recorders are widely used to record conference audio and images, but in order to understand what discussions took place during the conference, it is necessary to play back all recorded and recorded content. There is poor efficiency.

Therefore, research is being conducted to efficiently access the desired recorded / recorded content by analyzing the recorded / recorded content and adding information such as who made what remarks. (See Non-Patent Document 1.)
Jitendra Ajmera, et al. “Clustering and Segmenting speakers and their locations in meetning,” Proc. ICASSP 2004, Vol.I, pp.605-608, 2004.

  FIG. 5 is a flowchart showing a general procedure for creating conference minutes contents in such a research. The contents recorded and recorded in the conference are analyzed, and the process of extracting information on when and who has made a statement is performed. Made. Here, such processing is called “structuring”.

  Speech is structured and segmented by speaker (segmentation) .The speech content is analyzed by speech recognition, and advanced information is added by keyword extraction, topic classification, semantic summarization, etc. Minutes content is produced.

  Here, in order to perform speech recognition with high accuracy, the utterance content must be clear, but in a meeting place where participants can speak freely, while a speaker (target speaker) is speaking If there are other speakers' conflicts, interrupted speech, coughing, etc., the speech of other speakers (competitive speakers) is superimposed on the content of the target speaker's speech. There was a problem that the accuracy was significantly lowered.

In the conventional research for separating a plurality of sound sources, the problem of separating a speaker's speech in the presence of noise (for example, sound of a television or radio) was dealt with for the purpose of application to a voice interface or the like.
In this case, as shown in FIG. 6, if the overlap period between the noise source and the target sound source (utterance) is sufficiently long, the noise can be detected by a conventionally known method such as blind sound source separation (BSS) using independent component analysis (ICA). And the speaker's speech can be separated. (For example, refer nonpatent literature 2.)
Te-Won Lee, “Independent Component Analysis,” Kluwer Academic Publishers, 1998

  However, as shown in FIG. 7, there are many cases where the voice fogging in the conference has a very short overlapping section as shown in FIG. 7, such noise and utterances are separated by the BSS method described above. This conventional method cannot be applied.

On the other hand, a method of sound source separation using an adaptive beamformer has also been proposed, but in order to use this method, the “speaker position” that contains information on the arrival time from the speaker position to multiple microphones used for recording is proposed. You must give a "vector". (For example, refer nonpatent literature 3.)
Don Johnson and Dan Dudgeon, “Array signal processing,” Prentice hall, 1993

  In the method using the adaptive beamformer, generally, a virtual sound source is arranged at an expected speaker position (for example, the center of the microphone every 5 degrees of the central angle), and a position vector at this position is measured. In the actual conference voice, a method using the closest one of the position vector candidates obtained in advance is used.

  This work is called calibration, and it is necessary to perform measurement work for each set of microphones (hereinafter referred to as microphone array) used for recording, which is very disadvantageous for mass production of input devices. .

  In addition, it is necessary to provide a noise spatial correlation matrix that includes spatial noise information, and in the past, this correlation matrix was estimated from the pause interval of the speaker's speech, assuming that the surrounding noise is stationary. It was. However, in conferences, competing speakers as noise sources are not constant and change constantly, so such a method could not be used.

  Therefore, the present invention solves the problems of the conventional adaptive beamformer as described above, removes the fog of the utterance during the conference, and separates the utterance contents with high accuracy for each speaker. An utterance event separation method, an utterance event separation system, and an utterance event separation program are provided.

  In order to achieve the above object, the utterance event separation method in the conference record of the present invention performs sound source localization from multi-channel audio data recorded at the conference site in consecutive sections of the conference, and the spatial spectrum A first step of detecting a peak value and estimating a sound source direction for each time in the section; and a second step of clustering the peak values over the entire section to estimate a range where a speaker serving as a sound source exists. The third step of identifying which speaker is speaking at each time from the sound source direction at each time and the range where the speaker exists, and the purpose from the data obtained in the third step In the fourth step and the third step, a block where a speaker who speaks alone finds a block speaking in the section and estimates the target speaker's position vector from the block is obtained. A fifth step of finding a block in which the other speaker is speaking alone in the section from the obtained data and calculating a noise spatial correlation matrix of the other speaker with respect to the target speaker from the block; A sixth step of generating a filter from the position vector of the speaker estimated in the fourth step and the noise spatial correlation matrix calculated in the fifth step; and the target speaker and the other speaker The seventh step is to apply the filter to the block in which the utterances of と are superimposed and perform filtering to separate and output the utterances of only the target speaker.

  Further, the speech event separation system of the present invention performs sound source localization from multi-channel audio data recorded at the conference site in consecutive sections during the conference, detects the peak value of the spatial spectrum, and A sound source direction estimating unit that estimates a sound source direction at each time in the middle, and a speaker that estimates a range where a speaker serving as a sound source exists by clustering the peak values detected by the sound source direction estimating unit over the entire section Speaker identification for identifying which speaker is speaking at each time from the range estimation means, the sound source direction at each time obtained by the sound source direction estimation means and the speaker range estimation means and the range where the speaker exists And the data obtained by the speaker identification means find a block where the target speaker is speaking alone within the section, and the target speaker is extracted from the block. Speaker position vector estimation means for estimating a position vector, and from the data obtained by the speaker identification means, a block in which another speaker is speaking alone within the section is searched, and the purpose and A noise spatial correlation matrix calculating means for calculating a noise spatial correlation matrix of the other speaker with respect to the speaker who performs, a position vector estimated by the speaker position vector estimating means, and a noise spatial correlation matrix calculating means Filter generating means for generating a filter from the noise-space correlation matrix, and applying the filter to a block in which the speech of the target speaker and another speaker is superimposed, Filtering means for separating and outputting the utterances of only the speaking speaker.

  In the speech event separation system according to the present invention, it is desirable to record multi-channel audio data using a microphone array configured by arranging a plurality of microphones radially.

  Further, the speech event separation program of the present invention is such that multi-channel audio data recorded at the conference is input to a computer and stored in the memory in consecutive sections of the conference, and the computer stores the audio A first step of performing sound source localization from data, detecting a peak value of the spatial spectrum and estimating a sound source direction at each time in the section, and clustering the peak values over the entire section to become a sound source A second step of estimating a range in which a speaker is present, and a third step of identifying which speaker is speaking at each time from the sound source direction estimated at each time and the range in which the speaker is present Then, from the data obtained in the third step, the block where the target speaker is speaking alone within the section is searched, and the target is determined from the block. From the data obtained in the fourth step of estimating the position vector of the speaker and the third step, a block where another speaker is speaking alone in the section is searched, and the object A fifth step of calculating a noise spatial correlation matrix of the other speaker with respect to the speaker who performs the step, a speaker position vector estimated in the fourth step, and a noise spatial correlation matrix calculated in the fifth step And a sixth step of generating filter data from the block, and a block on which the speech of the target speaker and another speaker is superimposed is filtered based on the filter data, and the target story And a seventh step of separating and outputting the utterances only by the user.

  According to the first aspect of the present invention, even when the speech of the target speaker is superimposed on the speech of the target speaker during the conference, the content of the speech of the target speaker is separated with high accuracy. A speech event separation method that can be extracted can be provided.

  In addition, by using the method of the present invention, it is possible to improve the recognition rate of voice recognition in the case of automatically creating a meeting record by voice recognition from the voice recorded during the meeting, so it was automatically created. It is possible to reduce the labor and time spent for correcting the minutes.

  According to the invention described in claim 2, it is possible to provide an utterance event separation system for realizing the utterance event separation method described in claim 1.

  According to the third aspect of the present invention, since one microphone array configured by arranging a plurality of microphones radially can be placed in the center of the conference table to record the voice, Compared to recording audio by attaching a tie-pin microphone to the chest of all participants, the wiring can be simplified, and there is no risk that the recording will be incomplete due to forgetting to attach the microphone etc. .

  According to the invention described in claim 4, it is possible to provide a computer program that can realize an utterance event separation system easily and at low cost using a computer such as a notebook computer.

  Hereinafter, embodiments in the case of carrying out the present invention will be described with reference to the drawings. FIG. 1 is a system configuration diagram showing an embodiment of an utterance event separation system for implementing an utterance event separation method in a conference record according to the present invention.

  As shown in the figure, the speech event separation system 1 is used by being connected to a microphone array 2 composed of a plurality of microphones 2A, and the sound captured by each microphone 2A is used as an individual multi-channel analog signal. The signal is input from the microphone array 2 to the analog / digital signal conversion means 4 via the cable 3, converted into a digital signal, and input to the speech event separation system 1.

  In the present embodiment, the microphone array 2 is configured by arranging a plurality of microphones 2A radially on the peripheral surface of a cylindrical case, and installing this one in the center of the conference table to record audio. I have to.

  When recording audio in a conventional conference, etc., in order to improve the signal-to-noise ratio of the microphone as much as possible, all the participants in the conference had to attach a tie-pin microphone to the chest. .

  However, when all the participants have microphones attached, the microphone cable wiring becomes complicated, and the participants forget to attach the microphones, and the recorded content is often incomplete.

  On the other hand, in the present embodiment, since the microphone array 2 composed of a plurality of microphones 2A directed radially in different directions is used as a device for recording audio, each participant in the conference is a tiepin type microphone. There is an advantage that the sound from all directions can be recorded.

  Note that the microphone array is not limited to that of the present embodiment, and may be configured by arranging the number of microphones in a linear or arc shape as appropriate according to the shape of the conference table and the arrangement of the seats of the conference participants. .

  On the other hand, the utterance event separation system 1 incorporates a storage means 5 for recording digital voice data input from the analog / digital signal conversion means 4 over a necessary section in the time during the conference. Yes.

  A hard disk or the like can be used as the storage means 5. In this embodiment, the storage means 5 is incorporated in the apparatus constituting the utterance event separation system 1, but as an independent unit like the analog / digital signal conversion means 4. It may be provided outside. Further, together with the analog / digital signal conversion means 4, it may be incorporated in a device constituting the utterance event separation system 1.

  The utterance event separation system 1 of the present invention includes a sound source direction estimation means 6, a speaker range estimation means 7, a speaker identification means 8 for performing “structuring” processing in the previous stage, and “speech separation in the subsequent stage. ”, A noise correlation matrix calculation unit 10, a filter generation unit 11, and a filtering unit 12 are included.

  FIG. 2 is a flowchart showing an outline of processing by the utterance event separation system 1. In the present invention, the information obtained by the previous “structured” processing shown in steps S1 to S3 in FIG. It is characterized in that the subsequent “utterance separation” processing shown in steps S4 to S7 is performed.

  In the “structured” processing stage in the previous stage, the multi-channel audio input inputted to the microphone array 2 is analyzed to estimate who and when during the conference. In particular, according to the present invention, since the purpose is to remove the fog of the utterance and to isolate only the utterance of the target speaker, it is detailed even for a small utterance event such as a conflict that the competing speaker enters while the target speaker is speaking. Need to be analyzed.

  Therefore, in the “structuring” processing stage, first, the voice data recorded in the storage means 5 is read, the sound source direction estimation means 6 analyzes the multi-channel voice input recorded by the microphone array 2 by sound source localization, and the time The direction of arrival of the sound is estimated every time. (Step S1 in FIG. 2)

  For sound source localization, a conventionally well-known general sound source localization method (see, for example, Don Johnson and Dan Dudgeon, “Array signal processing,” Prentice hall, 1993) can be used. However, in this embodiment, in order to further improve the performance, the MUSIC method (RO Schmidt, “Multiple emitter location and signal parameter estimation,” IEEE Transactions on Antennas and Propagation, vol. AP-24, No. 3, pp. 276 -280, 1986) (F. Asano et al. “Fusion of audio and video information for detecting speech events,” Proc. Fusion 2003, pp. 386-393, 2003).

  Next, the sound peak value data for each time detected by the sound source direction estimating means 6 is input to the speaker range estimating means 7. The speaker range estimation means 7 generates the histogram data shown in FIG. 3 by accumulating the peaks of the spatial spectrum of the sound for the entire conference or for a necessary section during the conference. Clustering is performed by means method (see, for example, R. Duda, E. Hart and D. Sort, “Pattern Classification,” Wiley-Interscience publication 2001).

  Here, the number of clusters required for clustering is the number of conference participants. The cluster center ± R ° obtained from the clustering result is estimated as the speaker's existence range. (Step S2) Here, R is an arbitrary angle (for example, 20 °), and is set for each conference according to the number of participants.

  Next, in the speaker identification means 8, which speaker speaks at each time from the result of the sound source localization obtained by the sound source direction estimation means 6 and the speaker range estimated by the speaker range estimation means 7. To estimate. For example, if a spatial spectrum peak falls within a certain speaker range at a certain time, the speaker is identified as speaking. (Step S3)

  In the subsequent “speech separation” process following the “structured” process as described above, utterances are superimposed (covered) based on the information obtained in the “structured” process stage. Perform speaker separation for parts. For the separation of utterance events, a maximum likelihood estimation method (see, for example, Don Johnson and Dan Dudgeon, “Array signal processing,” Prentice hall, 1993) is used.

  The maximum likelihood estimation method requires two pieces of information: a position vector for the target speaker and a correlation matrix (noise spatial correlation matrix) for noise (competitive speaker). The speaker position vector estimation means 9 estimates the position vector for the target speaker based on the information obtained in the “structuring” processing step described above.

  FIG. 4 is a diagram for explaining the estimation procedure of the speaker position vector and the noise spatial correlation matrix. The information obtained in the “structuring” process stage is used to utter the target speaker alone during the conference. Find the block you are doing. Here, a short time unit, usually about 0.5 seconds, is called a “block”, and the process is executed in block units.

  The speaker position vector estimation means 9 calculates a correlation matrix from a single utterance block, and performs eigenvalue decomposition (see G.Strang, “Linear Algebra and its applications,” Harcourt Brace Jovanovich College Publishers, 1988). To calculate eigenvalues and eigenvectors.

  When the sound source is single, use the property that the eigenvector for the maximum eigenvalue becomes the position vector for that sound source (see Don Johnson and Dan Dudgeon, “Array signal processing,” Prentice hall, 1993), and the eigenvector corresponding to the maximum eigenvalue. Is taken as a candidate for the speaker position vector of this block.

  As shown in FIG. 4, since there are a plurality of blocks in which the target speaker is speaking alone during the conference, the speaker position vector estimating means 9 determines the optimum position from the speaker position vector candidates calculated for these blocks. Is estimated as the speaker position vector.

As a standard, first, select the one with the smallest difference between the direction of the target speaker of the block covered by the utterance and the direction of the speaker of the block that is the candidate of the speaker position vector, and still select multiple candidates. If there is, the one with the largest index of single utterance is selected. (Step S4)
Note that, as an index for the single utterance, an average of the ratios of the maximum eigenvalue and the second largest eigenvalue on the frequency axis among the already calculated eigenvalues is used.

  On the other hand, the noise spatial correlation matrix calculation means 10 searches for a block where the competing speaker is speaking alone, as in the speaker position vector estimation procedure in the speaker position vector estimation means 9. Then, a noise space correlation matrix is calculated in the corresponding block. In addition, the noise space correlation matrix is calculated in the same manner for the other blocks that the competing speaker is speaking alone. (Step S5)

  Next, the filter generation means 11 calculates the target speaker's position vector estimated by the speaker position vector estimation means 9 and the noise spatial correlation matrix calculation means 10 for each block where the competing speaker is speaking alone. A maximum likelihood estimation filter that performs utterance separation for each of the generated noise spatial correlation matrices, and applies it to the block covered by the utterance to be separated. The one that minimizes is generated as the final filter. (Step S6)

  The filtering means 12 applies the filter constructed by the filter generation means 11 to the digital audio signal sent from the analog / digital signal conversion means 4 and separates and outputs only the speech information of the target speaker as the filter output. To do. (Step S7)

  The utterance event separation system 1 described above can be constructed by dedicated hardware, but can also be realized by executing the utterance event separation program on a general-purpose computer such as a notebook computer.

The utterance event separation program includes a computer, a sound source direction estimation unit 6, a speaker range estimation unit 7, a speaker identification unit 8, a speaker position vector estimation unit 9, a noise spatial correlation matrix calculation unit 10, a filter generation unit in FIG. The output signal from the analog / digital signal conversion means 4 is input from USB (Universal Serial Bus Specification Rev. 2.0) or PCI (Peripheral Component Interconnect) or the like. It can be input to the computer via the output interface.
Further, the analog / digital signal conversion means 4 may be incorporated in a computer as a signal conversion module.

  The utterance event separation program is preinstalled in a computer storage device (for example, a hard disk or an optical disk) together with an operating system, and when the program is started, the CPU reads it into a computer RAM (Random Access Memory). Then, the process of each step shown in FIG. 2 described above is executed.

  Here, in the case of using a computer, digital audio data input to the computer via the analog / digital signal conversion means 4 is a hard disk or the like built in or externally attached to the computer over a section necessary for recording during the conference. Accumulated in storage means.

  The utterance event separation program executes each step described above using the accumulated data, finally separates only the utterance of the target speaker, and outputs the voice data to the outside through the output interface. Note that the speech data separated by speech may be output and stored on a hard disk or the like.

  The utterance event separation method, the utterance event separation system, and the utterance event separation program according to the present invention can be used when, for example, a conference record is automatically created by voice recognition from voice data recorded in a small meeting. is there.

It is a system configuration figure showing one embodiment of an utterance event separation system for carrying out an utterance event separation method of the present invention. It is a flowchart which shows the outline | summary of the process by the speech event separation system 1 of this invention. It is a figure showing the speaker range estimated from the histogram used for speaker range estimation, and a clustering result. It is a figure explaining the estimation procedure of a speaker position vector and a noise correlation matrix. It is a flowchart which shows the preparation procedure of the conventional general meeting minutes content. It is a figure which shows typically the state where the superimposition area of a target sound source and a noise source is long. It is a figure which shows typically the state in which the speech of a target speaker and a competition speaker has overlapped the short area.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Speech event separation system 2 Microphone array 2A Microphone 3 Cable 4 Analog / digital signal conversion means 5 Storage means 6 Sound source direction estimation means 7 Speaker range estimation means 8 Speaker identification means 9 Speaker position vector estimation means 10 Noise correlation matrix calculation Means 11 Filter generation means 12 Filtering means

Claims (4)

First, sound source localization is performed from multi-channel audio data recorded at a meeting section in a continuous section of the conference, and a peak value of the spatial spectrum is detected to estimate a sound source direction at each time in the section. And the steps
A second step of clustering the peak values over the entire section to estimate a range in which a speaker serving as a sound source exists;
A third step of identifying which speaker is speaking at each time from the sound source direction at each time and the range where the speaker exists;
A fourth step of searching for a block in which the target speaker is speaking alone within the section from the data obtained in the third step, and estimating a position vector of the target speaker from the block; ,
From the data obtained in the third step, a block where another speaker is speaking alone within the section is searched, and the noise space correlation matrix of the other speaker with respect to the target speaker is determined from the block. A fifth step of calculating;
A sixth step of generating a filter from the speaker position vector estimated in the fourth step and the noise spatial correlation matrix calculated in the fifth step;
Applying the filter to a block in which the speech of the target speaker and another speaker is superimposed, filtering, and outputting the speech of only the target speaker separately An utterance event separation method comprising steps. A sound source that performs sound source localization from multi-channel audio data recorded at the conference site in successive sections of the conference, detects the peak value of the spatial spectrum, and estimates the sound source direction at each time in the section Direction estimation means;
Clustering the peak values detected by the sound source direction estimating means over the entire section to estimate a range in which a speaker serving as a sound source exists;
Speaker identification means for identifying which speaker is speaking at each time from the sound source direction for each time obtained by the sound source direction estimating means and the speaker range estimating means and the range where the speaker exists;
Speaker position from which the target speaker finds a block that speaks alone in the section from the data obtained by the speaker identification means and estimates the position vector of the target speaker from the block Vector estimation means;
From the data obtained by the speaker identification means, a block in which another speaker speaks alone in the section is searched, and the noise space correlation of the other speaker with respect to the target speaker from the block A noise space correlation matrix calculating means for calculating a matrix;
Filter generating means for generating a filter from the position vector estimated by the speaker position vector estimating means and the noise spatial correlation matrix calculated by the noise spatial correlation matrix calculating means;
Filtering means for performing filtering by applying the filter to a block in which utterances of the target speaker and other speakers are superimposed, and separating and outputting utterances of only the target speaker; An utterance event separation system characterized by comprising:   3. The speech event separation system according to claim 2, wherein multi-channel audio data is recorded using a microphone array configured by arranging a plurality of microphones radially. In continuous sections of the conference, multi-channel audio data recorded at the conference is input to the computer and stored in its memory.
In the computer,
A first step of performing sound source localization from the audio data, detecting a peak value of the spatial spectrum and estimating a sound source direction at each time in the section;
A second step of clustering the peak values over the entire section to estimate a range in which a speaker serving as a sound source exists;
A third step of identifying which speaker is speaking at each time from a sound source direction estimated at each time and a range where the speaker exists;
A fourth step of searching for a block in which the target speaker is speaking alone within the section from the data obtained in the third step and estimating a position vector of the target speaker from the block When,
From the data obtained in the third step, a block where another speaker speaks alone within the section is searched, and the noise space correlation matrix of the other speaker with respect to the target speaker from the block A fifth step of calculating
A sixth step of generating filter data from the speaker position vector estimated in the fourth step and the noise spatial correlation matrix calculated in the fifth step;
Filtering based on the filter data on a block in which the speech of the target speaker and another speaker is superimposed, and outputting the speech of only the target speaker separately An utterance event separation program characterized in that the steps are executed.

Images