JP4716962B2 - CONFERENCE SYSTEM, CONFERENCE SERVER, AND CONFERENCE SYSTEM DISTRIBUTION VOICE CONTROL METHOD - Google Patents

Description

  The present invention relates to a conference system, a conference server, and a distributed audio control method for the conference system, and can be applied to, for example, an audio signal processing system in an audio conference system or a video conference system.

In general, in an audio conference system, when the voices of all the members other than the person are mixed and distributed to the participants, it becomes difficult to hear each member's voice. For this reason, even if a conference is held with a large number of participants, it is also conceivable that only a few voices are preferentially mixed and distributed in the conference server. For example, several voices to be used for mixing are determined as follows. The voices of the top few people with high voice levels captured by the voice samples for a certain period of time (the amount inserted in the packet) are extracted. Notification is made (see Patent Document 1).
JP 2005-504450 gazette

  However, in the method of determining the audio signal to be distributed by mixing in the next cycle from the sample in a certain cycle, for example, even if a participant insists loudly, if the voice is low in the previous cycle, the target for mixing in the current cycle Therefore, the beginning of the assertion is not mixed.

  In order to prevent such a situation that the sound immediately after the change is not mixed at the time of such a sudden change in the sound level, it is conceivable to shorten the period of a certain period and shorten the period of not being mixed. However, if this is done, there will be a problem of increasing the processing load on the conference server and client, and the audio used for mixing will be reviewed in a short cycle, so the combination of multiple audio used for mixing is frequently switched. It is quite possible that it will be done, making the conference participants feel uncomfortable, or being regarded as sound quality degradation.

  Further, in the determination of the mixing target based on the audio level, the audio signal related to the conference progress by the chairperson or the like is also excluded from the mixing target if the audio level is low, and there is a possibility that the notification of the conference progress is insufficient.

  Therefore, when controlling the audio signal to be mixed to a part of the audio signal, it is possible to accurately include the audio signal having a high priority in the object to be mixed, and to ensure sufficient quality for the object to be mixed. A conference system, a conference server, and a distributed audio control method for the conference system are desired.

  The first aspect of the present invention synthesizes only a predetermined number of audio signals from a plurality of conference terminals, transmits the synthesized audio signals to all the conference terminals, and sets the level of each audio signal in the current cycle. Based on this, the conference server determines a predetermined number of audio signals to be synthesized in the next cycle, and (1) synthesis that determines the necessity of changing to a synthesis target for each audio signal that is not to be synthesized in the current cycle. The target review means, and (2) synthesis that adds the discriminated audio signal to the compositing target in the current cycle after the discriminating when there is an audio signal that has been discriminated to be changed to the synthesis target And an object adding means.

  The second aspect of the present invention synthesizes only a predetermined number of audio signals out of audio signals from a plurality of conference terminals and a plurality of conference terminals that transmit audio signals and receive synthesized audio signals. In a conference system including a conference server that transmits to all conference terminals, the conference server of the first aspect of the present invention is applied as the conference server.

  In the distributed audio control method for a conference system according to the third aspect of the present invention, the conference server synthesizes only a predetermined number of audio signals among audio signals from a plurality of conference terminals, and transmits the synthesized audio signals to all the conference terminals. In the conference system that determines a predetermined number of audio signals to be synthesized in the next cycle based on the level of each audio signal in the current cycle, the conference server The necessity of change to the synthesis target is determined, and if there is an audio signal that is determined to be required to be changed to the synthesis target, the determined audio signal is synthesized in the current cycle after the determination. It is characterized by adding to.

  According to the present invention, when the audio signal to be mixed is controlled to be a part of the audio signal, it is possible to accurately include the audio signal having a high priority in the mixing target, and the mixing target is sufficient. Quality can be ensured.

(A) First Embodiment Hereinafter, a first embodiment in which a conference system, a conference server, and a distributed voice control method for a conference system according to the present invention are applied to a voice conference system will be described with reference to the drawings.

(A-1) Configuration of the First Embodiment FIG. 2 is a block diagram showing the overall configuration of the audio conference system of the first embodiment.

  In FIG. 2, the audio conference system 1 according to the first embodiment includes a plurality of conference clients 10-1 to 10-5 (FIG. 2 shows a case of 5) and a conference server 20. The conference server 20 receives voice signals from all the conference clients 10-1 to 10-5 (for example, inserted into an RTP packet and transmitted) via the network 2, and the conference server 20 receives the total number of participants. A smaller predetermined number of audio signals are selected and mixed, and the mixed audio signals (for example, inserted into an RTP packet and transmitted) are distributed to the conference clients 10-1 to 10-5 via the network 2. .

  FIG. 2 shows an example in which the audio signals B, E, and F from the conference clients 10-2, 10-4, and 10-5 are selected to be used for mixing.

  The conference client 10 (10-1 to 10-5) may be, for example, a conference dedicated terminal, and an information processing device (for example, a personal computer) that can be attached / detached with headphones, or an information processing device equipped with a microphone and a speaker. Voice conferencing software may be installed on the top, and the configuration may be the same as that of an existing conference client. The conference client 10 functionally has a configuration as illustrated in FIG.

  In FIG. 3, a microphone 11 captures the speech of a conference participant and outputs a speech signal, and an analog / digital converter (A / D) 12 converts the captured speech signal (analog signal) into a digital signal. Is.

  The communication control unit 13 cuts the audio signal (audio stream) from the analog / digital conversion unit 12 into a predetermined number of samples, inserts the audio signal into the payload of each RTP packet, and transmits the RTP packets to the conference server 20 one after another. Is. Further, the communication control unit 13 extracts the audio signal inserted in the RTP packet coming from the conference server 20, returns it to the audio stream, and gives it to the digital / analog conversion unit (D / A) 14.

  The digital / analog conversion unit 14 converts a given audio signal (digital signal) into an analog signal, and the speaker 15 outputs a sound signal output from the digital / analog conversion unit 14.

  The conference control unit 16 performs control at the time of joining or leaving the conference in accordance with a command from a key input unit (not shown). For example, the communication control unit 13 can control the conference server by this control. The voice signal (RTP packet) is exchanged with the 20 side, or the voice signal exchange state is terminated.

  In the case of the first embodiment, the communication control unit 13 of the conference clients 10-1 to 10-5 transmits an RTP packet including an audio signal in the same time zone by negotiation with the conference server 20 at the time of conference participation. It is made to do. In other words, all the conference clients 10-1 to 10-5 are configured to transmit RTP packets in synchronization.

  The conference server 20 may be realized by any server on the network. For example, if the ISP provides a voice conference service, the provider server may function as a conference server. The conference server 20 is functionally configured as shown in FIG.

  In FIG. 1, voice receiving units 21-1 to 21-5 receive RTP packets from corresponding conference clients 10-1 to 10-5, respectively, and extract inserted voice signals. Each of the sound receiving units 21-1 to 21-5 has a built-in sound buffer, temporarily stores the extracted sound signal in the sound buffer, and stores it in the corresponding sound level measuring units 22-1 to 22-5 and the mixing unit 24. It is given at a predetermined timing.

  The sound level measuring units 22-1 to 22-5 measure the sound levels (sound power) of long and short periods of sound signals obtained by the corresponding sound receiving units 21-1 to 21-5, respectively. Is. For example, as the long-term voice level, the sum of squares of all voice samples related to one RTP packet can be applied. The short-term audio level is the current audio level given to the mixing unit 24. For example, the sum of squares of the audio samples including the current audio sample to be given to the mixing unit 24 and the audio sample having a very small number of samples (for example, about 10 samples) immediately before that can be applied.

  Based on the long-term audio levels from the audio level measuring units 22-1 to 22-5, the mixing target determining unit 23 outputs a predetermined number of audio signals related to the next RTP packet period to be used for mixing in the mixing unit 24. At the same time, based on the voice level for a short period of time, whether or not there is a voice signal related to the current RTP packet period to be mixed is reviewed. In the following description, the predetermined number is assumed to be 3. However, the predetermined number is not limited to this, and the predetermined number itself may change according to a long-term audio level (for example, All audio signals whose long-term audio levels exceed the threshold may be set as mixing targets).

  The mixing target determining unit 23 determines that three types of audio signals having higher levels among the five types of long-term audio levels from the audio level measuring units 22-1 to 22-5 are to be used for mixing. The mixing target determining unit 23 monitors the short-term sound level of the remaining sound signals that are not used for mixing, and when it is determined that the short-term sound level exceeds the threshold, from the determination time point, The audio signal is switched to an audio signal used for mixing. If the condition is satisfied, there may be a plurality of audio signals to be switched to the mixing target.

  Here, the audio signal switched to be mixed may be added to the three types of audio signals that have been mixed so far, and may be mixed. Among the three types of audio signals that have been mixed so far, Then, it may be replaced with the one with the lowest long-term audio level (mixing order is the lowest) and the mixing target (in this case, the mixing order is the highest). Further, the threshold value to be compared with the short-term audio level may be a fixed value, or may be adaptively changed according to the long-term audio levels related to the three types of audio signals to be mixed. For example, α may be a threshold value that is α (α is constant; for example, α is greater than 1) of three types of long-term audio levels.

  The mixing unit 24 mixes a plurality of types of audio signals determined to be mixed by the mixing target determining unit 23.

  The mixed voice transmission unit 25 assembles an RTP packet including the mixed voice signal output from the mixing unit 24 in a payload, and broadcasts the RTP packet to all the conference clients 10-1 to 10-5.

  Although omitted in FIG. 1, the conference server 20 also includes a portion that performs control associated with participation or withdrawal of the conference clients 10-1 to 10-5.

(A-2) Operation of the First Embodiment Next, the operation of the audio conference system 1 according to the first embodiment will be described.

  Each of the conference clients 10-1 to 10-5 transmits an RTP packet including the audio signal of the conference participant related to the conference client 10-1 to the conference server 20 in synchronization.

  In the conference server 20, RTP packets from the conference clients 10-1 to 10-5 are received by the corresponding audio receivers 21-1 to 21-5, and the audio signals inserted in the RTP packets are extracted. .

  The audio signals extracted by the audio receiving units 21-1 to 21-5 are respectively supplied to the corresponding audio level measuring units 22-1 to 22-5, and the audio levels (audio power) for a long period and a short period are measured. Then, it is given to the mixing target determining unit 23. The mixing target determining unit 23 determines that three types of audio signals having higher levels among all types of long-term audio levels are to be mixed when the next RTP packet is received. The mixing target determining unit 23 monitors the short-term sound level of the sound signal that is not used for mixing, and switches the sound signal to the one that is used for mixing from the point in time when the short-time sound level exceeds the threshold.

  That is, in a situation where the short-term audio level does not exceed the threshold, a predetermined type of audio signal determined based on the long-term audio level at the time of reception of the previous RTP packet is mixed in the mixing unit 24, When the audio level for a short period of the audio signal not subjected to mixing exceeds a threshold value, the audio signal is also mixed in the mixing unit 24.

  The mixed audio signal output from the mixing unit 24 is inserted into the payload of the RTP packet in the mixed audio transmission unit 25, and the RTP packet is broadcast to all the conference clients 10-1 to 10-5.

  Each of the conference clients 10-1 to 10-5 extracts a voice signal (mixed voice signal) from the received RTP packet and outputs the voice signal.

(A-3) Effects of the First Embodiment According to the first embodiment, the current audio level is monitored in an audio signal that is not subject to mixing in the determination based on a past long-term audio level. When the audio level suddenly increased, we changed to the mixing target from that point, so the audio signals of the conference participants who suddenly started to make loud claims almost started to make claims. It can be a target for mixing from that point in time, and there are very few additional configurations. Thereby, the voice of the participant who started the claim can be mixed without leaking without shortening the level measurement cycle.

  Basically, since the mixing target is determined based on the past long-term audio level, except for the above cases, the mixing target can be determined stably, and the mixing target has sufficient quality. Can be secured.

(B) Second Embodiment Next, a second embodiment in which the conference system, the conference server, and the distributed voice control method of the conference system according to the present invention are applied to the voice conference system will be described with reference to the drawings.

  In the audio conference system of the second embodiment, the conference server 20A is changed from that of the first embodiment. FIG. 4 is a block diagram illustrating a functional configuration of the conference server 20A according to the second embodiment.

  In FIG. 4, the conference server 20A of the second embodiment includes specific pattern recognition units 26-1 to 26-5 in addition to the configuration of the first embodiment, and the determination method of the mixing target determination unit 23A is also the first. The first embodiment is slightly changed.

  The specific pattern recognizing units 26-1 to 26-5 each include a specific pattern if the specific voice is included in the current voice signal extracted from the received RTP packet by the corresponding voice receiving units 21-1 to 21-5. The recognition result is given to the mixing target determining unit 23A. Here, the specific pattern is a statement indicating the chairman's remarks such as “conference start”, “conference end”, “break start”, “break end”, and the like.

  For example, the specific pattern recognition units 26-1 to 26-5 store the reference audio signal waveform related to the specific pattern, and the specific pattern recognition units 26-1 to 26-5 are specified by collating the reference audio signal waveform with the audio signal waveform to be processed. Recognize patterns. Further, for example, the specific pattern recognition units 26-1 to 26-5 convert the input speech signal into a text string, perform morphological analysis, and recognize the specific pattern.

  The specific pattern is not limited to the voice signal extracted from one received RTP packet, but may be recognized from a time series of voice signals extracted from a plurality of received RTP packets. You may make it give to the mixing object determination part 23A.

  Similarly to the first embodiment, the mixing target determining unit 23A of the second embodiment determines a mixing target and, based on the recognition results of the specific pattern recognition units 26-1 to 26-5, Review whether there is any audio signal to be mixed in the current RTP packet period. When the specific pattern is recognized for the audio signal related to the current RTP packet period, the mixing target determining unit 23A recognizes the mixing target even if the audio signal is not the target of mixing based on the past long-term audio level determination. Switch to.

  Also according to the second embodiment, the same effects as those of the first embodiment can be obtained, and further, the following effects can be obtained.

  In other words, since a voice signal of a specific pattern of a participant who has a specific role in the conference is recognized and changed to a target for mixing, a natural conference can be realized, and the system load is also affected so much. Don't give.

(C) Other Embodiments The conference server in each of the above embodiments may realize characteristic control by software or hardware. In the case of realization with hardware, if an IC chip is used, the processing load and speed become faster.

  In the second embodiment described above, both the review of the mixing target based on the short-term audio level and the review of the mixing target based on the specific pattern are shown. However, the conference is performed so that only the latter is performed. You may build a server.

  Further, in each of the above-described embodiments, a configuration including a configuration (for example, a short-term audio level measurement configuration or a specific pattern recognition unit) necessary for reviewing the mixing target as many as the upper limit that can participate in the conference is shown. However, such a configuration may be provided by a difference from the upper limit number to the number of mixing targets, and a switch for inputting a voice signal of a participant who is not a mixing target may be provided in these configurations.

It is a block diagram which shows the functional structure of the conference server which concerns on 1st Embodiment. It is a block diagram which shows the whole structure of the audio conference system which concerns on 1st Embodiment. It is a block diagram which shows the functional structure of the conference client which concerns on 1st Embodiment. It is a block diagram which shows the functional structure of the conference server which concerns on 2nd Embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Voice conference system, 10-1 to 10-5 ... Conference client, 20 ... Conference server, 211-1 to 21-5 ... Voice receiving unit, 222-1 to 22-5 ... Voice level measuring unit, 23, 23A ... mixing target determination unit, 24 ... mixing unit, 25 ... mixed voice transmission unit, 26-1 to 26-5 ... specific pattern recognition unit.

Claims (5)

Of the audio signals from a plurality of conference terminals, only a predetermined number of audio signals are synthesized, and the synthesized audio signals are transmitted to all the conference terminals and synthesized in the next period based on the level of each audio signal in the current period. A conference server that defines a predetermined number of target audio signals,
In the current cycle, for each audio signal that is not the synthesis target, a synthesis target review unit that determines the necessity of changing to the synthesis target;
A synthesis target adding means for adding the determined voice signal to the synthesis target in the current cycle after the determination when there is an audio signal determined to be changed to the synthesis target. Feature conference server.   The synthesis target review means confirms whether or not a short-term level sufficiently shorter than the cycle exceeds a threshold value for each voice signal that is not a synthesis target, and it is necessary to change the voice signal exceeding the threshold value to a synthesis target. The conference server according to claim 1, wherein the conference server is determined to have a property.   The synthesis target review unit confirms whether or not the waveform pattern of the audio signal matches a specific pattern stored in advance for each audio signal that is not the synthesis target, and converts the audio signal that matches the specific pattern to the synthesis target. The conference server according to claim 1, wherein it is determined that there is a need for a change. A plurality of conference terminals that transmit audio signals and receive synthesized audio signals, and a predetermined number of audio signals are synthesized among audio signals from the plurality of conference terminals, and the synthesized audio signals are transmitted to all conference terminals. In a conference system including a conference server,
The conference system according to claim 1, wherein the conference server according to claim 1 is applied. The conference server synthesizes only a predetermined number of audio signals from the audio signals from a plurality of conference terminals, transmits the synthesized audio signal to all the conference terminals, and based on the level of each audio signal in the current period, In a conference system that defines a predetermined number of audio signals to be synthesized in the next period,
The conference server
In the current cycle, for each audio signal that is not the synthesis target, determine the necessity to change to the synthesis target,
Distribution of a conference system characterized in that, when there is an audio signal that is determined to need to be changed to the synthesis target, the determined audio signal is added to the synthesis target in the current cycle after the determination. Voice control method.

Images