JP2001145103A - Transmission device and communication system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve reliability of a multimedia communication system. SOLUTION: This transmission device is equipped with a band recognizing means which recognizes the quantity of a band allowed by a network for transmission according to a congestion state, a video encoding means which generates video information by inputting and encoding a video signal, an audio data encoding means which generates audio data information corresponding to audio information by inputting and encoding an audio signal, an audio text encoding means which generates audio text information corresponding to the audio information by converting the inputted audio signal into a text signal through speech recognition and encoding the text information, and a transmission information selecting means which sends the video information and audio data information as transmit information, when the allowed band is wide and the audio text information as transmit information, when it is small.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a communication system for transmitting multimedia data such as video and audio.
This can be applied to a case where communication is performed via a network.

[0002] The present invention also relates to a transmission device as a component of such a communication system.

[0003]

2. Description of the Related Art Japanese Patent Laid-Open No. 10-164 discloses a conventional technique for communicating video information and audio information via a network.
No. 533 (Reference 1) and Japanese Patent Application Laid-Open No. 10-285275.
There is one described in Japanese Patent Application Laid-Open Publication No. H10 (Document 2).

In the image communication method and apparatus of Document 1, when multimedia data is communicated in real time over a communication network such as an IP (Internet Protocol) network for which bandwidth is not guaranteed, the network becomes congested and the available bandwidth is changed. When the number decreases, optimal multimedia data is communicated within the limited allowable band.

[0005] In this communication, an available bandwidth is estimated from the data loss rate of the network and the like, and based on the setting of the priority of each data (whether to transmit with priority to voice) set by the user, the application performs Control of image priority and audio priority is performed.

Therefore, for example, in the case of audio priority, a band as requested is allocated to audio data, and the remaining band is allocated to video data.

[0007] Further, in Reference 1, an image parameter such as a frame rate, an image quality (compression parameter), and an image size (resolution) is adjusted to realize a video quality required by a user within a transmission capacity.

[0008] Next, in the telephone call method of Reference 2, on the transmitting side,
The voice data corresponding to the voice of the speaker is once converted into character data by voice recognition, and the character data is packetized and transmitted to the network. On the receiving side that receives the packet from the network, the character data extracted from the packet is converted into voice data by voice synthesis and output as voice.

[0009] Thus, the document 2 enables easy-to-understand conversations while minimizing voice deterioration, unnatural division of words, and skipping of sound due to communication instability and load fluctuation, and enabling natural conversation. The purpose is to enable conversation.

[0010]

[0005] However, Document 1
The method of optimally setting the spatial resolution, frame rate, and compression parameters of an image does not reflect the importance of the contents of individual frames, but specifies the handling of multiple time-series frames as a whole. It's just that.

[0011] Note that in Document 1, audio-priority or image-priority processing can be performed.
The problem is not whether the processing reflects the importance of each frame but whether the sound is given priority over the image as a whole.

With such a method, it is considered that even if the congestion degree of the network is low, it is difficult to cope with the congestion degree. This is a situation where, for example, even if the video priority is set, it is not possible to allocate the requested bandwidth even to the video.

[0013] The fact that the bandwidth is not allocated as requested means that even for high-priority video information, there is a high possibility that normal transmission cannot be performed due to an increase in the packet loss ratio in the transmission process in the network.

In such a situation, if the packet is transmitted from the transmitting communication device because the allowable band is extremely small, the packet transmission itself becomes the loss ratio of the subsequent packet transmitted from the transmitting communication device. The problem is how to utilize the extremely small allowable band efficiently.

[0015] Next, in Document 2, transmission of audio information is performed by converting audio data into character data by voice recognition, packing the character data into packets, and transmitting the data through a network. The relationship between information transmission is not clear. However, it is merely described that video data such as the face of a speaker may be transferred, and the video data may be transferred to the other party together with the sound.

Therefore, how to relate the transmission of the video information and the transmission of the audio information becomes a problem.

[0017]

According to a first aspect of the present invention, there is provided a transmitting apparatus for transmitting video information and / or audio information as transmission information and transmitting the transmission information to a receiving apparatus via a network. (1) a band recognizing means for recognizing an amount of a permissible band permitted for the transmission by the network according to the congestion state; and (2) the video information by inputting and encoding a video signal. And (3) audio data encoding means for generating audio data information corresponding to the audio information by inputting and encoding an audio signal, and (4) converting the input audio signal Voice text encoding means for converting into a text signal by voice recognition and encoding the text signal to generate voice text information corresponding to the voice information; (5) When the allowable band is large, the video information and the audio data information are transmitted as the transmission information, and when the allowable band is small, the transmission information selecting unit that transmits the audio text information as the transmission information. It is characterized by having.

According to the second and third aspects of the present invention, at least the transmitting device for forming video information as transmission information and transmitting the transmission information to the receiving device via a network has the following configuration requirements. Features.

That is, in the second invention, (1) band recognizing means for recognizing the amount of allowable band allowed for the transmission by the network according to the congestion state, and (2)
When the video information is a moving image, from among a plurality of time-series screens constituting the moving image, it is determined that a screen having a larger amount of change regarding an image in the screen has a higher importance, and the importance is higher. Important screen selecting means for selecting a screen; (3)
If the permissible bandwidth is large, all time-series screens are transmitted as the transmission information, and if the permissible bandwidth is small, the screen is selected by the important screen selecting means from a plurality of time-series screens. Video information selecting means for transmitting only a screen as the transmission information.

Further, in the third invention, (1) a band recognizing means for recognizing an amount of an allowable band allowed for the transmission by the network according to the congestion state, and (2) photographing a subject. Shooting means for generating a time-series shooting screen;
(3) Encoding the photographing screen generated by the photographing means,
Photographing screen encoding means for obtaining time-series screen information constituting the video information; (4) screen information amount detecting means for detecting the information amount of each screen information; and (5) when the allowable band is small. Changing the optical or spatial relationship between the photographing unit and the subject while referring to the information amount of the screen information, thereby causing the photographing unit to generate a photographing screen having a small information amount of the screen information. Control means.

Further, according to the fourth invention, in a communication system including a receiving device for receiving at least video information transmitted by the transmitting device as reception information via a network, the receiving device includes: Bandwidth-related information acquisition means for obtaining bandwidth-related information relating to the amount of allowable bandwidth allowed for transmission of the video information by the network according to the content of the received information; Control information transmitting means for transmitting control information including the band-related information to the transmitting device so as to allow the transmitting device to recognize the amount of allowable bandwidth permitted for the transmission according to (4). ) The transmitting device comprises:
The transmission device according to any one of claims 1 to 3.

[0022]

BEST MODE FOR CARRYING OUT THE INVENTION (A) Embodiment Hereinafter, a first embodiment relating to a transmitting apparatus and a communication system of the present invention will be described.
Third to third embodiments will be described.

In the first embodiment, the type of media used for communication is changed according to the allowable bandwidth that can be secured.

(A-1) Configuration of First Embodiment FIG. 1 shows the configuration of a multimedia communication system 10 of the present embodiment. The multimedia transmitting unit 101 may be a part of a general-purpose communication terminal such as a personal computer terminal, or may be a part of a dedicated communication device. The same applies to the multimedia receiving unit 120 that communicates with the transmitting unit 101 via the IP network 11.

The relationship between the video input signal VS and the audio input signal AS input to the transmission unit 101 is a video signal and an audio signal that constitute a moving image with audio such as movie software and are closely related to each other. Alternatively, they may be completely independent and unrelated video signals and audio signals, and may be signals that happen to be transmitted simultaneously by the same transmitting unit 101.

In FIG. 1, a multimedia transmitting unit 10
1 is a video encoding unit 102 and an audio encoding unit 103
, A speech-to-text converter 104, and a multiplexing controller 105
And

The video encoding unit 102 is a part that receives the video input signal VS, encodes the data so as to compress the data amount, and outputs the encoded data to the multiplexing control unit 105 as encoded video data CV. Here, the video encoding method is not particularly limited, and an MPEG method or the like can be used. In the present embodiment, JPEG is used as an example.

[0028] The speech encoding unit 103 outputs the speech input signal AS.
Is input and encoded to compress the data amount, and is output to the multiplexing control unit 105 as encoded audio data CA. Here, the audio encoding method is not particularly limited, but in the present embodiment, G723.1, which is the ITU-T standard, is used as an example. According to the present encoding method, the audio input signal AS can be compressed to a data amount of 5 to 6 Kbps.

The speech-to-text conversion unit 104 receives the speech input signal AS, performs recognition processing on the input phoneme sequence, converts the input phoneme sequence into a text signal, and converts the text signal into text conversion data CT.
Is output to the multiplexing control unit 105.

Although the recognition method is not particularly limited, in the present embodiment, as an example, a recognition target phoneme and standard frequency spectrum information of each syllable transition portion are prepared in advance, and the speech input signal AS in each frame is prepared. It is possible to use a method of calculating a frequency spectrum, calculating a correlation with the previously prepared standard frequency spectrum, and obtaining a phoneme sequence giving a correlation equal to or greater than a predetermined threshold value as a recognition result.

When the audio signal AS is converted to a text signal, the data amount is assumed to include, for example, eight syllables per second. If the character code per syllable is 8 bits, the data amount is 64 bits per second, and the accent, Even if the same amount of control information such as intonation is given,
The data amount becomes 0.5 Kbps or less.
It is possible to compress the data amount to one digit less than one encoding.

Next, as shown in FIG. 4, the multiplexing control unit 105 includes a selector 12, a comparison / determination unit 13, a transmission data amount storage unit 14, and a header control unit 15.

The selector 12 includes a comparison / determination unit 13
The selection is switched among the three input data in accordance with the selector control signal SE supplied from. The three input data are video encoded data CV supplied from the video encoding unit 102, audio encoded data CA supplied from the audio encoding unit 103, and text conversion data CT supplied from the audio text conversion unit 104. It is.

The comparing / determining unit 13 transmits the transmission control signal SC from the multimedia receiving unit 120 via the network 11.
In the transmission data amount storage unit 14, the transmission data amount (or transmission data rate) of the past transmission encoded data CD and the reception side (reception unit) In step 120), the size of the band (allowable band) usable in the network 11 is recognized in real time by comparing the received data amount (or the received data rate or the packet loss ratio) actually received.

That is, the transmission control signal SC has information on the allowable bandwidth of the communication network 11.
This information includes, for example, a packet loss rate, an encoded data rate that can be actually received by the receiving unit 120, and the like.

In the process of the comparing and judging unit 13, the reception data rate actually received by the reception unit 120 is lower than the transmission data rate transmitted by the transmission unit 101, or the reception data rate is smaller than the transmission data amount. It is determined that the smaller the amount (that is, the larger the packet loss ratio), the higher the congestion degree of the network 11 and the smaller the allowable bandwidth.

When it is determined that the allowable band is too small to transmit the video coded data CV, the comparison / determination unit 13 converts the video coded data CV and the audio coded data CA into the transmission coded data CA. In order not to output as a CD, the text conversion data CT
Is output as transmission encoded data CD.

In general, when the packet loss rate exceeds 20%, it is impossible to output the video coded data CV to the communication path, and the packet of the voice coded data CA is also locally lost. Probability is high. in this case,
On the receiving unit 120 side, a sound skipping occurs in the sound that the user is listening to, giving the user discomfort.

The selector control signal SE causes the selector 12 to select the text conversion data CT. As described above, the data amount of the text conversion data CT with respect to the audio input signal AS having the same content is the audio encoded data CA.
For example, the data amount becomes about one tenth as compared with the data amount.

In other words, by converting the input speech signal AS into text data CT by speech recognition, the data amount is reduced to about 1/10 compared to the standard speech coding. It can be considered that the compression ratio has been increased until.

In a situation where the packet loss ratio exceeds 20%, since the allowable bandwidth is extremely small, the transmitting unit 101
If you send a packet from, the packet transmission itself,
It is also considered that this acts as a factor that increases the loss ratio of the subsequent packet transmitted from the transmitting unit 101,
In order to maintain the communication quality, it is effective to select the text conversion data CT having a small data amount and a small use band as the transmission encoded data CD.

The header generation section 15 provided in the multiplex control section 105 is a section for generating and outputting a header of a packet transmitted from the multiplex control section 105. Therefore, when the selector control signal SE causes the selector 12 to select the header generation unit 15, the header portion of the packet is transmitted to the network 11.

That is, the selector 12 is provided with the header generation unit 15
Is selected, the packet header is output. When the video encoder 102, the audio encoder 103, or the audio / text converter 104 is selected, the data portion of the packet following the packet header is output. Will be done.

At the time of such packetization, the transmission encoded data CD is packetized in an appropriate unit, which may be, for example, a frame unit.

The header of each packet generated by the header generation unit 15 is provided with control information such as a packet number indicating the output order between packets, media identification information, and packet length.

On the other hand, a multimedia receiving unit 120 having a function and a structure corresponding to the multimedia transmitting unit 101 includes a separation control unit 121 and a video decoding unit 122.
, A voice decoding unit 123, and a text-to-speech conversion unit 124.

As shown in FIG. 5, the separation control unit 121 includes an identification distribution unit 21 and a transmission control signal transmission unit 22.
And a received data amount detection unit 23.

The identification / distribution unit 21 receives a packet including the transmission coded data CD via the network 11 and decomposes the packet, and, based on the media identification information included in the packet header, a data portion of the packet. Is provided with a function of identifying whether the content of the data is video (that is, video encoded data CV), audio (that is, audio encoded data CA), or text (that is, text conversion data CT).

Then, according to the identification result, the identification distribution unit 2
1 supplies the encoded video data CV to the video decoding unit 122, supplies the encoded audio data CA to the audio decoding unit 123, and converts the text conversion data CT to the text / audio conversion unit 1.
24.

The transmission coded data CD actually received by the multimedia receiving section 120 is the received data. With respect to the received data, the received data amount and the received data rate are detected by the received data amount detection unit 23.

The reception data amount detection unit 23 checks the packet header of the packet having the transmission coded data CD in the data part, calculates the packet loss ratio from the packet number, and determines the reception data rate from the packet length. calculate.

The transmission control signal transmission section 22 is a section for transmitting the transmission control signal SC. In this transmission, the transmission control signal transmission unit 22 calculates the reception data amount detection unit 23,
The received packet loss ratio, received data rate, received data amount, and the like are received, and a transmission control signal SC is configured based on these.

The transmission control signal SC is also transmitted by the transmission unit 10
As in 1, the transmission encoded data CD is packetized by a packet assembling unit (not shown) and transmitted to the network 11.

The video encoding data C from the identification distribution unit 21
The video decoding unit 122 receiving the input of V
01, a video output signal VS is output by decoding using a method corresponding to the coding method used by the video coding unit 102 in the video encoding unit 102. In the present embodiment, the video coding unit 102
Adopted the JPEG method, the video decoding unit 12
In 2, the moving image is reproduced using the Motion-JPEG method.

In the present embodiment, the output of the video output signal VS is based on the video encoded data CV supplied immediately before the interruption even if the supply of the video encoded data CV included in the transmission encoded data CD is interrupted. Shall be continued.

When the image in the frame is stationary,
When the motion of the image is slow or small, even if the video coded data CV is interrupted, the user who is viewing the video is less likely to cause discomfort, and the video is interrupted when the motion is large. It is considered better to display the immediately preceding image even if no image is displayed, when no image is displayed.

The voice encoding data CA from the identification distribution unit 21
The voice decoding unit 123 receiving the input of the
Of the ITU-T standard G723.1, which is used by the audio encoder 103, and outputs an audio output signal AS.

In this embodiment, the transmission coded data C
It is assumed that the audio signal AS is not output except during the input of the audio encoded data CA included in D. Therefore, when the input of the voice coded data CA is interrupted, the receiving unit 120
On the side, skipping will occur.

In general, a video does not give a user discomfort even if the same image is continuously displayed, but if the same sound is repeatedly output, the user is likely to give a great discomfort. It is. Therefore, when the data reception is interrupted, in the present embodiment, the image continues to be displayed as it is immediately before, and the output of the sound is stopped.

Finally, the text-to-speech conversion unit 124
Receives the text conversion data CT from the identification distribution unit 21, converts the text conversion data into voice data, and outputs a voice output signal AS. In converting the text conversion data CT into the audio signal AS, a speech synthesis technique can be used.

In this embodiment, the transmission coded data C
Except during the input of the text conversion data CT included in D,
It is assumed that the audio signal AS is not output.

Also, if necessary, the text conversion data CT is not converted to the audio signal AS, and the
The screen may be displayed on T or the like, and both the conversion to the audio signal AS and the screen display may be performed simultaneously.

When the text is displayed on the screen, even if the supply of the text conversion data CT included in the transmission encoded data CD is interrupted, the screen display immediately before the interruption may be continued.

(A-2) Overall Operation of the First Embodiment When the allowable bandwidth for the transmission unit 101 and the reception unit 120 of the IP network 11 is sufficiently large, the comparison and determination unit 13 uses the selector control signal SE to generate a header. In addition to the output of the generation unit 15, the video encoded data CV
And the audio encoded data CA are both selected.

Thus, the encoded video data CV and the encoded audio data CA are used as the encoded transmission data CD.
Is selected with an appropriate time distribution, and the header generation unit 15 is selected for each packet. The contents of the time distribution are
It is determined according to the data rate of the encoded video data CV and the data rate of the encoded audio data CA.

For example, when the video of the same movie software is used as the video input signal VS and the audio is used as the audio input signal AS, the video has a much higher data rate.
Most of the time, video encoded data CD will normally be selected.

At this time, the receiving section 120 outputs both the video output signal VS and the audio output signal AS without any problem, so that the user watching the movie software can continue to view comfortably.

However, when the degree of congestion of the network 11 increases and the allowable bandwidth becomes extremely small, the packet loss ratio of the video encoded data CV and the audio encoded data CA becomes extremely high, and the same image is displayed on the screen. Subsequently, the voice shows a tendency for skipping to occur frequently.

At this time, based on the content of the received transmission control signal SC, the comparison / determination section 13 performs
04 will be selected. Voice text converter 104
Has a sufficiently low data rate compared to the encoded voice data CA, so that communication with almost no packet loss is possible even in a situation where the allowable bandwidth is extremely small.

As described above, the text conversion data CT
Is 10 times higher than that of the encoded voice data CA.
Although it can be reduced to about one-half, it is considered that the fact that the transmission of the video coded data CV having a much higher data rate than the voice coded data CA is stopped here has a large effect. The data rate of the text conversion data CT is extremely small when compared with the total data rate of the video coded data CV and the voice coded data CA, and can be set to be much less than one hundredth.

Even in such a situation where the allowable band is extremely small, the user on the receiving unit 120 side can continue to listen to the synthesized speech without interruption according to the audio signal AS output from the text-to-speech conversion unit 124. Assuming that there is almost no packet loss, the meaning content transmitted by the synthesized voice changes without interruption or repetition and changes comfortably with the progress of the movie software or the like.

(A-3) Effects of the First Embodiment As described above, according to the present embodiment, even when the communication network is congested, for example, in the Internet environment during the daytime. This enables relatively comfortable communication of multimedia information.

In the present embodiment, the speech recognition rate in the speech-to-text conversion unit (104) of the transmission unit is not necessarily 1.
Although it is not 00%, if the recognition rate is 80% to 90%, it is sufficiently possible to identify the content on the receiving side. Therefore, a high-performance multimedia communication device that operates in a wide range of network environments can be realized.

(B) Second Embodiment In this embodiment, a frame having a large amount of scene change of an input video is preferentially subjected to data compression conversion to reduce the data rate of the transmission coded data, and has a limited tolerance. The band is used effectively.

(B-1) Configuration and Operation of the Second Embodiment FIG. 2 shows a multimedia communication system 30 of this embodiment.
Is shown. The multimedia transmitting unit 201 included in the multimedia communication system 30 corresponds to the multimedia transmitting unit 101 of the first embodiment, and the multimedia receiving unit 220 corresponds to the multimedia receiving unit 120.

In FIG. 2, the multimedia transmitting unit 20
1 is a video encoding unit 202 and an audio encoding unit 203
, A multiplexing control unit 205, and a change amount detecting unit 206. The multimedia receiving unit 220 includes a demultiplexing control unit 221, a video decoding unit 222, and an audio decoding unit 223.

Each part of this embodiment shown in FIG. 2 corresponds to each part of the first embodiment with a corresponding reference numeral.

That is, the video encoding section 202 corresponds to the video encoding section 102, the audio encoding section 203 corresponds to the audio encoding section 103, and the multiplex control section 205 corresponds to the multiplex control section 105. The separation control unit 221 corresponds to the separation control unit 121, the video decoding unit 222 corresponds to the video decoding unit 122, the audio decoding unit 223 corresponds to the audio decoding unit 123, and the network 31 corresponds to the network. 11 is supported.

In particular, the speech encoder 203 and the speech encoder 103, the network 31 and the network 1
1. The video decoding unit 222 and the video decoding unit 122, and the audio decoding unit 223 and the audio decoding unit 123 are exactly the same in terms of function. Therefore, in this embodiment,
As video encoding methods, for example, JPEG method, MPE
The G system can be used, and, for example, ITU-T standard G723.1 can be used as the audio coding system.

Further, each signal in the present embodiment often corresponds to each signal in the first embodiment. That is, the encoded video data CV1 corresponds to the CV, the encoded audio data CA1 corresponds to the CA, the encoded transmission data CD1 corresponds to the CD, the transmission control signal SC1 corresponds to the SC, The input (output) signal VS1 is
S, and the audio input (output) signal AS1 corresponds to the AS.

In FIG. 6 showing the internal configuration of the multiplexing control unit 205, the multiplexing control unit 205 includes a transmission data amount storage unit 14, a header generation unit 15, and a selector 32.
And a comparison determination unit 33.

The selector 32 corresponds to the selector 12, and the comparison / determination section 33 corresponds to the comparison / determination section 13. However, the selection switching performed by the selector 12 of the first embodiment is performed when the allowable band is large, and the video encoded data CV or the audio encoded data CA and the header generation unit 15
Is selected, and the text conversion data CT and the header generation unit 15 are selected when the allowable bandwidth is reduced. However, the selection switching of the selector 32 in the present embodiment is an operation for packetization only, and the size of the allowable bandwidth is Even if it fluctuates, it does not substantially change.

In the multiplexing control unit 205, the header generation unit 15 and the transmission data amount storage unit 14 are the same in terms of function as the header generation unit and the transmission data amount storage unit of the first embodiment. Signs are attached.

The other components and the like also have many points in common in terms of function.
Only the points different from the above embodiment will be described.

This difference is substantially limited to a portion mainly related to the change amount detecting section 206.

In FIG. 6 showing the internal configuration of the multiplexing control unit 205, the comparison / determination unit 33 sends a transmission control signal SC from the multimedia receiving unit 220 via the network 31.
The transmission data amount (or transmission data rate) of the past transmission encoded data CD1 stored in the transmission data amount storage unit 14 and the transmission control signal SC
1 by comparing the received data amount (or received data rate, packet loss rate, etc.) actually received by the receiving side (receiving unit 220) brought by the No. 1 in real time. This is similar to the comparison / determination unit 13 in that the band information signal BS1 is output to the change amount detection unit 206 instead of the selector 32.

This band information signal BS 1 is a signal for controlling the operation of the change amount detecting section 206, including information corresponding to the information on the allowable band carried by the transmission control signal SC 1. However, the band information signal BS1 is effective only when the allowable band is small and the transmission data rate needs to be reduced. Therefore, when the size of the allowable band is sufficient, the comparison determination unit 33 determines whether the band information signal BS1 The transmission itself may not be performed.

The change amount detector 203 receiving the band information signal BS1 receives the same video input signal VS1 as the video encoder 202, and performs video encoding according to the contents of the video input signal VS1 and the band information signal BS1. The coding control signal CC for controlling the operation of the unit 202 is changed.

In the operation of the change amount detecting section 206, for example, when the packet loss ratio included in the band state signal BS1 falls below a predetermined value (1-2%), it is necessary to reduce the amount of data output to the communication path. Since there is no encoding control signal CC
Turns on. On the other hand, when the packet loss ratio exceeds a predetermined value, in each frame of the video input signal VS1,
Detect a frame with a large amount of change from the adjacent frame,
The coding control signal CC is turned on only for that frame, and the coding control signal CC is turned off for frames before and after that.

As a change amount detection method, for example, a density distribution (density histogram) of each pixel of each frame is calculated, and a frame whose difference value from the density histogram of the immediately preceding frame exceeds a predetermined threshold value has a large change amount. A method of determining is used. The density histogram is a histogram in which the horizontal axis indicates the tone value, and the vertical axis indicates the frequency of appearance of each tone value (the number of appearing pixels of each density value on the screen). It appears as a change in the density histogram, and the larger the change in the density histogram, the larger the change in the image.

The specific number of gradations may be, for example, 256 gradations of 8 bits.

The difference between the frames of the density histogram is represented by the sum of the absolute values of the differences in the number of appearances of each density between the frames. Generally, a frame having a large difference between the density histogram frames is determined by the adjacent (previous) frame. It has the property that the change from is large. A frame having a large change from the immediately preceding frame contains a lot of information useful to the user in the video scene input in time series.

In other words, the multimedia receiving unit 22
On the 0 side, even if a large number of frames are received and decoded in time series as the video coded data CV1, if there is almost no change in the image in each frame, 1
Compared to the case where only the frame is received and the frame is continuously displayed on the screen, the viewing user cannot perceive a great difference, but cannot receive and decode a frame having a large change from the immediately preceding frame. If so, the user is likely to feel great discomfort.

For example, in a situation where the packet loss exceeds 10%, the video encoded data C
Normally, it is difficult to normally receive and decode all of V1. However, if all frames, both large and small, are transmitted from the immediately preceding frame, both large and small frames are equal. Because of the probability of being lost due to packet loss, the utilization efficiency of the small allowable bandwidth is substantially reduced, and on the user's experience,
Satisfaction with image display decreases, but if only frames with a large amount of change are selectively transmitted, only packets having frames that are truly important to the user are transmitted, and the transmission data rate decreases. As a result, it can be expected that the packet loss itself is reduced and the user satisfaction is also increased.

The coding control signal C from the change amount detecting section 206
The video encoding unit 202 receiving the supply of the C performs the encoding process when the encoding control signal CC is on, outputs encoded video data CV1, and outputs the encoded control signal CCV.
Is off, the encoding process is not performed, and the encoded video data CV1 is not output.

Therefore, if the above-mentioned predetermined threshold value relating to the difference value from the density histogram of the immediately preceding frame is made to have a proportional relationship with the packet loss ratio, the packet loss ratio brought by the transmission control signal SC1 is high, Is smaller, the data rate of the encoded video data CV1 can be reduced, and the data rate of the encoded transmission data CD1 transmitted from the transmitting unit 201 to the network 31 can be reduced.

In the present embodiment, the internal configuration of the separation control unit 221 in the multimedia receiving unit 220 may be the same as the internal configuration of the separation control unit 120 shown in FIG. However, in this embodiment, since the text conversion data CT is not used, the function of the identification and distribution unit 21 for identifying and distributing the text conversion data CT can be omitted.

(B) Effects of the Second Embodiment As described above, according to the present embodiment, when the available bandwidth decreases due to the network congestion, the frame is not simply thinned out, Therefore, it is expected that a high-performance multimedia communication device that effectively uses an allowed band can be realized because a frame including information important to the user is selected and encoded with priority.

(C) Third Embodiment In this embodiment, zooming and panning of a camera are performed.
The present invention is characterized in that the data rate of the transmission coded data is reduced and a limited allowable band is used effectively.

(C-1) Configuration and Operation of Third Embodiment FIG. 3 shows a multimedia communication system 40 according to the third embodiment.
Is shown. The multimedia transmitting unit 301 constituting the multimedia communication system 40 corresponds to the multimedia transmitting unit 201 of the second embodiment, and the multimedia receiving unit 320 corresponds to the multimedia receiving unit 220.

In FIG. 3, the multimedia transmitting unit 30
1 is a video encoding unit 302 and an audio encoding unit 303
And a multiplexing control unit 305, a camera control unit 306, and a camera 307.
0 includes a separation control unit 321, a video decoding unit 322, and an audio decoding unit 323.

Each part of this embodiment shown in FIG. 3 corresponds to each part of the second embodiment with a corresponding reference numeral.

That is, the video encoding unit 302 corresponds to the video encoding unit 202, the audio encoding unit 303 corresponds to the audio encoding unit 203, and the multiplexing control unit 305 corresponds to the multiplexing control unit 205. The separation control unit 321 corresponds to the separation control unit 221; the video decoding unit 322 corresponds to the video decoding unit 222; the audio decoding unit 323 corresponds to the audio decoding unit 223; 31.

In particular, the speech encoder 303 and the speech encoder 203, the network 41 and the network 3
1. The video decoding unit 322 and the video decoding unit 222, the audio decoding unit 323 and the audio decoding unit 223 include a multiplexing control unit 305, the multiplexing control unit 205, and a demultiplexing control unit 321.
And the separation control unit 221 are exactly the same in terms of their functions. Therefore, also in the present embodiment, for example, the JPEG system or the MPEG system can be used as the video coding system, and the ITU-T
Standard G723.1 can be used.

The signals in this embodiment also often correspond to the signals in the second embodiment. That is, the encoded video data CV2 corresponds to the CV1, the encoded audio data CA2 corresponds to the CA1, the encoded transmission data CD2 corresponds to the CD1, and the transmission control signal SC2
Corresponds to the SC1, and the band information signal BS2 is the BS1.
1 and the video input (output) signal VS2 is
1 and the audio input (output) signal AS2 is the AS1
Corresponding to

As described above, the multiplexing control unit 305 is exactly the same in function as the multiplexing control unit 205, and FIG. 6 also shows the internal configuration of the multiplexing control unit 305 as it is.

[0107] The other components and the like also have many points in common in terms of function.
Only the points different from the above embodiment will be described.

This difference is substantially limited mainly to the portion related to the camera control section 306.

The video input signal VS1 of the second embodiment
May be a video signal obtained by reproducing an image that has already been recorded. However, the video input signal VS2 of the present embodiment can be obtained by capturing an image of a subject T by using a video camera 307 for capturing a moving image. It is assumed that it can be obtained.

The camera 307 includes a camera control unit 306.
In response to the camera control signal VC2 supplied from the camera, the mounted optical system and its own direction are automatically changed, and control data on zoom, pan, tilt, focus, and the like can be changed.

The video encoder 302, which has received the video input signal VS2 obtained by the camera 307 taking an image of the subject T, encodes the video input signal VS2 to obtain, for example, a video code obtained for each frame. The coded data CV2 is sent to the multiplexing control unit 305, and the code amount COV of the frame is output to the camera control unit 306.

The camera control unit 306 receives the band information signal BS2 and the code amount COV, and
C2 is output to the camera 307. As in the case of BS1, if the allowable band is sufficiently large, the band information signal BS2 may not be transmitted.

When the packet loss ratio included in the band status signal BS2 is lower than a predetermined value (1-2%), the camera control unit 306 determines that the video signal can be communicated in a stable state. The content of the camera control signal VC2 is not changed. On the other hand, if the packet loss ratio exceeds a predetermined value, the control data such as zoom and rotation (corresponding to pan and tilt) are changed to calculate camera input conditions that can further reduce the code amount COV, and the camera control signal VC2 Change the content of

As a result, the photographing condition of the subject T by the camera 307 (that is, the camera input condition) changes. For example, the zoom is switched in four steps and the pan is switched in about four steps, and video input is performed under the condition that gives the minimum code amount COV. If the image may be slightly blurred, the image quality may be blurred by shifting the focus point.

When the focus point is shifted, the high frequency component is removed, which is effective in reducing the code amount COV.
07, the angle and the brightness of the subject T change, and for a frame obtained by photographing the same subject T, for example, the density histogram also changes.
It is possible to obtain low V frames.

When the code amount COV of each frame constituting the coded video data CV2 decreases, the data rate of the coded transmission data CD2 decreases, and the packet loss ratio can be reduced.

According to the present embodiment, for the user watching the screen corresponding to the video output signal VS2 on the multimedia receiving unit 320 side, the transmitting unit 301 controls the camera 307 regardless of his / her intention. Subject T in the screen
May change in position or size, or the focus state may change, but even when the allowable band is small, the subject T
This is advantageous in that the possibility that the user can continue to view the image increases.

(C) Effects of the Third Embodiment According to the present embodiment, a system such as a TV conference application is unlikely to significantly reduce user satisfaction even if camera input conditions are slightly changed. In such cases, it is possible to obtain optimal input conditions without increasing the system scale, and even when the allowable bandwidth is small,
It is possible for the user to continue looking at the subject without impairing the real-time property.

(D) Other Embodiments In the first embodiment, when the allowable band becomes extremely small, both the selection of the video encoded data CV and the selection of the audio encoded data CA are not performed. But
Even if the selection of the video encoded data CV is not continued and only the audio encoded data CA is selected, and the selection of the text conversion data CT is performed, the data rate of audio is reduced. Can be expected.

Further, the speech encoding unit 10 of the first embodiment
3 and the audio-to-text converter 104, and the multimedia transmitter including the video encoder 202 and the variation detector 206 of the second embodiment.

When the allowable band is sufficiently large, the multimedia transmitting section encodes all the frames by video encoding section 202 and outputs the output of audio encoding section 103 to a multiplexing control section (multiplexing control section 105). However, when the allowable band is reduced, only the frame with a large change amount may be encoded, and the output of the speech-to-text conversion unit 104 may be selected by the multiplexing control unit.

The main part of the first embodiment and the main part of the third embodiment can coexist in the same multimedia transmitting unit.

That is, in the multimedia transmitting unit 301 of the third embodiment, the audio encoding unit 303 is
It is also possible to use a configuration in which the voice encoding unit 103 and the voice / text conversion unit 104 of the embodiment are replaced.

Further, the main part of the second embodiment and the main part of the third embodiment can coexist in the same multimedia transmitting unit.

In this case, when the allowable band is reduced, it is possible to selectively encode only a frame having a large image change amount, and to control the camera input conditions so as to reduce the transmission data rate.

In the third embodiment, zoom, pan, tilt, focus, and the like are performed in order to change the camera input condition. However, it is not necessary to perform control for all of them. Only the change of the focus state may be performed. Conversely, it is also possible to control the conditions not listed here.

In the third embodiment, the change of the camera input condition is automatically executed by the system.
For example, an operator operating the camera based on an instruction from the system may execute the operation manually.

Further, the transmission control signal S of the third embodiment
C2 specifies not only information such as the amount of received data, the received data rate, and the packet loss ratio actually received by the receiving unit 320, but also the user of the receiving unit 320 specifying the camera input conditions of the camera 307 for the transmitting unit 301. May be included. In this case, if the allowable band is small, the camera control unit 306 arbitrates between the camera input condition change request and the above-described request for reduction of the frame code amount COV generated in the transmission unit 301, and performs actual camera input. The conditions will be decided.

In the first to third embodiments, the multimedia transmitting unit transmits both video and audio to the network. However, in the first embodiment, the components for transmitting video are omitted. Second and third
In the embodiment, components for transmitting voice may be omitted.

That is, according to the present invention, a transmitting device that transmits video information and / or audio information as transmission information, and transmits the transmission information to a receiving device via a network. The present invention can be widely applied to a transmitting device that transmits to a receiving device via a network, or a communication system including a receiving device that receives at least video information transmitted by the transmitting device as receiving information via a network.

[0131]

As described above, according to the first to fourth aspects of the present invention, normal communication of video information or audio information can be continued even if the allowable bandwidth is reduced according to the congestion state of the network. The probability is high, and the reliability of communication is improved.

Further, according to the first or fourth aspect, communication can be continued even when the allowable band is extremely small.

Further, in the second or fourth aspect of the present invention, the priority is given to the communication of a screen that is truly important to the user, so that the utilization efficiency of the allowable band can be substantially improved.

Further, the third or fourth invention has a relatively small scale and is also excellent in real-time properties and the like.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a schematic configuration of a multimedia communication system according to a first embodiment.

FIG. 2 is a block diagram illustrating a schematic configuration of a multimedia communication system according to a second embodiment.

FIG. 3 is a block diagram illustrating a schematic configuration of a multimedia communication system according to a third embodiment.

FIG. 4 is a block diagram illustrating a schematic configuration of a multiplexing control unit according to the first embodiment.

FIG. 5 is a block diagram illustrating a schematic configuration of a separation control unit according to the first embodiment.

FIG. 6 is a block diagram illustrating a schematic configuration of a multiplexing control unit according to the second or third embodiment.

[Description of References] 10, 30, 40 ... Multimedia communication system, 1
1, 31, 41: network, 13, 33: comparison / determination unit, 14: transmission data amount storage unit, 102, 202, 30
2. Video encoder, 101, 201, 301 Multimedia transmitter, 103, 203, 303 Audio encoder, 104 Audio text converter, 120, 220, 3
20: multimedia receiving unit, 206: change amount detecting unit,
306: Camera control unit, 307: Camera, VS, VS
1, VS2: video signal, AS, AS1, AS2: audio signal, SC, SC1, SC2: transmission control signal, CD, C
D1, CD2: coded transmission data, T: subject.

Claims (4)

[Claims] 1. A transmitting apparatus for transmitting video information and / or audio information as transmission information and transmitting the transmission information to a receiving apparatus via a network, wherein the network allows the transmission according to the congestion state. Band recognition means for recognizing the amount of allowable band; video coding means for generating the video information by inputting and coding a video signal; and inputting and coding an audio signal to the audio information. Voice data encoding means for generating corresponding voice data information; converting an input voice signal into a text signal by voice recognition and coding the text signal to generate voice text information corresponding to the voice information. An audio text encoding unit that transmits the video information and the audio data information as the transmission information when the allowable band is large. , Wherein when the tolerance band is small, the transmission device being characterized in that a transmission information selecting means for transmitting the voice text information as the transmission information. 2. At least video information is formed as transmission information,
A transmitting apparatus for transmitting the transmission information to a receiving apparatus via a network, wherein the network recognizes an amount of an allowable bandwidth allowed for the transmission according to the congestion state; and the video information is a moving image. If the image is an image, it is determined that, from among a plurality of time-series screens constituting the moving image, a screen having a larger amount of change in the image in the screen is determined to have a higher importance, and a screen having a higher importance is selected. Screen selecting means, if the allowable band is large, transmit all the time-series screens as the transmission information, and if the allowable band is small, select the important screen from a plurality of time-series screens. Transmitting means for transmitting only the screen selected by the means as the transmission information. 3. At least video information is formed as transmission information,
A transmitting device that transmits the transmission information to a receiving device via a network, wherein the network recognizes an amount of an allowable band allowed for the transmission according to the congestion state, and captures an object. A photographing means for generating a time-series photographing screen; a photographing screen encoding means for encoding the photographing screen generated by the photographing means to obtain time-series screen information constituting the video information; and information of each screen information. Screen information amount detecting means for detecting the amount, when the allowable band is small, by changing the optical or spatial relationship between the imaging means and the subject while referring to the information amount of the screen information, A transmitting apparatus, wherein the photographing means includes a photographing control means for generating a photographing screen having a small information amount of the screen information. 4. In a communication system including a receiving device that receives at least video information transmitted by a transmitting device as reception information via a network, the receiving device performs the following processing in accordance with the content of the reception information actually received. Band-related information obtaining means for obtaining band-related information relating to the amount of allowable band allowed by the network for transmission of the video information; and allowable band allowed by the network for the transmission according to the congestion state. Control information transmitting means for transmitting control information including the band-related information to the transmitting device so that the transmitting device recognizes the amount of the transmission device. The transmitting device according to any one of claims 1 to 3, A communication system, being a transmission device.