JP2007318283A - Packet communication system, data receiver - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a packet communication system and a data receiver capable of reducing difference in delay time between voice packets and video image packets, while maintaining audio quality. <P>SOLUTION: The packet communication system comprises a data transmitter and a data receiver. The data receiver comprises an accumulation amount detection means, an information amount calculation means, an information amount determination means, and an information amount determination means. The accumulation amount detection means detects the amount of video image packets accumulated in a video image jitter buffer. The information amount calculation means calculates the amount of information of voice packets. The information amount determination means determines whether the amount of information calculated by the information amount calculation means is below a predetermined value or not. A reproduction control means changes a reproduction time duration for the voice packets determined as the ones below the predetermined value by the information amount determination means, according to the amount of video image packets detected by the accumulation amount detection means. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a packet communication system and a data receiving device.

  In a system that transmits and receives video and audio over a communication line, such as a TV conference system, it is important to synchronize the movement of the speaker's mouth and the voice reproduced from the speaker in order to eliminate the uncomfortable feeling of the call. So-called lip sync adjustment is performed.

  For example, a method has been proposed in which the output timing of video and audio is always matched to changes in image quality and images by detecting the delay amount of the video signal and changing the delay amount of the audio signal in accordance with the delay amount. (For example, refer to Patent Document 1).

  Also, for example, the timing signal recorded at the same time as the timing signal recorded on the video signal received on the receiving side is recorded on the video signal and the audio signal at the same time and transmitted on separate lines. There has been proposed a method of reproducing an audio signal possessed by a video signal (for example, see Patent Document 2).

  However, in Patent Documents 1 and 2, since the audio signal is reproduced in accordance with the video signal, there is a problem that if the audio signal is delayed with respect to the video signal, the sound is interrupted and the reproduction is difficult to hear.

In order to deal with such problems, the same unique mark is attached to the video signal and audio signal generated at the same time, packetized and transmitted, and the video packet delayed from the audio packet is discarded on the receiving side. Therefore, a method of synchronizing and reproducing audio and video while maintaining audio quality has been proposed (see, for example, Patent Document 3).
JP-A-7-184182 JP 2001-24992 A Japanese Patent Laid-Open No. 7-50818

  However, in the method of Patent Document 3, when the audio packet is delayed from the video packet, priority is given to the reproduction of the audio packet, so that the audio is reproduced but the video is not reproduced at all.

  The present invention has been made in view of the above problems, and a packet communication system capable of reproducing video in synchronization with audio while maintaining the quality of the audio even when the audio packet is delayed from the video packet, and data reception It is an object to provide a device.

1.
A video packet encoder that packetizes video signals into video packets;
A voice packet encoder that packetizes voice signals into voice packets;
A time stamp giving unit for giving time information to video packets and audio packets;
A data transmission device comprising: a transmission unit that sequentially transmits the video packet and the audio packet to which the time information is attached;
A receiving unit for receiving the video packet and the audio packet transmitted via a transmission line;
A video jitter buffer for sequentially storing the video packets received by the receiver;
A voice jitter buffer for sequentially storing the voice packets received by the receiver;
A video packet decoder for decoding and reproducing the video packet;
An audio packet decoder for decoding and reproducing the audio packet;
Time information detecting means for detecting the time information;
Based on the time information detected by the time information detection means, a data receiving device comprising the video packet decoder, and a reproduction control means for instructing the audio packet decoder to reproduce,
In a packet communication system having
The data receiving device is:
Accumulated amount detecting means for detecting the amount of image packets accumulated in the image jitter buffer;
An information amount calculating means for calculating an information amount of an audio packet to be reproduced by the audio packet decoder;
An information amount determination means for determining the information amount calculated by the information amount calculation means;
The reproduction control means includes
A packet communication system characterized in that, according to the amount of video packets detected by the accumulated amount detection means, the reproduction time of the audio packet determined by the information amount determination means to be an information amount equal to or less than a predetermined value is changed. .

2.
The reproduction control means includes
When the amount of video packets detected by the accumulated amount detection means is a predetermined value or more,
2. The packet communication system according to 1, wherein voice packets whose information amount is equal to or greater than a predetermined value are sequentially reproduced without reproducing the voice packet determined by the information amount determination means as being an information amount equal to or smaller than a predetermined value. .

3.
The reproduction control means includes
3. The packet communication system according to 2, wherein a video packet stored in the video jitter buffer having time information older than an audio packet to be reproduced is discarded.

4).
The reproduction control means includes
When the amount of video packets detected by the accumulated amount detection means is below a predetermined value,
2. The reproduction of audio packets determined by the information amount determination means to be an information amount of a predetermined value or less is repeated according to the amount of video packets, and then the next audio packets are sequentially reproduced. Packet communication system.

5).
The packet communication system according to any one of claims 1 to 4, wherein the information amount calculation means calculates the information amount based on an average value of the audio signal.

6).
The information amount calculation means calculates each frequency component of the audio signal, and calculates the difference between the addition average value of each frequency component value of the audio packet calculated up to the previous time and the value of each frequency component calculated this time. The packet communication system according to any one of 1 to 4, wherein the information amount is calculated based on the information.

7).
A receiver for receiving video packets and audio packets;
A video jitter buffer for sequentially storing the video packets received by the receiver;
A voice jitter buffer for sequentially storing the voice packets received by the receiver;
A video packet decoder for decoding and reproducing the video packet;
An audio packet decoder for decoding and reproducing the audio packet;
Time information detecting means for detecting time information attached to the video packet and the audio packet;
In a data receiving device comprising: the video packet decoder based on the time information detected by the time information detection means; and a reproduction control means for instructing the audio packet decoder to reproduce,
The data receiving device is:
Accumulated amount detecting means for detecting the amount of image packets accumulated in the image jitter buffer;
An information amount calculating means for calculating an information amount of an audio packet to be reproduced by the audio packet decoder;
An information amount determination means for determining the information amount calculated by the information amount calculation means;
The reproduction control means includes
A data receiving device characterized in that, according to the amount of video packets detected by the accumulated amount detecting means, the reproduction time of the audio packet determined by the information amount determining means to be an information amount equal to or less than a predetermined value is changed. .

8).
The reproduction control means includes
When the amount of video packets detected by the accumulated amount detection means is a predetermined value or more,
8. The data receiving apparatus according to claim 7, wherein the information packet determining unit sequentially reproduces the voice packets having the information amount equal to or greater than the predetermined value without reproducing the voice packets determined to be the information amount equal to or smaller than the predetermined value. .

9.
The reproduction control means includes
9. The data receiving apparatus according to 8, wherein the video packet stored in the video jitter buffer having time information older than the audio packet to be reproduced is discarded.

10.
The reproduction control means includes
When the amount of video packets detected by the accumulated amount detection means is below a predetermined value,
8. The reproduction of an audio packet determined by the information amount determination unit to be an information amount equal to or less than a predetermined value is repeated according to the amount of the video packet, and then the next audio packet is sequentially reproduced. Data receiving equipment.

11.
11. The data receiving device according to claim 7, wherein the information amount calculation unit calculates the information amount based on an average value of an audio signal.

12
The information amount calculation means calculates each frequency component of the audio signal, and calculates the difference between the addition average value of each frequency component value of the audio packet calculated up to the previous time and the value of each frequency component calculated this time. 11. The data receiving device according to any one of 7 to 10, wherein the information amount is calculated based on the information amount.

  According to the present invention, the difference between the delay times of the audio packet and the video packet is adjusted by changing the playback time of the audio packet with a small amount of information. Therefore, even if the audio packet is delayed from the video packet, the audio quality Thus, it is possible to provide a packet communication system and a data receiving device that can reproduce video while synchronizing with audio while maintaining video.

  Hereinafter, the present invention will be described in detail with reference to embodiments, but the present invention is not limited thereto.

  FIG. 1 is a block diagram showing an example of a packet communication system according to the present invention.

  FIG. 1A is a block diagram showing the overall configuration, and FIG. 1B is a detailed block diagram of the main microcomputer unit 713.

  A data transmission device 701 and a data reception device 702 in FIG. 1A are connected to a transmission line 717. The data transmission device 701 encodes the input video and audio into audio packets and video packets and transmits them to the transmission line 717, and the data reception device 702 that receives the audio and video packets via the transmission line 717 receives the video and audio. Play.

  The data transmission device 701 includes an audio input unit 703, an audio packet encoder 704, a video packet encoder 705, a video input unit 706, a time stamp assigning unit 707, and a transmission unit 708.

  The voice signal input from the voice input unit 703 is packetized by the voice packet encoder 704 to become a voice packet. Also, the video signal input from the video input unit 706 is packetized by the video packet encoder 705 to become a video packet. Next, the audio packet and the video packet are input to the time stamp adding unit 707, and the time stamp adding unit 707 records the time information of the input time as a time stamp in the audio packet and the video packet. The transmission unit 708 transmits the audio packet and the video packet to which the time stamp is given to the transmission line 717.

  The transmission line 717 is a wired network or a wireless network, and may be either the Internet or an intranet. Further, the transmission path of the video packet and the audio packet in the transmission line 717 is not limited to the same path, and may be transmitted to the data receiving apparatus 702 via different paths.

  The data receiving device 702 includes an audio output unit 709, an audio packet decoder unit 710, a video packet decoder unit 711, a video output unit 712, a main microcomputer unit 713, a reception unit 714, a video jitter buffer 715, and an audio jitter buffer 716. Yes.

  Audio packets and video packets received by the receiving unit 714 via the transmission line 717 are sequentially stored in the audio jitter buffer 716 and the video jitter buffer 715, respectively. The audio jitter buffer 716 and the video jitter buffer 715 are buffer memories provided for correcting variations in arrival times of audio packets and video packets received via the transmission line 717. The received audio packets and video packets are And once stored in the audio jitter buffer 716 and the video jitter buffer 715, respectively.

  The audio packet decoder unit 710 and the video packet decoder unit 711 decode the packetized audio packet and video packet, respectively, reproduce the decoded audio signal and video signal, and output from the audio output unit 709 and the video output unit 712, respectively. To do.

  Next, the main microcomputer unit 713 will be described with reference to FIG.

  The main microcomputer unit 713 includes a microcomputer. That is, the main microcomputer unit 713 includes a CPU (Central Processing Unit) 20 that performs various arithmetic processes, a RAM 21 that is a work area for performing arithmetic, and a ROM 22 that stores a control program and the like, and a data receiving device 702. Centrally controls the operation of each processing unit. As the ROM 22 which is a non-volatile memory, for example, an EEPROM capable of electrically rewriting data is adopted.

  The CPU 20 of this embodiment includes a time information detection unit 10, a reproduction control unit 11, an accumulation amount detection unit 12, an information amount calculation unit 13, and an information amount determination unit 14. The time information detection unit 10, the reproduction control unit 11, the accumulation amount detection unit 12, the information amount calculation unit 13, and the information amount determination unit 14 are the time information detection unit, the reproduction control unit, the accumulation amount detection unit, and the information amount calculation according to the present invention. Means for determining the amount of information.

  The time information detection unit 10 detects time information from time stamps of video packets and audio packets.

  The reproduction control unit 11 instructs the video packet decoder 711 and the audio packet decoder 710 to perform decoding and reproduction based on the time information detected by the time information detection unit 10.

  The accumulation amount detection unit 12 detects the amount of video packets accumulated in the video jitter buffer 715. Since video packets are sequentially stored in the video jitter buffer 715, the storage amount detection unit 12 calculates the amount of video packets stored in the video jitter buffer 715 from the address of the first video packet stored in the video jitter buffer 715. It can be detected.

  The information amount calculation unit 13 calculates the information amount of the voice packet, and the information amount determination unit 14 determines whether or not the calculated information amount is a predetermined value or less. The amount of information is, for example, the level of an audio signal, and the predetermined value is a level at which it can be determined that there is no sound. Thus, when the level of the audio signal is low, it is considered that the amount of information is low and important information is not included.

  In this embodiment, as an example, the data transmission device 701 packetizes a still image of each frame of a 15 fps moving image and transmits it as a video packet, and packetizes 20 msec of audio into audio packets. The following processing will be described on the assumption that the time stamp assigning unit 707 assigns the time information obtained by the built-in clock as a time stamp to the video packet and the audio packet, and transmits the time packet from the transmitting unit 708 to the data receiving device 702.

  A specific example of the processing of the CPU 20 of this embodiment will be described.

  The playback control unit 11 does not start playback until the received voice packet is stored in the voice jitter buffer 716 by a predetermined amount. For example, the playback control unit 11 starts playback when the amount of voice packets accumulated in the voice jitter buffer 716 detected by the accumulation amount detection unit 12 reaches 50% of the storage capacity of the voice jitter buffer 716.

  The reproduction control of the reproduction control unit 11 is based on the time information described in the time stamp of the voice packet detected by the time information detection unit 10. The playback control unit 11 searches the video jitter buffer 715 for a video packet to be played back during playback of the audio from the time information of the audio packet to be played, sends it to the video packet decoder unit 711, and plays back the video.

  For example, if the time stamp of the audio packet A to be reproduced is T1, the video packet to be reproduced during the reproduction of the audio is a video packet having a time stamp in the range of T1 + 20 msec. For example, if the main microcomputer unit 713 searches the video jitter buffer 715 for the video packet A having a time stamp of T1 + 10 msec, the playback control unit 11 starts the playback of the audio packet A, and the video packet decoder unit 10 seconds later. In step S711, control is performed so that the video of the video packet A is reproduced. In this way, the playback control unit 11 performs the audio and video synchronization processing.

  The audio jitter buffer 716 and the video jitter buffer 715 may underflow or overflow due to the influence of the propagation delay of the network and the data processing time of the data receiving device 702. This phenomenon will be described with reference to FIG.

  FIG. 2 is an explanatory diagram for schematically explaining underflow and overflow. In FIG. 2, 715 is a diagram schematically illustrating the state of the video packet stored in the video jitter buffer 715, and 716 is a diagram schematically illustrating the state of the audio packet stored in the audio jitter buffer 716. The outer shapes of 715 and 716 in FIG. 2 represent the maximum memory capacities of the audio jitter buffer 716 and the video jitter buffer 715, respectively. The horizontal axis is a memory address, and audio packets and video packets are stored in FIFO (First In First Out) in order from the leftmost memory address in FIG. The hatched portion represents a portion where audio packets and video packets are accumulated in the audio jitter buffer 716 and the video jitter buffer 715. Therefore, the voice packet and video packet stored in the arrow direction on the left side of FIG. 2 are “newer” packets, and the voice packet and video packet stored in the right arrow direction are “older” packets.

  FIG. 2A is an explanatory diagram for explaining an overflow state.

  FIG. 2A shows a state in which more video packets are stored in the video jitter buffer 715 than audio packets stored in the audio jitter buffer 716. An arrow A1 indicates the first voice packet to be reproduced next. On the other hand, the video packet corresponding to the time information of the audio packet indicated by the arrow A1 is the video packet indicated by the arrow V1, and the old video packet remains without being reproduced in the video jitter buffer 715 on the right side of the arrow V1.

  Such a phenomenon is, for example, when the processing time of the video packet decoder unit 71 is longer than the time interval described in the time stamp. In this case, the processing cannot catch up and video packets are steadily stored in the video jitter buffer 715. As it accumulates, overflow occurs.

  FIG. 2B shows a state where more audio packets are stored in the audio jitter buffer 716 than in the video packets stored in the video jitter buffer 715. An arrow A2 indicates the first audio packet to be reproduced next. On the other hand, the video packet corresponding to the time information of the audio packet indicated by the arrow A2 is the first video packet indicated by the arrow V2, and indicates that the video packet stored in the video jitter buffer 715 is small.

  Such a phenomenon is, for example, the case where a video packet that has arrived without delay initially has a delay in reaching the packet because the network is congested. In this case, the video packets stored in the video jitter buffer 715 are depleted and there are no video packets to be reproduced.

  Such underflow and overflow are phenomena that cannot be avoided because data is transmitted and received via the IP network. It also occurs due to individual differences in the processing time of the encoder and decoder.

  Next, processing when there is a possibility of underflow or overflow in the present embodiment will be described.

  FIG. 3 is a flowchart process showing the flow of reproduction control by the main microcomputer unit 713.

  In the flowchart of FIG. 3, processing after the number of received voice packets reaches 50% of the storage capacity of the voice jitter buffer 716 and playback is started will be described.

  S101: Decoding a voice packet having the oldest time information.

  The playback control unit 11 instructs the time information detection unit 10 to search for a voice packet having the oldest time information stored in the voice jitter buffer 716, and sends the searched voice packet to the voice packet decoder unit 710. The packet decoder unit 710 decodes the audio packet into audio data.

  S500: This is a step of calculating the information amount of the decoded voice packet.

  The information amount calculation unit 13 calls a subroutine for calculating the information amount of the decoded voice packet, and obtains the value of the information amount of the voice packet. For example, the amount of information in a voice packet is large when speech is actively performed during a conference, and the amount of information is small when speech is interrupted and the sound is almost silent, or only background sounds such as air conditioning sounds are heard. An algorithm for calculating the amount of information from the audio data will be described in detail later.

  S102: This is a step of starting audio reproduction.

  The reproduction control unit 11 instructs the audio packet decoder unit 710 to reproduce the decoded audio data and output it from the audio output unit 709.

  S103: This is a step of searching for a video packet synchronized with the audio being reproduced.

  For example, if the reproduction time of the audio packet A is 20 msec and the time stamp is T1, the video packet to be reproduced during the reproduction of the audio is a video packet having a time stamp in the range of T1 + 20 msec. The playback control unit 11 instructs the time information detection unit 10 to search the video jitter buffer 715 for a video packet having a time stamp in this range.

  When there is no video packet to be synchronized (step S103; No), the process proceeds to step S122.

  If there is no video packet to be synchronized, the video jitter buffer 715 is in an underflow state, and the process proceeds to step S122.

  If there is a video packet to be synchronized (step S103; Yes), the process proceeds to step S104.

  S104: A step of determining whether or not there is a possibility that the video packet may overflow in the video jitter buffer 715.

  The accumulation amount detector 12 checks the start address of the video packet accumulated in the video jitter buffer 715 and detects the accumulation amount. The reproduction control unit 11 determines whether there is a possibility of overflow in the video jitter buffer 715 by comparing with a predetermined value. For example, when the predetermined value is 70% and the video packet stored in the video jitter buffer 715 is 70% or more, it is determined that there is a possibility of overflow.

  If there is no risk of overflow (step S104; No), the process proceeds to step S105.

  If there is a possibility of overflow (step S104; Yes), the process proceeds to step S108.

  S108: This is a step of reproducing the video packet.

  The reproduction control unit 11 instructs the video packet decoder unit 711 to decode and reproduce the video packet searched in step S103 at a predetermined timing.

  S109: This is a step of discarding old video packets.

  The playback control unit 11 discards video packets older than the currently playing audio packet among the video packets stored in the video jitter buffer 715. When there is a possibility of overflow, old video packets that are not played back are discarded because old video packets are accumulated more and more late.

  S110: A step of determining whether or not the amount of audio information is equal to or less than a predetermined value.

  The information amount determination unit 14 determines whether or not the value of the audio information amount calculated in step S500 is equal to or less than a predetermined value. The predetermined value is a threshold value for determining, for example, a silent state or a state in which only background sound is heard, and is set in advance according to the use environment.

  If it is greater than or equal to the predetermined value (step S110; No), the process proceeds to step S112.

  The audio reproduction is continued, and the process proceeds to step S112.

  If the value is equal to or smaller than the predetermined value (step S110; Yes), the process proceeds to step S111.

  S111: This is a step of stopping audio reproduction.

  The reproduction control unit 11 instructs the audio packet decoder unit 710 to stop audio reproduction, and proceeds to the next step S112. In this case, since the arrival of the voice packet is delayed, the voice delay is large, for example, a sense of incongruity when talking is large. For this reason, the reproduction of a voice packet with a small amount of information, for example, a silent state or a background sound that is only heard, is stopped to reduce the voice delay. In this way, there is little sense of incongruity for the person who is listening, and the voice delay time can be shortened without missing important information.

  S105: This is a step of determining whether or not the video packet is likely to underflow in the video jitter buffer 715.

  The accumulation amount detection unit 12 checks the start address of the video packet accumulated in the video jitter buffer 715 and detects the accumulation amount of the video packet accumulated in the video jitter buffer 715. The reproduction control unit 11 determines whether or not there is a possibility of underflow by comparing with a predetermined value. For example, if the predetermined value is 30% and the video packet stored in the video jitter buffer 715 is 30% or less, the reproduction control unit 11 determines that there is a possibility of overflow.

  When there is no possibility of underflow (step S105; No), the process proceeds to step S130.

  If there is a possibility of underflow (step S105; Yes), the process proceeds to step S121.

  S130: This is a step of reproducing a video packet.

  The reproduction control unit 11 instructs the video packet decoder unit 711 to decode and reproduce the video packet searched in step S103 at a predetermined timing. Next, the process proceeds to step S112.

  S121: This is a step of reproducing a video packet.

  The reproduction control unit 11 instructs the video packet decoder unit 711 to decode and reproduce the video packet searched in step S103 at a predetermined timing.

  S122: This is a step of determining whether or not the amount of audio information is a predetermined value or less.

  The reproduction control unit 11 determines whether or not the information amount value of the audio data calculated in step S500 is equal to or less than a predetermined value. The predetermined value is a threshold value for determining, for example, a silent state or a state in which only background sound is heard, and is set in advance according to the use environment.

  If it is equal to or greater than the predetermined value (step S122; No), the process proceeds to step S112.

  The audio reproduction is continued, and the process proceeds to step S112.

  If it is equal to or smaller than the predetermined value (step S122; Yes), the process proceeds to step S123.

  S123: This is a step of repeatedly reproducing the sound.

  The reproduction control unit 11 repeatedly reproduces the audio data determined as the audio data having the information amount equal to or smaller than the predetermined amount in step S122 until the audio packet decoder 710 stores the predetermined amount of the video packet in the video jitter buffer 715. To In this case, since the arrival of the video packet is delayed from the arrival of the audio packet, an audio packet having a small amount of information, for example, a sound packet in which only a silent state or background sound is heard is repeatedly reproduced, and the video packet is stored in the video jitter buffer 715. Wait until a predetermined amount is accumulated. When the accumulation amount detection unit 12 detects that a predetermined amount of video packets have been accumulated in the video jitter buffer 715, the main microcomputer unit 713 proceeds to step 112.

  S112: A step of determining whether or not there is a next voice packet.

  The reproduction control unit 11 searches the audio jitter buffer 716 and searches for the next audio packet to be reproduced.

  If there is a voice packet (step S112; Yes), the process returns to step S101.

  Returning to step S101, the next voice packet is reproduced.

  When there is no voice packet (step S112; No), the process ends.

  In this way, the delay time from the real time of the audio is adjusted by stopping or repeating the reproduction of the audio packet with a small amount of information, for example, the silent state or the background sound only heard, and the video jitter buffer 715 Since the video packets to be synchronized with the audio are secured with a certain amount of the stored video packets, important audio information is not lost, and the caller may feel uncomfortable with the reproduced audio Fewer packet communication systems can be provided.

  Next, the routine for calculating the amount of audio information described in step S500 will be described with reference to FIGS.

  FIG. 4 shows a first embodiment of the audio information amount calculation routine of the present invention.

  S501: The average signal level of the audio signal is calculated.

  The information amount calculation unit 13 calculates the average signal level of the audio data decoded by the audio packet decoder 710, and uses this value as the audio information amount.

  For example, it is assumed that the audio data is 12 bits and the complete silence state is a digital value of 0. If the average value calculated in step S500 is a digital value 80, the digital value 80 is the value of the amount of audio information. For example, if the predetermined value determined in step S110 is 100, if the value of the audio information amount is 80, it is determined that there is no sound.

  FIG. 5 shows a second embodiment of the sound information amount calculation routine of the present invention.

  S601: Frequency conversion of an audio signal is performed.

  The information amount calculation unit 13 performs frequency conversion on the audio data decoded by the audio packet decoder 710, and obtains a signal level value for each frequency.

  S602: Addition averaging with the previous value.

  The main microcomputer unit 713 weights and adds the signal level value for each frequency stored in the built-in memory until the previous time and the current value to obtain an average value for each frequency.

  When the addition average of the frequency distribution of each time is calculated in this way, the average value of the frequency distribution of the background sound such as the air-conditioning sound that always generates the same sound is calculated.

  S603: The difference between the addition average value and the current value is calculated.

  The information amount calculation unit 13 calculates the difference between the addition average value for each frequency obtained in step S602 and the current value.

  S604: The sum of the differences for each frequency is calculated.

  The information amount calculation unit 13 calculates the sum of differences for each frequency obtained in step S603.

  The sum total obtained in step S604 is used as the information amount, and the process returns to the original routine.

  For example, when there is a silent state with a low amount of information, the value calculated in this way is 0. On the other hand, when multiple people speak in a meeting or the like, the frequency distribution is often different each time. The value obtained is large.

  As described above, according to the present invention, it is possible to provide a packet communication system and a data receiving device capable of reproducing video in synchronization with audio while maintaining audio quality even when the audio packet is delayed from the video packet.

It is a block diagram which shows one Embodiment of the structure of the packet communication system which concerns on this invention. It is explanatory drawing which illustrates underflow and overflow typically. It is a flowchart process which shows the flow of reproduction | regeneration control by the main microcomputer part 713. FIG. It is 1st Embodiment of the audio | voice information amount calculation routine of this invention. It is 2nd Embodiment of the audio | voice information amount calculation routine of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Time information detection part 11 Reproduction | regeneration control part 12 Accumulation amount detection part 13 Information amount calculation part 14 Information amount determination part 20 CPU
21 RAM
22 ROM
701 Data transmission device 713 Main microcomputer unit 702 Data reception device 717 Transmission line

Claims (12)

A video packet encoder that packetizes video signals into video packets;
A voice packet encoder that packetizes voice signals into voice packets;
A time stamp giving unit for giving time information to video packets and audio packets;
A data transmission device comprising: a transmission unit that sequentially transmits the video packet and the audio packet to which the time information is attached;
A receiving unit for receiving the video packet and the audio packet transmitted via a transmission line;
A video jitter buffer for sequentially storing the video packets received by the receiver;
A voice jitter buffer for sequentially storing the voice packets received by the receiver;
A video packet decoder for decoding and reproducing the video packet;
An audio packet decoder for decoding and reproducing the audio packet;
Time information detecting means for detecting the time information;
Based on the time information detected by the time information detection means, a data receiving device comprising the video packet decoder, and a reproduction control means for instructing the audio packet decoder to reproduce,
In a packet communication system having
The data receiving device is:
Accumulated amount detecting means for detecting the amount of image packets accumulated in the image jitter buffer;
An information amount calculating means for calculating an information amount of an audio packet to be reproduced by the audio packet decoder;
An information amount determination means for determining the information amount calculated by the information amount calculation means;
The reproduction control means includes
A packet communication system characterized in that, according to the amount of video packets detected by the accumulated amount detection means, the reproduction time of the audio packet determined by the information amount determination means to be an information amount equal to or less than a predetermined value is changed. . The reproduction control means includes
When the amount of video packets detected by the accumulated amount detection means is a predetermined value or more,
2. The packet according to claim 1, wherein voice packets whose information amount is equal to or greater than a predetermined value are sequentially reproduced without reproducing the voice packets determined as having an information amount equal to or smaller than a predetermined value. Communications system. The reproduction control means includes
3. The packet communication system according to claim 2, wherein the video packet stored in the video jitter buffer having time information older than the audio packet to be reproduced is discarded. The reproduction control means includes
When the amount of video packets detected by the accumulated amount detection means is below a predetermined value,
2. The audio packet determined by the information amount determination means to be an information amount equal to or less than a predetermined value is repeated according to the amount of the video packet, and then the next audio packet is sequentially reproduced. The packet communication system described in 1. 5. The packet communication system according to claim 1, wherein the information amount calculation unit calculates the information amount based on an average value of the audio signal. The information amount calculation means calculates each frequency component of the audio signal, and calculates the difference between the addition average value of each frequency component value of the audio packet calculated up to the previous time and the value of each frequency component calculated this time. The packet communication system according to any one of claims 1 to 4, wherein the information amount is calculated based on the information. A receiver for receiving video packets and audio packets;
A video jitter buffer for sequentially storing the video packets received by the receiver;
A voice jitter buffer for sequentially storing the voice packets received by the receiver;
A video packet decoder for decoding and reproducing the video packet;
An audio packet decoder for decoding and reproducing the audio packet;
Time information detecting means for detecting time information attached to the video packet and the audio packet;
In a data receiving device comprising: the video packet decoder based on the time information detected by the time information detection means; and a reproduction control means for instructing the audio packet decoder to reproduce,
The data receiving device is:
Accumulated amount detecting means for detecting the amount of image packets accumulated in the image jitter buffer;
An information amount calculating means for calculating an information amount of an audio packet to be reproduced by the audio packet decoder;
An information amount determination means for determining the information amount calculated by the information amount calculation means;
The reproduction control means includes
A data receiving device characterized in that, according to the amount of video packets detected by the accumulated amount detecting means, the reproduction time of the audio packet determined by the information amount determining means to be an information amount equal to or less than a predetermined value is changed. . The reproduction control means includes
When the amount of video packets detected by the accumulated amount detection means is a predetermined value or more,
8. The data according to claim 7, wherein the voice packet whose information amount is equal to or greater than the predetermined value is sequentially reproduced without reproducing the voice packet determined as having the information amount equal to or smaller than the predetermined value. Receiver equipment. The reproduction control means includes
9. The data receiving apparatus according to claim 8, wherein the video packet stored in the video jitter buffer having time information older than the audio packet to be reproduced is discarded. The reproduction control means includes
When the amount of video packets detected by the accumulated amount detection means is below a predetermined value,
8. The reproduction of audio packets determined by the information amount determination means to be an information amount of a predetermined value or less is repeated according to the amount of video packets, and then the next audio packets are sequentially reproduced. The data receiving device described in 1. 11. The data receiving device according to claim 7, wherein the information amount calculation unit calculates the information amount based on an average value of an audio signal. The information amount calculation means calculates each frequency component of the audio signal, and calculates the difference between the addition average value of each frequency component value of the audio packet calculated up to the previous time and the value of each frequency component calculated this time. The data receiving device according to claim 7, wherein the information amount is calculated based on the information amount.

Images