JP3977784B2 - Real-time packet processing apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a real-time packet processor and its method capable of appropriately interpolating a frame while minimizing mutual disagreement of pieces of correlation data of an encoding processing part and a decoding processing part and reproducing sound while reducing quality deterioration when a sound packet is lost and delayed according to an arrival state of the packet. <P>SOLUTION: A receiver performs only decoding at the decoding processing to N (N is a natural number) pieces of frames from the frames which become discontinuous when the frame to be processed is not successive, performs the sound reproduction processing from a frame next to the N frames without performing sound reproduction processing to the N pieces of decoded frames, makes the correlation data to be used at the decoding processing part in the case of performing decoding coincident with the correlation data in the encoding processing part on the transmitter side and performs appropriate sound reproduction processing from a frame next to the N pieces of frames. <P>COPYRIGHT: (C)2005,JPO&amp;NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to a real-time packet processing apparatus and method for processing frame data contained in a received packet in real time.
[0002]
[Prior art]
  2. Description of the Related Art Conventionally, along with the digitization of electronic devices, information to be transferred is generally packetized and transferred in information communication. For example, in the case of transferring an audio signal, on the transmitting side, audio data sampled at a predetermined sampling frequency is distributed and stored in separate packets by a predetermined amount and transferred in units of packets. On the receiving side, the audio data is extracted from the received packet, and the extracted audio data is connected and played back or mixed.
[0003]
  That is, in an electronic device that performs packet communication as described above, the transmission side performs processing to form and transmit a packet when one packet of data is obtained, and the reception side stores the packet in the received packet. A process of reading data in the packet is performed every time required for reproducing the data. Thereby, on the receiving side, for example, in the case of real-time transfer of audio data, it is possible to perform reproduction processing or mixing processing on continuous audio from a plurality of dividedly received packets.
[0004]
  Such packet communication is almost always performed using a computer device. For example, a mobile phone using wireless communication, a well-known IP phone using a communication network such as the Internet, music from a distribution server, etc. It is used in a system for distributing content to a user terminal device, a remote conference system, and the like.
[0005]
  For example, a speech coding method (predictive coding method) using inter-frame prediction is ITU standard G.264. 729, G.G. 723.1, G.M. 722.1.
[0006]
  In these encoding schemes, the correlation data stored in the internal buffer of the encoding processing unit on the transmitting device side and the correlation data stored in the internal buffer of the decoding processing unit on the receiving device side must match. There is a restriction that a correct acoustic signal cannot be restored. The correlation data is the G. 729, G.G. 723.1, G.M. This is data used in the predictive coding method described in 722.1.
[0007]
  For example, when audio frames 0, 1, 2, and 3 are encoded by the decoding processing unit on the transmission device side and each frame is included in packets 0, 1, 2, and 3 and transmitted, the packet reception order is changed. Regardless, the decoding processing unit on the receiving apparatus side encodes each frame unless the decoding processing is performed in the order of frames 0, 1, 2, and 3 in the same manner as the encoding processing unit encoded. The correlation data of the encoding processing unit at the time point and the correlation data of the decoding processing unit at the time of decoding the frame do not match, and a correct decoded waveform cannot be obtained.
[0008]
  In addition, when a packet is lost during transfer, a packet loss compensation (PLC) process may be performed in the receiving apparatus.
[0009]
  As packet loss compensation processing, G.I. 711 Appendix 1 and G. A method that is based on the 729 standard is known.
[0010]
[Patent Document 1]
  JP 2000-83050 A
[Non-Patent Document 1]
  ITU-T Recommendation G. 729
[Non-Patent Document 2]
  ITU-T Recommendation G. 723.1
[Non-Patent Document 3]
  ITU-T Recommendation G. 722.1
[Non-Patent Document 4]
  ITU-T Recommendation G. 711 Appendix 1
[0011]
[Problems to be solved by the invention]
  When the packet loss occurs due to the restriction in the speech coding method (predictive coding method) described above, the correlation data of the decoding processing unit may be updated using the lost frame as an input to the decoding processing unit. Therefore, when decoding a frame in a packet correctly received after erasure, the correlation data of the encoding processing unit on the transmitting device side and the correlation data of the decoding processing unit on the receiving device side are inconsistent, and the audio frame is It may not be restored correctly.
[0012]
  In the above-described conventional method, since no special measures are taken for correlation data mismatch caused by packet loss or delay as described above, degradation in perceivable quality occurs in reproduced speech.
[0013]
  In addition, in the case of an application such as VoIP, there is no guarantee that all voice packets transmitted from the transmitting device side will correctly arrive at the receiving device side, even though there are restrictions when using the predictive coding method as described above. Absent. For example, when a packet is lost in a communication network or the like, the correlation data of the encoding processing unit and the decoding processing unit easily mismatch, and when the communication session is established, it is correctly received from the first packet transmitted by the transmitting device. Since it does not always arrive at the apparatus side, there is a problem that communication continues from the initial stage of communication with the correlation data of the encoding processing unit and the decoding processing unit being inconsistent.
[0014]
  In view of the above problems, the object of the present invention is to minimize the mismatch of correlation data between the encoding processing unit and the decoding processing unit when the packet is lost or delayed according to the arrival state of the voice packet. It is an object to provide a real-time packet processing apparatus and method capable of appropriately interpolating frames and reproducing sound with reduced quality deterioration.
[0015]
[Means for Solving the Problems]
  In general, the prediction used in the encoding processing unit and the decoding processing unit uses a correlation between several frames, and several frames from the point in time when the internal states of the encoding processing unit and the decoding processing unit do not match. During this time, a correct acoustic signal cannot be restored, and perceived quality degradation occurs in the reproduced sound.
[0016]
  However, even if there is a mismatch in the correlation data between the encoding processing unit and the decoding processing unit, the correlation data gradually matches after decoding the received frames in several frames in the correct order, and as a result the decoded waveform The quality degradation gradually decreases as the correct frames are successively decoded.
[0017]
  In the present invention, when there is a mismatch in the correlation data of the encoding processing unit and the decoding processing unit, the audio whose quality is deteriorated by not reproducing the audio of the first few frames is not reproduced and the listening quality is improved. A real-time packet processing apparatus and a method thereof are proposed.
[0018]
  In the present invention, a continuous input audio signal is cut out at predetermined intervals, and a plurality of sampling data obtained by sampling the cut signals at predetermined sampling times shorter than the cycle is encoded by an encoding processing unit. A packet including the frame is generated for each frame, and the packet is received by a receiver via a communication network from a transmitter that sequentially transmits the frame, and the received packet is received by the receiver In the real-time packet processing using a predictive encoding method in which the frame included is decoded by a decoding processing unit, and audio reproduction processing is performed on sampling data included in the decoded frame, the receiving device includes:A packet received from the transmission device is stored in a buffer and included in a packet input from the buffer.Analyzing the encoded frame when decoding the frame, holding the analysis result as correlation data, and when performing the decoding process by the decoding processing unit, when the frame to be processed is not continuous, Only the decoding processing unit performs decoding on N frames from the discontinuous frames, and the N frames without performing the audio reproduction processing on the decoded N frames. The audio reproduction processing is performed by decoding the stored correlation data from the next frame.Further, the receiving device is configured such that when the number of packets stored in the buffer exceeds a predetermined number, the last N packets among a predetermined number of consecutive packets to be discarded stored in the buffer. A packet other than that frame is discarded, the last N frames are only decoded, and the subsequent M packet frames are subjected to a fade-in process for gradually increasing the volume from the silent state, A fade-out process for gradually reducing the sound volume to silence is performed on the frame of M packets before the discarded packet.
[0019]
  According to the present invention, when the frame to be processed is not continuous when performing the decoding process by the decoding processing unit, the decoding processing unit is applied to N frames from discontinuous frames. Only the decoding is performed, and without performing the sound reproduction process on the decoded N frames, the decoding is performed using the held correlation data from the next frame of the N frames. And the audio reproduction process is performed. Since only the decoding process is performed on the N frames, the correlation data used in the decoding processing unit when decoding is completely matched with the correlation data in the encoding processing unit on the transmission device side or It is possible to substantially match, and appropriate audio reproduction processing can be performed from the next frame of the N frames..
[0020]
MaFurther, in the present invention, the receiving apparatus performs the processing on the N frames from the discontinuous frames when the frames to be processed are not continuous when performing the decoding process by the decoding processing unit. Decoding in the decoding processing unit is performed, and the audio reproduction processing with the volume reduced is performed on the decoded N frames.
[0021]
  According to the present invention, when the frame to be processed is not continuous when performing the decoding process by the decoding processing unit, the decoding processing unit is applied to N frames from discontinuous frames. After the decoding in step S3 is performed, the sound reproduction process with the volume reduced is performed on the decoded N frames.
[0022]
  At this time, similarly to the above, the decoding processing is performed on the N frames, so that the correlation data used in the decoding processing unit when decoding is performed in the encoding processing unit on the transmission device side. The correlation data can be completely matched or almost matched. Further, since the sound reproduction processing with a reduced volume is performed on the N frames, no abnormal sound is generated at this transition portion, and deterioration of sound quality is reduced..
[0023]
MaIn the present invention, the reception device stores the packet received from the transmission device in the buffer, and decodes the frame included in the packet input from the buffer, and stores the packet stored in the buffer. When the number of frames exceeds a predetermined number, packets other than the last N frames of a predetermined number of consecutive packets to be discarded stored in the buffer are discarded, and the last N Only the frame is decoded, and a fade-in process for gradually increasing the volume from the silent state is performed on the subsequent frame of M packets, and the frame of the M packets before the discarded packet is subjected to fade-in processing. Apply fade-out processing to gradually reduce the volume to silence.
[0024]
  According to the present invention, when the number of packets accumulated in the buffer of the receiving apparatus exceeds a predetermined number, a delay occurs in the audio reproduction processing, so that a predetermined number of consecutive discards stored in the buffer are targeted for discarding. Packets other than the last N frames of the packets are discarded.
[0025]
  Thus, since the fade-in process for gradually increasing the volume from the silent state is performed on the M frames, the voice is reproduced at the portion where the transition is made from the silent state to the voiced state when the voice is reproduced in the receiving device. Since the waveform does not become discontinuous, abnormal noise does not occur in this transition portion, and deterioration of voice quality is reduced.
[0026]
  Furthermore, since the fade-out process for gradually reducing the sound volume to silence is performed on the M frames before becoming discontinuous, the sound level of the part from the sounded state to the silent state is gradually reduced. When audio is played back by the receiver, the audio waveform does not become discontinuous at the transition from the voiced state to the silent state, so no abnormal noise is generated at this transition and the degradation of voice quality is reduced. Is done.
[0027]
  Furthermore, since only the last N frames of the frame of the packet to be discarded are decoded, predictive coding is performed in the encoding processing unit on the transmission side and the decoding processing unit on the reception side. Correlation data used in the method can be completely matched or almost matched.
[0028]
  In the present invention, the receiving device determines that the frame is discontinuous when the sequence number becomes discontinuous based on the sequence number included in the packet.
[0029]
  According to the present invention, based on the sequence number included in the packet, the receiving device determines that the frame is discontinuous when the sequence number becomes discontinuous.
[0030]
  In the present invention, the receiving apparatus reproduces audio by superimposing the frame subjected to the fade-out process and the frame subjected to the fade-in process.
[0031]
  According to the present invention, since the frame subjected to the fade-out process and the frame subjected to the fade-in process are overlapped and reproduced by sound, a silent state does not occur in a discontinuous portion, and the sound quality is improved. Degradation is further reduced.
[0032]
DETAILED DESCRIPTION OF THE INVENTION
  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0033]
  (First embodiment)
  FIG. 1 is a block diagram showing a functional configuration of a real-time packet processing apparatus according to the first embodiment of the present invention. FIG. 2 is a diagram for explaining packetization of a voice signal by a voice packet transmitting apparatus according to the first embodiment of the present invention. 3 is a diagram for explaining a real-time transfer protocol (hereinafter referred to as RTP) header used in the first embodiment of the present invention. In the figure, 1 is a voice packet transmitting device (hereinafter simply referred to as a transmitting device), 2 is a voice packet receiving device (hereinafter simply referred to as a receiving device), and 3 is a communication network such as the Internet. In the present embodiment, as an example, a description will be given of an apparatus that transfers voice packets from the transmission apparatus 1 to the reception apparatus 2 in real time using UDP / IP via the communication network 3.
[0034]
  The transmission device 1 is composed of a well-known computer device and operates according to a preset program. The transmission device 1 includes an audio input unit 11, an analog / digital (A / D) conversion unit 12, an encoding processing unit 13, and a packet generation unit. 14 and a transmission unit 15. Each part which comprises these transmitters 1 is comprised by both hardware and software.
[0035]
  The receiving device 2 is composed of a known computer device, and operates according to a preset program. The receiving device 21, a packet analyzing unit 22, a decoding processing unit 23, and a digital / analog (D / A) converting unit 24. And an audio output unit 25. Each part which comprises these receivers 2 is comprised by both hardware and software.
[0036]
  The audio input unit 11 converts the audio signal into an analog electric signal 4 as shown in FIG. 2 and outputs it to the A / D conversion unit 12, and the A / D conversion unit 12 converts it into a digital signal at a predetermined sampling time. Audio data (samples) are sequentially stored in a data buffer (not shown) provided in the encoding processing unit 13.
[0037]
  Also, as shown in FIG. 2, the audio data stored in the data buffer of the encoding processing unit 13 is cut by the encoding processing unit 13 every predetermined period T to be an audio data frame 31. Packets 30 are generated and transmitted sequentially.
[0038]
  The encoding processing unit 13 performs encoding processing of the audio data frame to be encoded input from the A / D conversion unit 12, but the internal state of the result of encoding the previous frame when performing the encoding processing is Coding gain is improved by holding in the internal buffer 13a and performing prediction from the past.
[0039]
  In this embodiment, in order to reduce quality degradation due to mismatch of correlation data in the encoding processing unit 13 of the transmitting device 1 and the decoding processing unit 23 of the receiving device 2 due to packet loss, the silent state is changed to the voiced state. In this case, the internal buffer 13a of the encoding processing unit 13 is reset and the initial value is used to reduce the occurrence of quality degradation due to transmission errors.
[0040]
  Further, the encoding processing unit 13 encodes the audio data frame to be encoded based on the analysis result and sends it to the packet generation unit 14.
[0041]
  The packet generator 14 generates an RTP packet including the encoded voice data input from the encoding processor 13 and sends it to the transmitter 15. An RTP header as shown in FIG. 3 is added to the RTP packet at this time.
[0042]
  As is well known, the RTP header includes a 2-bit version information V, a 1-bit padding information P, a 1-bit extension information X, a 3-bit CSRC-Count information CC, a 1-bit Marker information (hereinafter referred to as a marker). M, 7-bit Payload-Type information PT, 16-bit sequence number (sequence number: Sequence Number), 32-bit time stamp (Timestamp), 32-bit synchronization signal source (SSRC) identifier, 32 bits , A contributing transmission source (CSRC) identifier or the like.
[0043]
  In the present embodiment, the marker bit M of the first packet to be transmitted after entering the voiced state after stopping the packet transmission in the silent state is set to “1”, and the marker bit M of the other packet is set. Is set to “0”.
[0044]
  The transmission unit 15 transmits the RTP packet input from the packet generation unit 14 to the reception device 2 via the communication network 3.
[0045]
  On the other hand, the reception unit 21 of the reception device 2 receives the RTP packet transmitted from the transmission device 1 via the communication network 3 and sends it to the packet analysis unit 22.
[0046]
  The packet analysis unit 22 analyzes the RTP packet input from the reception unit 21 and separates it into a header portion and an encoded audio data frame, analyzes the contents of the header portion, and transmits the result based on the RTP time stamp. The audio data frames encoded in the specified order are output to the decoding processing unit 23. Further, the packet analysis unit 22 notifies the decoding processing unit 23 of the value of the marker bit M in the RTP header.
[0047]
  The decoding processing unit 23 decodes the encoded audio data frame input from the packet analysis unit 22 and converts it into digital audio data, and outputs the digital audio data to the D / A conversion unit 23.
[0048]
  Further, the decoding processing unit 23 analyzes the encoded audio data frame when performing decoding, temporarily stores the analysis result in the internal buffer 23a, and stores it in the internal buffer 23a when performing data analysis. Data analysis is performed with reference to the temporarily stored analysis result or the preset initial analysis value. Here, by using the analysis result of the previous frame temporarily stored in the internal buffer 23, the optimum analysis and decoding can be performed in consideration of the correlation between the previous and subsequent frames.
[0049]
  The D / A converter 23 receives the digital audio data obtained by decoding by the decoding processor 23, converts it into an analog audio signal, and outputs it to the audio output unit 24.
[0050]
  The audio output unit 24 converts the analog audio data input from the D / A conversion unit 23 into audio and outputs it.
[0051]
  Next, the operation of the real-time packet processing apparatus according to the present embodiment having the above configuration will be described.
[0052]
  In VoIP communication, the sending device 1 side may start sending voice packets before the receiving device 2 side is ready to accept. In such a case, the receiving apparatus 2 cannot correctly receive the packet immediately after the start of communication, and misses the first few packets.
[0053]
  In this case, the correlation data stored in the internal buffer 13a in the encoding processing unit 13 on the transmitting device 1 side and the correlation data stored in the internal buffer 23a in the decoding processing unit 23 on the receiving device 2 side are It becomes inconsistent and a correct audio signal cannot be generated.
[0054]
  In the present embodiment, for example, G. A case where 729 is used will be described as an example. In this case, since one frame is 10 ms, a case where one frame of 10 ms worth of audio is assumed to be one packet will be described. In the following embodiments, the same condition is described as an example.
[0055]
  When performing VoIP voice packet communication using RTP / RTCP, it is impossible to know the sequence number of the packet sent first by the transmission device 1, so that the transmission device 1 receives the first packet received on the reception device 2 side. I don't know if this is the first packet sent out.
[0056]
  For this reason, it cannot be known whether or not the first encoded frame generated in a state where the correlation data of the internal buffer in the encoding processing unit 13 of the transmission device 1 is reset is included.
[0057]
  The receiving device 2 decodes the voice frame included in the received packet as it is without performing any additional processing even though the first several packets sent from the transmitting device 1 could not be received. When reproduced in this manner, there may be a case where the quality of the reproduced speech is deteriorated due to the mismatch of the correlation data between the encoding processing unit and the decoding processing unit at the head portion.
[0058]
  In order to avoid this problem, in the first embodiment, the decoding process is performed for several packets from the start of communication, but the decoded frame is not played back, and several frames until the signal waveform of the frame is stabilized. After the decryption process is performed, reproduction is performed using a fade-in process.
[0059]
  For example, in the example shown in FIG. 4, the transmitting apparatus 1 transmits in order from the packet including the frame having the sequence number 0, but the receiving apparatus 2 receives the packet from the packet having the sequence number 3 after starting reception. is doing. In this case, the receiving apparatus 2 only performs a decoding process on the N frames included in the first N packets that can be received. Thus, correlation data for normal decoding is stored in the internal buffer 23a of the decoding processing unit 23. Here, N = 2 and only the decoding process is performed for the two packets with the sequence numbers 3 and 4, and the correlation data is stored in the internal buffer 23a without performing the audio reproduction.
[0060]
  Further, the receiving apparatus 2 performs a fade-in process at the time of audio reproduction after performing a decoding process on a frame of M packets following the frames of the sequence numbers 3 and 4. Here, fade-in processing is applied to the frames of packets with sequence numbers 5 and 6 with M = 2. As for the frames of packets whose sequence numbers are No. 7 and later, normal decoding processing and audio reproduction processing are performed. In the following description, N and M may be numbers that are 0 or more and N + M is 1 or more.
[0061]
  As described above, according to the present embodiment, when the leading voice packet cannot be received, the voice can be reproduced with reduced quality deterioration.
[0062]
  (Second Embodiment)
  Next, a second embodiment of the present invention will be described.
[0063]
  In the second embodiment, the state mismatch of the correlation data stored in the internal buffer 13a in the encoding processing unit 13 of the transmitting device 1 and the internal buffer 23a in the decoding processing unit 23 of the receiving device 2 is caused in the network communication network 3. A case that occurs due to packet loss will be described. In addition, the apparatus structure in 2nd Embodiment is the same as that of 1st Embodiment.
[0064]
  When a packet is lost, the frame included in the lost packet cannot be decoded, so the internal buffer 13a of the encoder of the encoding processing unit 13 on the transmission device 1 side and the decoding on the reception device 2 side A state mismatch occurs with respect to the correlation data stored in the internal buffer 23a of the decoder of the conversion processing unit 23.
[0065]
  In the case where such a state mismatch regarding the correlation data stored in the internal buffers 13a and 23a occurs, in this embodiment, the frame immediately after the packet loss is not decoded and reproduced immediately, but received after the packet loss. The frames of several packets are decoded but not reproduced, and are reproduced by performing fade-in processing after performing several frames encoding processing until the signal waveform of the frame is stabilized.
[0066]
  The operation for realizing the above contents will be described below with reference to FIG.
[0067]
  Using the sequence numbers included in the received packets in the receiving unit 21 and the packet analyzing unit 22, the receiving device 2 can know the disappearance of the packets with the sequence numbers 1 and 2.
[0068]
  When receiving the loss of the packets with the sequence numbers 1 and 2, the receiving device 2 generates and reproduces silence for the frames of these lost packets.
[0069]
  Further, in the present embodiment, the receiving device 2 uses the internal buffer 23a of the decoding processing unit 23 before decoding the frame of the packet received next to the frames of the lost sequence numbers of the first and second packets. Initialize (reset).
[0070]
  Next, the receiving device 2 starts decoding the received sequence number 3 and subsequent packets, but only decodes and does not reproduce during N frames. That is, the frames of the two packets with the sequence numbers 3 and 4 are only decoded, and the audio is not reproduced.
[0071]
  Subsequent M frames are reproduced by performing a fade-in process for gradually increasing the sound volume from the sound volume 0 to the audio signal obtained by the decoding process. That is, for the frames of the two packets with the sequence numbers 3 and 4, the audio signal obtained by the decoding process is reproduced by performing a fade-in process for gradually increasing the volume from the volume 0.
[0072]
  After N + M frames, that is, for the frames of packets with sequence number 7 and subsequent packets, the audio signal obtained by the decoding process is reproduced as usual.
[0073]
  According to the second embodiment, when a packet is lost depending on the arrival state of a voice packet, it is possible to reproduce voice while reducing quality degradation.
[0074]
  (Third embodiment)
  Next, a third embodiment of the present invention will be described. The apparatus configuration in the third embodiment is the same as that in the first embodiment described above.
[0075]
  In the third embodiment, in packet communication, when a large number of IP packets arrive at one time, the packets are accumulated in the FIFO buffer in the receiving unit 21 of the receiving apparatus, and a part of the received packets needs to be discarded. A case where the above occurs will be described.
[0076]
  In such a case, when a part of the received packet is discarded and the audio frame included in all received packets is not decoded, it is stored in the internal buffer 13a in the encoding processing unit 13 of the transmission device 1. A state mismatch occurs between the correlation data and the correlation data stored in the internal buffer 23 a in the decoding processing unit 23 of the receiving device 2.
[0077]
  Also, it is often unacceptable to perform the decoding process for all frames to be discarded in consideration of the processing load.
[0078]
  In the third embodiment, an example is shown in which voice quality deterioration caused by not decoding all frames in the above case is reduced.
[0079]
  In the third embodiment, M frames are provided as boundary frames, and crossfading is performed to eliminate the discontinuity of the speech waveform and reduce quality degradation.
[0080]
  Here, when the number of discarded frames is very small, all the frames including the discarded frames may be decoded. For example, when the number of frames to be discarded X <N, all frames can be decoded.
[0081]
  Next, a specific example of the above processing will be described with reference to FIG.
[0082]
  The specific example shown in FIG. 6 shows processing when it is desired to discard packets having sequence numbers 11 to 14 because a large number of packets have accumulated in the FIFO buffer of the receiver 22.
[0083]
  At this time, packets with sequence numbers 11 and 12 are discarded. Further, the frames of the packets with the sequence numbers of 9 and 10 are used as boundary frames, and after these frames are decoded, fade-out processing for gradually reducing the sound volume to silence is performed on these boundary frames.
[0084]
  Further, the internal buffer 23a of the decoding processing unit 23 is initialized (reset) before the frame of the packet with the sequence number 13 is decoded.
[0085]
  Also, the packets with sequence numbers 13 and 14 are subjected to decoding processing, and the correlation data stored in the internal buffer 23a of the decoding processing unit 23 is updated. However, voice decoding is not performed for frames of packets with sequence numbers 13 and 14 that have been decoded.
[0086]
  Further, M frames subsequent to this are used as boundary frames, and an audio signal obtained by decoding these frames is subjected to fade-in processing for gradually increasing the volume from volume 0 and reproduced. That is, for the frames of the packets with sequence numbers 15 and 16, a fade-in process for gradually increasing the volume of the audio signal obtained by the decoding process from the volume 0 is performed.
[0087]
  Further, sound is reproduced in a state where the frames of the sequence numbers 9 and 10 and the frames of the 15 and 16 are overlapped and crossfade.
[0088]
  The frame of the packet after the packet with the sequence number of 17 reproduces the audio signal obtained by decoding as usual.
[0089]
  According to the third embodiment, even when a large number of IP packets arrive at one time and packets accumulate too much in the FIFO buffer in the receiving unit 21 of the receiving device 2, some of the received packets are discarded. Audio reproduction can be performed with reduced quality degradation.
[0090]
  (Fourth embodiment)
  Next, a fourth embodiment of the present invention will be described.
[0091]
  FIG. 7 is a block diagram showing a functional configuration of a voice packet communication system according to the fourth embodiment of the present invention. In the figure, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted. Further, the difference between the fourth embodiment and the first embodiment is that the reception device 2 is provided with an erasure compensation processing unit 26 and a mixer 27.
[0092]
  In the fourth embodiment, an example in which the packet loss compensation process is performed by extending the process when the packet shown in the second embodiment is lost will be described.
[0093]
  When a packet is lost, the frame included in the lost packet cannot be decoded and the internal state of the decoder cannot be updated. Therefore, the encoding processing unit 13 on the transmission apparatus 1 side encodes the frame. There is a discrepancy between the correlation data at the time of conversion to the correlation data stored in the internal buffer 23a of the decoding processing unit 23 that decodes the frame.
[0094]
  The receiving device 2 can know the packet loss by using the sequence number included in the received packet in the receiving unit 21 and the packet analyzing unit 22. In the present embodiment, a case where packets with sequence numbers 1 and 2 are lost will be described with reference to FIG. 8 as a specific example.
[0095]
  When receiving the loss of these packets, the receiving apparatus 2 performs the first reproduction process in the erasure compensation processing unit 26 and the second reproduction process in the decoding processing unit 24, and outputs from the erasure compensation processing unit 26. The signal and the output signal from the decoding processing unit 24 are mixed by the mixer 27 and input to the D / A conversion unit 24.
[0096]
  That is, when the receiving device 2 knows the loss of a packet, the correlation stored in the internal buffer 23a of the decoding processing unit 23 after decoding the last frame included in the last packet received normally. Data is copied to the internal buffer 26 a of the erasure compensation processing unit 26.
[0097]
  The erasure compensation processing unit 26 performs the first reproduction process. In the first reproduction process, the correlation data copied to the internal buffer 26a is used to artificially generate and interpolate an audio waveform to be reproduced instead of the lost frame, and N frames following the interpolated frame are generated. Is generated and decoded, and the subsequent M frames are pseudo-generated and decoded and subjected to fade-out processing, and output to the D / A converter 24 via the mixer 27.
[0098]
  The decryption processing unit 23 performs the second reproduction process. In this second reproduction process, the internal buffer 23a is initialized (reset) before decoding the frame of the next packet that has not been lost.
[0099]
  Next, the decoding processing unit 23 starts decoding the received packet, but only decodes the first N frames and does not reproduce it. The M frames subsequent to the N frames are subjected to a fade-in process for gradually increasing the volume of the audio signal obtained by the decoding process in the decoding processing unit 24 from zero.
[0100]
  A signal that has been subjected to the fade-in process in the decoding processing unit 24 and a signal that has been subjected to the fade-out process on the speech waveform generated by the erasure compensation processing unit are combined (cross-fade) and reproduced.
[0101]
  Further, in the decoding processing unit 23, a normal decoding process is performed on the frames following the N + M frames after the initialization (reset) of the internal buffer 23a, and the decoded audio signal is processed. Is played as usual.
[0102]
  Next, a specific example of the above processing will be described with reference to FIG.
[0103]
  In the specific example shown in FIG. 8, when the receiving device 2 knows that the packets with the sequence numbers 1 and 2 are lost, the receiving device 2 decodes the frame included in the packet with the sequence number 0 that has been normally received. The correlation data stored in the internal buffer 23 a of the subsequent decoding processing unit 23 is copied to the internal buffer 26 a of the erasure compensation processing unit 26.
[0104]
  In the erasure compensation processing unit 26, as the first reproduction process, the correlation data copied to the internal buffer 26a is used, and the lost sequence number 1 ′ to be reproduced instead of the frames of the first and second packets. And 2 ′ frame and the subsequent 3′-6 ′ frame speech waveform are simulated and interpolated, and the interpolated 1′-4 ′ frame is only decoded. Subsequently, the frames of the packets with sequence numbers 5 ′ and 6 ′ are decoded and subjected to fade-out processing to generate a speech waveform, which is then sent to the D / A converter 24 via the mixer 27. Output.
[0105]
  In the decoding processing unit 23, the internal buffer 23a is initialized (reset) before decoding the frame of the received packet with the sequence number 3 as the second reproduction processing.
[0106]
  Next, the decoding processing unit 23 only decodes the frames of the received sequence numbers 3 and 4 and does not reproduce them. Thereby, the correlation data stored in the internal buffer 23a of the decoding processing unit 23 becomes normal.
[0107]
  In addition, the frames of the packets with the sequence numbers 5 and 6 are subjected to fade-in processing for gradually increasing the volume of the audio signal obtained by the decoding processing in the decoding processing unit 24 from 0, and the mixing unit 27 Is output to the D / A converter 24.
[0108]
  As a result, the signal that has been subjected to the fade-in process in the decoding processing unit 23 by the mixing unit 27 and the signal that has been subjected to the fade-out process in the erasure compensation processing unit 26 are combined (crossfaded) and reproduced.
[0109]
  Further, the decoding processing unit 23 performs a normal decoding process on the frames of packets with sequence numbers of 7 and later. The decoded audio signal is reproduced as usual.
[0110]
  According to the fourth embodiment, even when a packet is lost in the communication network 3, it is possible to reproduce sound while reducing quality deterioration.
[0111]
  (Fifth embodiment)
  Next, a fifth embodiment of the present invention will be described. In addition, the apparatus structure in 5th Embodiment is the same as that of 4th Embodiment mentioned above.
[0112]
  In the fifth embodiment, as shown in FIG. 9, after the packet with the sequence number 0 is received, the packet with the sequence number 1 following this packet is delayed. A process in the case where the No. 2 and No. 2 ′ frames are simulated and reproduced will be described.
[0113]
  The receiving device 2 can know in the receiving unit 21 and the packet analyzing unit 22 that the packet to be received is delayed. In the present embodiment, a case where a packet having a sequence number of 1 and thereafter is delayed will be described with reference to FIG. 9 as a specific example.
[0114]
  In response to the packet delay, the receiving apparatus 2 performs the first reproduction process in the erasure compensation processing unit 26 and the second reproduction process in the decoding processing unit 24, and the output signal from the erasure compensation processing unit 26 and the decoding process are performed. The output signal from the unit 24 is mixed by the mixer 27 and input to the D / A conversion unit 24.
[0115]
  That is, when the receiving device 2 knows the delay of the packet, the correlation data stored in the internal buffer 23a of the decoding processing unit 23 after decoding the last frame included in the last packet received normally. Is copied to the internal buffer 26 a of the erasure compensation processing unit 26.
[0116]
  The erasure compensation processing unit 26 performs the first reproduction process. In the first reproduction process, the correlation data copied to the internal buffer 26a is used to generate and interpolate a voice waveform of a frame that should exist within the delay time instead of a frame that cannot be received with a delay, N frames following the interpolated frame are pseudo-generated and fade-out processing is performed on these N frames, which are output to the D / A conversion unit 24 via the mixing unit 27.
[0117]
  The decryption processing unit 23 performs the second reproduction process. In the second reproduction process, the decoding process of the packets having the sequence numbers received after the delay is started. At this time, after the frames of the first N packets are decoded, a fade-in process for gradually increasing the volume of the audio signal obtained by the decoding process from 0 is performed.
[0118]
  Further, the audio waveform subjected to the fade-in process in the decoding processing unit 23 and the audio waveform subjected to the fade-out process generated by the erasure compensation processing unit 26 are synthesized (cross-fade) by the mixing unit 27 and reproduced.
[0119]
  Further, in the decoding processing unit 23, a normal decoding process is performed on the frames of the packets whose sequence numbers are No. 3 and later, and the decoded audio signal is reproduced as usual.
[0120]
  Next, a specific example of the above processing will be described with reference to FIG.
[0121]
  In the specific example shown in FIG. 9, when the receiving apparatus 2 knows the delay of the packet whose sequence number is 1 or later, the decoding processor 23 includes the sequence number normally received by the decoding processor 23 in the 0th packet. The correlation data stored in the internal buffer 23 a of the decoding processing unit 23 after decoding the frame is copied to the internal buffer 26 a of the erasure compensation processing unit 26.
[0122]
  The erasure compensation processing unit 26 uses the correlation data stored in the internal buffer 26a to output the audio signals of the frames 1 ′ and 2 ′ to be reproduced instead of the frames included in the packet that must exist within the delay time. Simulate.
[0123]
  Next, packets 1 and 2 arrive late.
[0124]
  In the decoding processing unit 23, since the correlation data is held in the internal buffer 23a immediately after decoding the frame included in the packet with the sequence number 0, the sequence number is set to 1 using the correlation data. If the frame included in the second packet is decoded, the correlation data stored in the internal buffer 13a of the encoding processing unit 13 and the correlation data stored in the internal buffer 23a of the decoding processing unit 23 do not match. Does not occur.
[0125]
  Furthermore, the encoding processing unit 23 performs a fade-in process for gradually increasing the volume of the audio signal obtained by decoding after decoding the frames of the packets having the sequence numbers 1 and 2 after the decoding. To the D / A converter 24 via the mixer 27.
[0126]
  On the other hand, in the erasure compensation processing unit 26, N (here, N = 2) pseudo frames 3 ′ and 4 ′ are avoided in order to avoid deterioration in audio quality due to discontinuous audio signal waveforms to be reproduced. Are output to the D / A converter 24 via the mixer 27.
[0127]
  As a result, the audio signal waveform obtained by decoding the frames included in the packets having the sequence numbers 1 and 2 output from the encoding processing unit 23 and the pseudo frames 3 ′ and 4 ′ output from the erasure compensation processing unit 26 are obtained. Are mixed by the mixing unit 27 and subjected to cross-fade processing, which is output to the D / A conversion unit 24.
[0128]
  Further, the decoding processing unit 23 performs a normal decoding process on the frames of packets with sequence numbers of 3 and after. The decoded audio signal is reproduced as usual.
[0129]
  According to the fifth embodiment, even when a packet is delayed in the communication network 3, it is possible to reproduce sound while reducing quality deterioration.
[0130]
  (Sixth embodiment)
  Next, a sixth embodiment of the present invention will be described with reference to FIG. In addition, the apparatus structure in 6th Embodiment is the same as that of 4th and 5th Embodiment mentioned above.
[0131]
  In the sixth embodiment, as shown in FIG. 10, instead of the process of the fifth embodiment described above, the decryption processing unit 23 discards the packets with the sequence numbers 1 and 2 and the sequence number 3 The frames included in the No. 4 and No. 4 packets are subjected to a decoding process and further subjected to a fade-in process. Were mixed by the mixing unit 27 and cross-faded and output.
[0132]
  Also in the sixth embodiment, as in the fifth embodiment, even when a packet is delayed in the communication network 3, it is possible to reproduce sound while reducing quality degradation.
[0133]
  (Seventh embodiment)
  Next, a seventh embodiment of the present invention will be described.
[0134]
  FIG. 11 is a block diagram showing a receiving apparatus of a voice packet communication system according to the seventh embodiment of the present invention. In the figure, the same components as those of the fourth embodiment described above are denoted by the same reference numerals, and description thereof is omitted. The difference between the seventh embodiment and the fourth embodiment is that an internal buffer state holding unit 28 is provided in place of the erasure compensation processing unit 26 in the fourth embodiment and the mixing unit 28 is removed.
[0135]
  Also with the above configuration, the same processing as in the fifth embodiment can be performed. That is, G. When the decoding processing unit 23 and the erasure compensation processing unit 26 are substantially the same as in the case of 729, the correlation data stored in the internal buffer 23a of the decoding processing unit 23 is When the next frame that has been delayed is decoded and temporarily stored in the buffer status holding unit 28, the same processing is performed by starting decoding using the stored correlation data. It can be carried out.
[0136]
  In this case, immediately after decoding the frame included in the packet with the sequence number 0, the correlation data stored in the internal buffer 23a of the decoding processing unit 23 is copied to the internal buffer state holding unit 28. The pseudo frames from No. 1 'to No. 4' are generated in the same manner as the processing of the erasure compensation processing unit 26.
[0137]
  Next, when decoding the frame included in the packet with the sequence number 1 received with delay, the correlation data held in the internal buffer state holding unit 28 is converted into the internal buffer 23a of the decoding processing unit 26. After decoding and returning to the decryption process.
[0138]
  Also in the seventh embodiment, as in the fifth embodiment, even when a packet is delayed in the communication network 3, it is possible to reproduce sound with reduced quality degradation.
[0139]
  Each embodiment mentioned above is an example of the present invention, and the present invention is not limited only to the above-mentioned embodiment.
[0140]
  It goes without saying that the method of the present invention can also be applied to cases where packets are received in a state where the preceding and succeeding packets are switched and the frames of these packets have to be decoded.
[0141]
【The invention's effect】
  As described above, according to the real-time packet processing apparatus and the method of the present invention, when the frames to be processed are not continuous when performing the decoding process by the decoding processing unit, Only the decoding processing unit performs decoding on the N frames, and the audio reproduction processing is performed from the next frame of the N frames without performing the audio reproduction processing on the decoded N frames. Since the decoding is performed on the N frames from the discontinuous frames, and the sound reproduction processing with the volume reduced is performed on the decoded N frames. The correlation data used in the decoding processing unit when decoding can be matched with the correlation data in the encoding processing unit on the transmission device side, and the next frame of the N frames In which exhibits the excellent effect that it is possible to perform et appropriate audio reproduction process.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a functional configuration of a real-time packet processing apparatus according to a first embodiment of the present invention.
FIG. 2 is a diagram for explaining packetization of a voice signal by the voice packet transmitting apparatus according to the first embodiment of the present invention.
FIG. 3 is a diagram for explaining a real-time transfer protocol header used in the first embodiment of the present invention.
FIG. 4 is a timing chart illustrating processing of the real-time packet processing device according to the first embodiment of the present invention.
FIG. 5 is a timing chart for explaining processing of the real-time packet processing device according to the second embodiment of the present invention.
FIG. 6 is a timing chart for explaining processing of the real-time packet processing device according to the third embodiment of the present invention.
FIG. 7 is a block diagram showing a functional configuration of a real-time packet processing device according to a fourth embodiment of the present invention.
FIG. 8 is a timing chart illustrating processing of a real-time packet processing device according to the fourth embodiment of the present invention.
FIG. 9 is a timing chart illustrating processing of the real-time packet processing device according to the fifth embodiment of the present invention.
FIG. 10 is a timing chart illustrating processing of a real-time packet processing device according to a sixth embodiment of the present invention.
FIG. 11 is a block diagram showing a functional configuration of a receiving device of a real-time packet processing device according to a seventh embodiment of the present invention.
[Explanation of symbols]
  DESCRIPTION OF SYMBOLS 1 ... Transmission apparatus, 2 ... Reception apparatus, 3 ... Communication network, 11 ... Voice input part, 12 ... Analog / digital (A / D) conversion part, 13 ... Coding process part, 13a ... Internal buffer, 14 ... Packet generation 15, transmitting unit, 21, receiving unit, 22, packet analyzing unit, 23, decoding processing unit, 23 a, internal buffer, 24, digital / analog (D / A) conversion unit, 25, audio output unit, 26 ... disappearance compensation processing unit, 26a ... internal buffer, 27 ... mixing unit, 28 ... internal buffer state holding unit.

Claims (8)

A frame formed by encoding a plurality of sampling data obtained by cutting a continuous input audio signal every predetermined cycle and sampling the cut signal every predetermined sampling time shorter than the cycle by an encoding processing unit. And generating the packet including the frame for each frame and receiving the packet by a receiving device via a communication network from a transmitting device that sequentially transmits the frame, and the frame included in the received packet by the receiving device In a real-time packet processing apparatus using a predictive coding method for decoding audio data by a decoding processing unit and performing audio reproduction processing on sampling data included in the decoded frame,
A buffer for storing packets received from the transmitting device;
Means for analyzing a frame encoded when decoding a frame included in a packet input from the buffer and holding the analysis result as correlation data;
Means for determining whether or not the frames to be processed are continuous when performing the decoding process by the decoding processing unit;
As a result of the determination, when the frames to be decoded are not continuous, the decoding processing unit only performs decoding on N frames from the discontinuous frames, and the decoded N Means for performing the audio reproduction processing by performing decoding using the held correlation data from the next frame of the N frames without performing the audio reproduction processing on the frames ;
When the number of packets stored in the buffer exceeds a predetermined number, discard a packet other than the last N frames among a predetermined number of consecutive packets to be discarded stored in the buffer. Means to
Means for performing a fade-in process in which the last N frames are only decoded, and the volume of the subsequent M packets is gradually increased from a silent state;
A real-time packet processing apparatus comprising: means for performing a fade-out process for gradually reducing the volume of the M packets before the discarded packet to a silent state . A frame formed by encoding a plurality of sampling data obtained by cutting a continuous input audio signal every predetermined cycle and sampling the cut signal every predetermined sampling time shorter than the cycle by an encoding processing unit. And generating the packet including the frame for each frame and receiving the packet by a receiving device via a communication network from a transmitting device that sequentially transmits the frame, and the frame included in the received packet by the receiving device In a real-time packet processing apparatus using a predictive coding method for decoding audio data by a decoding processing unit and performing audio reproduction processing on sampling data included in the decoded frame,
A buffer for storing packets received from the transmitting device;
Means for analyzing a frame encoded when decoding a frame included in a packet input from the buffer and holding the analysis result as correlation data;
Means for determining whether or not the frames to be processed are continuous when performing the decoding process by the decoding processing unit;
As a result of the determination, when the frames to be decoded are not continuous, the decoding processing unit performs decoding on N frames from discontinuous frames, and then performs the decoding The audio reproduction processing is performed by performing the audio reproduction processing with the volume reduced on the N frames, and performing decoding using the held correlation data from the next frame of the N frames. Means for applying ,
When the number of packets stored in the buffer exceeds a predetermined number, discard a packet other than the last N frames among a predetermined number of consecutive packets to be discarded stored in the buffer. Means to
Means for performing a fade-in process in which the last N frames are only decoded, and the volume of the subsequent M packets is gradually increased from a silent state;
A real-time packet processing apparatus comprising: means for performing a fade-out process for gradually reducing the volume of the M packets before the discarded packet to a silent state . The said determination means has a means to determine that the said frame became discontinuous based on the sequence number contained in the said packet, when this sequence number becomes discontinuous. Item 3. The real-time packet processing device according to Item 2. The real-time packet processing apparatus according to claim 1, further comprising a unit that reproduces audio by superimposing the frame subjected to the fade-out process and the frame subjected to the fade-in process. A frame formed by encoding a plurality of sampling data obtained by cutting a continuous input audio signal every predetermined cycle and sampling the cut signal every predetermined sampling time shorter than the cycle by an encoding processing unit. And generating the packet including the frame for each frame and receiving the packet by a receiving device via a communication network from a transmitting device that sequentially transmits the frame, and the frame included in the received packet by the receiving device In a real-time packet processing method using a predictive coding method for decoding audio data by a decoding processing unit and performing audio reproduction processing on sampling data included in the decoded frame,
The receiving device stores the packet received from the transmitting device in a buffer, decodes the frame included in the packet input from the buffer , analyzes the encoded frame, and correlates the analysis result with the correlation data. And when the decoding process by the decoding processing unit is not continuous, the decoding processing unit applies to N frames from discontinuous frames when the frames to be processed are not continuous. Only decoding is performed, and the audio reproduction processing is not performed on the decoded N frames, and decoding is performed using the correlation data held from the next frame of the N frames. and facilities the audio reproduction process, when the number of packets stored in the buffer exceeds a predetermined number, discarding a predetermined number of successive stored in the buffer The packets other than the last N frames of the packets to be processed are discarded, and the last N frames are only decoded, and the subsequent M frames are gradually changed from the silent state. A real-time packet processing method , wherein a fade-in process for increasing the volume is performed, and a fade-out process for gradually decreasing the volume to a silence is applied to a frame of M packets before the discarded packet. A frame formed by encoding a plurality of sampling data obtained by cutting a continuous input audio signal every predetermined cycle and sampling the cut signal every predetermined sampling time shorter than the cycle by an encoding processing unit. And generating the packet including the frame for each frame and receiving the packet by a receiving device via a communication network from a transmitting device that sequentially transmits the frame, and the frame included in the received packet by the receiving device In a real-time packet processing method using a predictive coding method for decoding audio data by a decoding processing unit and performing audio reproduction processing on sampling data included in the decoded frame,
The receiving device stores the packet received from the transmitting device in a buffer, decodes the frame included in the packet input from the buffer , analyzes the encoded frame, and correlates the analysis result with the correlation data. And when the decoding process by the decoding processing unit is not continuous, the decoding processing unit applies to N frames from discontinuous frames when the frames to be processed are not continuous. After decoding, the audio reproduction processing with a reduced volume is performed on the decoded N frames, and the held correlation data is used from the next frame of the N frames. performs decoding by facilities the audio reproduction processing Te, when the number of packets stored in the buffer exceeds a predetermined number, is stored in the buffer A packet other than the last N frames of the predetermined number of packets to be discarded is discarded, and the last N frames are only decoded, and the frames of M packets that follow this are decoded. A real-time process in which a fade-in process for gradually increasing the volume from a silent state is performed, and a fade-out process for gradually decreasing the volume to a silence is performed on a frame of M packets before the discarded packet. Packet processing method. The receiving device, based on the sequence number included in the packet, when the sequence number becomes discontinuous, claim 5 or claim, characterized in that determines that the frame is discontinuous 6 The real-time packet processing method described in 1. 7. The real-time packet processing method according to claim 5 , wherein the receiving device reproduces audio by superimposing the frame subjected to the fade-out process and the frame subjected to the fade-in process. 8.

Images